From tjreedy at udel.edu Wed May 1 01:12:10 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 May 2019 01:12:10 -0400 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> Message-ID: On 4/30/2019 5:24 PM, Chris Withers wrote: > Hi All, > > I have a crazy idea of getting unittest.mock up to 100% code coverage. > > I noticed at the bottom of all of the test files in testmock/, there's a: > > if __name__ == '__main__': > ??? unittest.main() > > ...block. Such blocks should be excluded from coverage by the default .coveragerc file. Mine came with exclude_lines = # Don't complain if non-runnable code isn't run: if 0: if __name__ == .__main__.: if DEBUG: -- Terry Jan Reedy From chris at withers.org Wed May 1 02:13:16 2019 From: chris at withers.org (Chris Withers) Date: Wed, 1 May 2019 07:13:16 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> Message-ID: <98f5b8b6-6582-e348-c0fc-472ba228f131@withers.org> On 01/05/2019 06:12, Terry Reedy wrote: > Such blocks should be excluded from coverage by the default .coveragerc > file.? Mine came with > > exclude_lines = > ??? # Don't complain if non-runnable code isn't run: > ??? if 0: > ??? if __name__ == .__main__.: > ??? if DEBUG: Which .coveragerc are you referring to? There isn't one in the cpython repo and a current release of coverage.py doesn't appear to exclude these lines for me: https://circleci.com/gh/testing-cabal/mock/20 (line 44 in mock/tests/testsentinel.py) Chris From storchaka at gmail.com Wed May 1 02:46:57 2019 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 1 May 2019 09:46:57 +0300 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> Message-ID: 01.05.19 00:24, Chris Withers ????: > I have a crazy idea of getting unittest.mock up to 100% code coverage. > > I noticed at the bottom of all of the test files in testmock/, there's a: > > if __name__ == '__main__': > ??? unittest.main() > > ...block. > > How would people feel about these going away? I don't *think* they're > needed now that we have unittest discover, but thought I'd ask. These lines were added for purpose. They are needed for running tests in separate file as a script. $ ./python Lib/unittest/test/testmock/testcallable.py -v test_attributes (__main__.TestCallable) ... ok test_create_autospec (__main__.TestCallable) ... ok test_create_autospec_instance (__main__.TestCallable) ... ok test_hierarchy (__main__.TestCallable) ... ok test_non_callable (__main__.TestCallable) ... ok test_patch_spec (__main__.TestCallable) ... ok test_patch_spec_callable_class (__main__.TestCallable) ... ok test_patch_spec_instance (__main__.TestCallable) ... ok test_patch_spec_set (__main__.TestCallable) ... ok test_patch_spec_set_instance (__main__.TestCallable) ... ok test_subclasses (__main__.TestCallable) ... ok ---------------------------------------------------------------------- Ran 11 tests in 0.040s OK From chris at withers.org Wed May 1 03:09:55 2019 From: chris at withers.org (Chris Withers) Date: Wed, 1 May 2019 08:09:55 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> Message-ID: <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> On 01/05/2019 07:46, Serhiy Storchaka wrote: > 01.05.19 00:24, Chris Withers ????: >> I have a crazy idea of getting unittest.mock up to 100% code coverage. >> >> I noticed at the bottom of all of the test files in testmock/, there's a: >> >> if __name__ == '__main__': >> ???? unittest.main() >> >> ...block. >> >> How would people feel about these going away? I don't *think* they're >> needed now that we have unittest discover, but thought I'd ask. > > These lines were added for purpose. They are needed for running tests in > separate file as a script. > > $ ./python Lib/unittest/test/testmock/testcallable.py -v > test_attributes (__main__.TestCallable) ... ok > test_create_autospec (__main__.TestCallable) ... ok > test_create_autospec_instance (__main__.TestCallable) ... ok > test_hierarchy (__main__.TestCallable) ... ok > test_non_callable (__main__.TestCallable) ... ok > test_patch_spec (__main__.TestCallable) ... ok > test_patch_spec_callable_class (__main__.TestCallable) ... ok > test_patch_spec_instance (__main__.TestCallable) ... ok > test_patch_spec_set (__main__.TestCallable) ... ok > test_patch_spec_set_instance (__main__.TestCallable) ... ok > test_subclasses (__main__.TestCallable) ... ok Right, but that's not the documented way of running individual suites in the devguide. I'm happy to remove these on the basis that there should be one and only one way of doing things like this. Chris From tjreedy at udel.edu Wed May 1 06:24:59 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 May 2019 06:24:59 -0400 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: <98f5b8b6-6582-e348-c0fc-472ba228f131@withers.org> References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <98f5b8b6-6582-e348-c0fc-472ba228f131@withers.org> Message-ID: On 5/1/2019 2:13 AM, Chris Withers wrote: > On 01/05/2019 06:12, Terry Reedy wrote: >> Such blocks should be excluded from coverage by the default >> .coveragerc file.? Mine came with >> >> exclude_lines = >> ???? # Don't complain if non-runnable code isn't run: >> ???? if 0: >> ???? if __name__ == .__main__.: >> ???? if DEBUG: I am fairly sure these are the original lines. I added more to my personal copy to exclude some things specific to idlelib files. Idlelib files also have the same statement as the mock files. It is very handy for development. > Which .coveragerc are you referring to? The file in the directory that contains Ned's coverage package. At least that is where it is for me. > There isn't one in the cpython repo Since /coverage is not in the Python repo either, I would not expect it to be. > and a current release of coverage.py doesn't appear to exclude > these lines for me: > https://circleci.com/gh/testing-cabal/mock/20 > > (line 44 in mock/tests/testsentinel.py) I consider this a problem in the circleci coverage setup, not in the mock files. A faulty exclude-lines section leads to a faulty coverage calculation. Ned made it user-editable for a reason. -- Terry Jan Reedy From solipsis at pitrou.net Wed May 1 06:48:31 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 1 May 2019 12:48:31 +0200 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> Message-ID: <20190501124831.7faadb2b@fsol> On Tue, 30 Apr 2019 22:24:53 +0100 Chris Withers wrote: > Hi All, > > I have a crazy idea of getting unittest.mock up to 100% code coverage. > > I noticed at the bottom of all of the test files in testmock/, there's a: > > if __name__ == '__main__': > ??? unittest.main() > > ...block. > > How would people feel about these going away? I don't *think* they're > needed now that we have unittest discover, but thought I'd ask. If you are only asking to remove them because you want that score of 100% coverage, then I think you shouldn't. Regards Antoine. From vstinner at redhat.com Wed May 1 08:21:54 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 1 May 2019 08:21:54 -0400 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> Message-ID: Le mer. 1 mai 2019 ? 03:12, Chris Withers a ?crit : > Right, but that's not the documented way of running individual suites in > the devguide. Maybe, but I'm using that sometimes and it's useful for some specific issues. Is it possible to run an individual test file using unittest? Something like ./python -m unittest Lib/unittest/test/testmock/testcallable.py -v Victor -- Night gathers, and now my watch begins. It shall not end until my death. From chris at withers.org Wed May 1 08:30:40 2019 From: chris at withers.org (Chris Withers) Date: Wed, 1 May 2019 13:30:40 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> Message-ID: On 01/05/2019 13:21, Victor Stinner wrote: > Le mer. 1 mai 2019 ? 03:12, Chris Withers a ?crit : >> Right, but that's not the documented way of running individual suites in >> the devguide. > > Maybe, but I'm using that sometimes and it's useful for some specific > issues. Is it possible to run an individual test file using unittest? > > Something like ./python -m unittest > Lib/unittest/test/testmock/testcallable.py -v Yep: $ ./python.exe -m unittest Lib/unittest/test/testmock/testsentinel.py ...... ---------------------------------------------------------------------- Ran 6 tests in 0.002s $ ./python.exe -m unittest -v Lib/unittest/test/testmock/testsentinel.py testBases (Lib.unittest.test.testmock.testsentinel.SentinelTest) ... ok testCopy (Lib.unittest.test.testmock.testsentinel.SentinelTest) ... ok testDEFAULT (Lib.unittest.test.testmock.testsentinel.SentinelTest) ... ok testPickle (Lib.unittest.test.testmock.testsentinel.SentinelTest) ... ok testSentinelName (Lib.unittest.test.testmock.testsentinel.SentinelTest) ... ok testSentinels (Lib.unittest.test.testmock.testsentinel.SentinelTest) ... ok ---------------------------------------------------------------------- Ran 6 tests in 0.003s OK $ ./python.exe -m unittest -v unittest.test.testmock.testsentinel testBases (unittest.test.testmock.testsentinel.SentinelTest) ... ok testCopy (unittest.test.testmock.testsentinel.SentinelTest) ... ok testDEFAULT (unittest.test.testmock.testsentinel.SentinelTest) ... ok testPickle (unittest.test.testmock.testsentinel.SentinelTest) ... ok testSentinelName (unittest.test.testmock.testsentinel.SentinelTest) ... ok testSentinels (unittest.test.testmock.testsentinel.SentinelTest) ... ok ---------------------------------------------------------------------- Ran 6 tests in 0.003s cheers, Chris From p.f.moore at gmail.com Wed May 1 08:37:01 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 May 2019 13:37:01 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: <20190501124831.7faadb2b@fsol> References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> Message-ID: I agree - removing this just to make the coverage figures look pretty seems like the wrong motivation. Configuring coverage to understand that you want to exclude these lines from the checking would be fine, as would accepting that a coverage of slightly less than 100% is OK. Removing functionality that people use (whether or not they have other ways of getting the same results) needs a stronger justification, IMO. Paul On Wed, 1 May 2019 at 11:51, Antoine Pitrou wrote: > > On Tue, 30 Apr 2019 22:24:53 +0100 > Chris Withers wrote: > > Hi All, > > > > I have a crazy idea of getting unittest.mock up to 100% code coverage. > > > > I noticed at the bottom of all of the test files in testmock/, there's a: > > > > if __name__ == '__main__': > > unittest.main() > > > > ...block. > > > > How would people feel about these going away? I don't *think* they're > > needed now that we have unittest discover, but thought I'd ask. > > If you are only asking to remove them because you want that score of > 100% coverage, then I think you shouldn't. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/p.f.moore%40gmail.com From storchaka at gmail.com Wed May 1 08:39:15 2019 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 1 May 2019 15:39:15 +0300 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> Message-ID: 01.05.19 10:09, Chris Withers ????: > Right, but that's not the documented way of running individual suites in > the devguide. > > I'm happy to remove these on the basis that there should be one and only > one way of doing things like this. This principle is not applicable here because the Python testsuite is not a public API. It is not for use of Python users. We try to support several different ways of running tests. This allows to catch some environment depended flaws in tests and serves as a kind of the test of unittest itself. Not all test files are made discoverable yet, but we move in this direction. I do not see what is wrong here and suggest to not break working code. From chris at withers.org Wed May 1 08:51:22 2019 From: chris at withers.org (Chris Withers) Date: Wed, 1 May 2019 13:51:22 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> Message-ID: On 01/05/2019 13:37, Paul Moore wrote: > I agree - removing this just to make the coverage figures look pretty > seems like the wrong motivation. > > Configuring coverage to understand that you want to exclude these > lines from the checking would be fine, as would accepting that a > coverage of slightly less than 100% is OK. Removing functionality that > people use (whether or not they have other ways of getting the same > results) needs a stronger justification, IMO. It's an interesting point; I personally don't see much value in coverage of less than 100%, if you're going to look at coverage: no-one is going to focus on or potentially even notice whether coverage moved from 99% to 99% (no typo), even though a bunch of new uncovered code may have been introduced. If people are actually using these blocks, then so be it, but it feels like the people who want them to stick around are saying they're using them just on the off chance they might use them, which feels like a poor reason to keep a bunch of dead code around. I'm not sure I understand how triggering via unittest or via unittest (again, no typo, both these __main__ blocks and python -m unittest are using the same test runner) could make a difference. Chris From chris at withers.org Wed May 1 08:56:44 2019 From: chris at withers.org (Chris Withers) Date: Wed, 1 May 2019 13:56:44 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> Message-ID: Sorry, accidentally include a comment for this in a reply to Paul: On 01/05/2019 13:39, Serhiy Storchaka wrote: > We try to support several different ways of running tests. This allows > to catch some environment depended flaws in tests and serves as a kind > of the test of unittest itself. Not all test files are made discoverable > yet, but we move in this direction. I'm not sure I understand how triggering via unittest or via unittest (no typo, both these __main__ blocks and python -m unittest are using the same test runner) could make a difference. And sorry, to be clear, I'm only taking about the ones in unittest/mock/testmock - Rob Collins confirmed these where never needed in a private reply: > They were never needed ? > > Removal is fine with me. Chris From p.f.moore at gmail.com Wed May 1 09:22:48 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 1 May 2019 14:22:48 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> Message-ID: On Wed, 1 May 2019 at 13:53, Chris Withers wrote: > > On 01/05/2019 13:37, Paul Moore wrote: > > I agree - removing this just to make the coverage figures look pretty > > seems like the wrong motivation. > > > > Configuring coverage to understand that you want to exclude these > > lines from the checking would be fine, as would accepting that a > > coverage of slightly less than 100% is OK. Removing functionality that > > people use (whether or not they have other ways of getting the same > > results) needs a stronger justification, IMO. > > It's an interesting point; I personally don't see much value in coverage > of less than 100%, if you're going to look at coverage: no-one is going > to focus on or potentially even notice whether coverage moved from 99% > to 99% (no typo), even though a bunch of new uncovered code may have > been introduced. That's a fair point. > If people are actually using these blocks, then so be it, but it feels > like the people who want them to stick around are saying they're using > them just on the off chance they might use them, which feels like a poor > reason to keep a bunch of dead code around. If your argument was "this is dead code, and should be removed to help future maintenance", I'd have no problem with that. My point was simply that I think that adjusting the code to make the coverage stats hit 100% feels like going at things the wrong way round. Is it really that difficult to simply tell coverage to ignore them? I thought someone had already pointed to a coveragerc file that let you do this. Personally, I don't use those blocks at all, so it doesn't matter to me whether they stay or go in any practical sense. Paul From chris at withers.org Wed May 1 09:30:01 2019 From: chris at withers.org (Chris Withers) Date: Wed, 1 May 2019 14:30:01 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> Message-ID: <05ef1790-dc90-f5a6-a25c-9ba5abc66a3a@withers.org> On 01/05/2019 14:22, Paul Moore wrote: >> If people are actually using these blocks, then so be it, but it feels >> like the people who want them to stick around are saying they're using >> them just on the off chance they might use them, which feels like a poor >> reason to keep a bunch of dead code around. > > If your argument was "this is dead code, and should be removed to help > future maintenance", I'd have no problem with that. Yep, that's exactly my point :-) The dev guide shows how to run individual tests without these and doesn't mention them at all. As someone coming back to core dev late last year, I was left wondering whether I *should* be using them and that created confusion for me; having only one way to do things seems like a good thing here. > My point was > simply that I think that adjusting the code to make the coverage stats > hit 100% feels like going at things the wrong way round. Agreed, but my focus here is to get to 100% for mock so that it's clear that all the code is there for a reason; mock is very complicated by necessity, and having examples of why code needs to be there is what I'm aiming for most of all. > Is it really that difficult to simply tell coverage to ignore them? I > thought someone had already pointed to a coveragerc file that let you > do this. It would be if the cpython repo had a coveragerc, but it does not. People maintaining their own ad-hoc coverage configs seems like a pretty bad idea. > Personally, I don't use those blocks at all, so it doesn't matter to > me whether they stay or go in any practical sense. Right, that's by gut feeling here: I don't want people encountering this mock codebase to have to wonder whether they should be running the tests using these blocks versus the way described in the dev guide, and stressing about what the differences might be, when there aren't any... Chris From tir.karthi at gmail.com Wed May 1 09:52:23 2019 From: tir.karthi at gmail.com (Karthikeyan) Date: Wed, 1 May 2019 19:22:23 +0530 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> Message-ID: On Wed, May 1, 2019 at 6:13 PM Serhiy Storchaka wrote: > 01.05.19 10:09, Chris Withers ????: > > Right, but that's not the documented way of running individual suites in > > the devguide. > > > > I'm happy to remove these on the basis that there should be one and only > > one way of doing things like this. > > This principle is not applicable here because the Python testsuite is > not a public API. It is not for use of Python users. > > We try to support several different ways of running tests. This allows > to catch some environment depended flaws in tests and serves as a kind > of the test of unittest itself. Not all test files are made discoverable > yet, but we move in this direction. > Agreed. This also has helped in the past to find issues like https://bugs.python.org/issue29512#msg299045. > > I do not see what is wrong here and suggest to not break working code. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/tir.karthi%40gmail.com > -- Regards, Karthikeyan S -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris at withers.org Wed May 1 10:20:48 2019 From: chris at withers.org (Chris Withers) Date: Wed, 1 May 2019 15:20:48 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <10a06aab-7190-c41e-d5f7-64e1b64005ad@withers.org> Message-ID: On 01/05/2019 14:52, Karthikeyan wrote: > > > We try to support several different ways of running tests. This allows > to catch some environment depended flaws in tests and serves as a kind > of the test of unittest itself. Not all test files are made > discoverable > yet, but we move in this direction. > > Agreed. This also has helped in the past to find issues like > https://bugs.python.org/issue29512#msg299045. My read of that issue is that issue is that the only problem that was found is that an ambiguous way of running tests, which isn't included in the devguide, and maybe for good reason, caused a problem which sucked up a bunch of Victor's time. The reason Lib/test/ was on sys.path was because the tests were run with ./python Lib/test/test_bisect.py, when it shouldn't be. Had it been run with: ./ python -m test.test_bisect ...it would not have been, and bisect.py would not have needed renaming to bisect_cmd.py. Chris From jerrywiltse at gmail.com Wed May 1 11:09:19 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Wed, 1 May 2019 11:09:19 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro Message-ID: I looked through all the windows builders here and could not find any which generate this package, can anyone tell me where the work happens? I also tried searching looking for scripts and CI files which create the package in all the relevant repos and could not find those either. Is this information not public? Is it a manual build? Gerald R. Wiltse jerrywiltse at gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Wed May 1 12:09:13 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 2 May 2019 01:09:13 +0900 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> Message-ID: <23753.50345.124026.479885@turnbull.sk.tsukuba.ac.jp> Executive summary: "There should be a tool" (sorry, I'm not volunteering any time soon) that could be added to $VCS diff (say, "git coverage-diff" or "git diff --coverage"). Chris Withers writes: > It's an interesting point; I personally don't see much value in coverage > of less than 100%, if you're going to look at coverage: no-one is going > to focus on or potentially even notice whether coverage moved from 99% > to 99% (no typo), even though a bunch of new uncovered code may have > been introduced. I agree with the point you're making (that the difference between 100% and 99% is a very significant indicator that "something needs to be done here, it's obvious what, and [depending on project process] it's obvious who, too"), but it's also true that 99% is better than 98% and definitely better than 90% or less. Your point that it matters *which* 1% is more important, I think (see "Executive summary"). > If people are actually using these blocks, then so be it, but it feels > like the people who want them to stick around are saying they're using > them just on the off chance they might use them, which feels like a poor > reason to keep a bunch of dead code around. At least one person says he uses it, although I don't know how that fits with Robert's statement that "it was never needed". Steve From steve.dower at python.org Wed May 1 12:22:55 2019 From: steve.dower at python.org (Steve Dower) Date: Wed, 1 May 2019 12:22:55 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: On 01May2019 1109, Gerald Wiltse wrote: > I looked through all the windows builders here and could not find any > which?generate this package, can anyone tell me where the work happens? > I also tried searching looking for scripts and CI files which create the > package in all the relevant repos and could not find those either. > > Is this information not public? It's just not obviously documented (yet). We use the "PC/layout" script (run as "python PC/layout -h" to get arguments). You'll want at least --preset-embed and either --copy or --zip to specify where the layout is created. > Is it a manual build? It's semi-automated, but it's only triggered for releases and so it's not part of the normal CI configuration. Hope that helps, Steve From solipsis at pitrou.net Wed May 1 12:36:07 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 1 May 2019 18:36:07 +0200 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> <05ef1790-dc90-f5a6-a25c-9ba5abc66a3a@withers.org> Message-ID: <20190501183607.12fcdbe9@fsol> On Wed, 1 May 2019 14:30:01 +0100 Chris Withers wrote: > > > Is it really that difficult to simply tell coverage to ignore them? I > > thought someone had already pointed to a coveragerc file that let you > > do this. > > It would be if the cpython repo had a coveragerc, but it does not. Well, perhaps that could be added ;-) Regards Antoine. From jerrywiltse at gmail.com Wed May 1 13:05:36 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Wed, 1 May 2019 13:05:36 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: That does help, thanks so much for the fast response! I now see how it's executed on Azure pipelines also, so that should be enough for me to reverse engineer the rest. Followup questions if you have time: 1. It seems that directory doesn't exist in the 3.6 branches/tags, so it must have been done a different way back then. Do you think it's back-portable to 3.6? 2. One of our objectives is to effectively create a nearly identical embeddable package for linux. Do you think the layout code is generic enough to be modified to create a linux equivalent? Obviously, there are many alternative methods and documents for creating "embeddable" pythons on linux, but the windows package has all the characteristics we want, and of course we'd like our Windows and Linux process build and packaging processes to be as similar as possible, so it seems attractive to try to port this layout script to linux. Gerald R. Wiltse jerrywiltse at gmail.com On Wed, May 1, 2019 at 12:22 PM Steve Dower wrote: > On 01May2019 1109, Gerald Wiltse wrote: > > I looked through all the windows builders here and could not find any > > which generate this package, can anyone tell me where the work happens? > > I also tried searching looking for scripts and CI files which create the > > package in all the relevant repos and could not find those either. > > > > Is this information not public? > > It's just not obviously documented (yet). We use the "PC/layout" script > (run as "python PC/layout -h" to get arguments). You'll want at least > --preset-embed and either --copy or --zip to specify where the layout is > created. > > > Is it a manual build? > > It's semi-automated, but it's only triggered for releases and so it's > not part of the normal CI configuration. > > Hope that helps, > Steve > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed May 1 13:57:34 2019 From: steve.dower at python.org (Steve Dower) Date: Wed, 1 May 2019 13:57:34 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: On 01May2019 1305, Gerald Wiltse wrote: > That does help, thanks so much for the fast response!? I now see how > it's executed on Azure pipelines also, so that should be enough for me > to reverse engineer the rest. > > Followup questions if you have time: > 1. It seems that directory doesn't exist in the 3.6 branches/tags, so it > must have been done a different way back then.? Do you think it's > back-portable to 3.6? Yep, it should work with 3.6. There were scripts under Tools/msi back then that did the same thing, but with the Store app I rewrote it to be more flexible. > 2. One of our objectives is to effectively create a nearly identical > embeddable package for linux. Do you think the layout code is generic > enough to be modified to create a linux equivalent? Probably not, it's under the "PC" directory because it is entirely CPython specific. I'd love to be able to have a similar embeddable distro for Linux (and macOS), and I'll be investing some of my time in working towards this, but it's far more complicated than simply laying out the right files. If you're interested in helping with this, then we're in the right place to discuss it, but we'll likely have to move to the issue tracker at some point once we figure out who else wants/needs to be involved. > Obviously, there are many alternative methods and documents for creating > "embeddable" pythons on linux, but the windows package has all the > characteristics we want, and of course we'd like our Windows and Linux > process build and packaging processes to be as similar as possible, so > it seems attractive to try to port this layout script to linux. "make install" already does the same thing, but the install location matters at build time, so it's a little more complex to make it work as an embedded runtime. I'm sure there are people who know how to make it work though. Cheers, Steve From chris at withers.org Wed May 1 14:13:51 2019 From: chris at withers.org (Chris Withers) Date: Wed, 1 May 2019 19:13:51 +0100 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: <23753.50345.124026.479885@turnbull.sk.tsukuba.ac.jp> References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> <23753.50345.124026.479885@turnbull.sk.tsukuba.ac.jp> Message-ID: On 01/05/2019 17:09, Stephen J. Turnbull wrote: > Executive summary: > > "There should be a tool" (sorry, I'm not volunteering any time soon) > that could be added to $VCS diff (say, "git coverage-diff" or "git > diff --coverage"). That sounds like a very hard problem to solve... > > If people are actually using these blocks, then so be it, but it feels > > like the people who want them to stick around are saying they're using > > them just on the off chance they might use them, which feels like a poor > > reason to keep a bunch of dead code around. > > At least one person says he uses it, although I don't know how that > fits with Robert's statement that "it was never needed". Right, but the more we discuss this, the stronger my feeling that these should be removed everywhere, rather than just the few in unittest.test.test_mock that I intend to remove. Running: ./python some/package/test_whatever.py ...sets up a fundamentally hostile sys.path. As Karthikeyan highlighted, in https://bugs.python.org/issue29512 this resulted in a year's mucking around on a new tool that was added, and resulted in that tool getting a slightly more clumsy name. Running: ./python -m unittest some.package.test_whatever ...uses exactly the same runner, just without the unfortunate sys.path addition. I thought https://www.python.org/dev/peps/pep-0582/ had something about no longer add '.' to sys.path, but looks like I was mistaken. Chris From tir.karthi at gmail.com Wed May 1 15:00:13 2019 From: tir.karthi at gmail.com (Karthikeyan) Date: Thu, 2 May 2019 00:30:13 +0530 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> <23753.50345.124026.479885@turnbull.sk.tsukuba.ac.jp> Message-ID: > I thought https://www.python.org/dev/peps/pep-0582/ had something about > no longer add '.' to sys.path, but looks like I was mistaken. > It was discussed [0] in the past with respect to security since Perl did a similar change and PEP 432 is also mentioned in the thread. [1] [0] https://mail.python.org/pipermail/python-ideas/2017-June/045842.html [1] https://mail.python.org/pipermail/python-ideas/2017-June/045849.html -- Regards, Karthikeyan S -------------- next part -------------- An HTML attachment was scrubbed... URL: From jerrywiltse at gmail.com Wed May 1 14:47:23 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Wed, 1 May 2019 14:47:23 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: Yes, I'll help with this, and I try to get signed up with an account for the issue tracker ASAP. I've just started trying to build create the embeddable package for windows locally (all from source). Once I have that automated, I'll take a naive stab at the Linux layout. I'm sure i'll soon hit the additional complications you're talking about and come back to you with questions. Gerald R. Wiltse jerrywiltse at gmail.com On Wed, May 1, 2019 at 1:57 PM Steve Dower wrote: > On 01May2019 1305, Gerald Wiltse wrote: > > That does help, thanks so much for the fast response! I now see how > > it's executed on Azure pipelines also, so that should be enough for me > > to reverse engineer the rest. > > > > Followup questions if you have time: > > 1. It seems that directory doesn't exist in the 3.6 branches/tags, so it > > must have been done a different way back then. Do you think it's > > back-portable to 3.6? > > Yep, it should work with 3.6. There were scripts under Tools/msi back > then that did the same thing, but with the Store app I rewrote it to be > more flexible. > > > 2. One of our objectives is to effectively create a nearly identical > > embeddable package for linux. Do you think the layout code is generic > > enough to be modified to create a linux equivalent? > > Probably not, it's under the "PC" directory because it is entirely > CPython specific. > > I'd love to be able to have a similar embeddable distro for Linux (and > macOS), and I'll be investing some of my time in working towards this, > but it's far more complicated than simply laying out the right files. If > you're interested in helping with this, then we're in the right place to > discuss it, but we'll likely have to move to the issue tracker at some > point once we figure out who else wants/needs to be involved. > > > Obviously, there are many alternative methods and documents for creating > > "embeddable" pythons on linux, but the windows package has all the > > characteristics we want, and of course we'd like our Windows and Linux > > process build and packaging processes to be as similar as possible, so > > it seems attractive to try to port this layout script to linux. > > "make install" already does the same thing, but the install location > matters at build time, so it's a little more complex to make it work as > an embedded runtime. I'm sure there are people who know how to make it > work though. > > Cheers, > Steve > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 1 15:49:24 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 1 May 2019 12:49:24 -0700 Subject: [Python-Dev] PEP 574 ready for review In-Reply-To: <20190430101428.204457e6@fsol> References: <20190430101428.204457e6@fsol> Message-ID: Thanks Antoine. As BDFL-Delegate I'm happy with this version of the PEP, so it's my pleasure to accept it for inclusion in Python 3.8. Regards, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Wed May 1 16:21:56 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 2 May 2019 05:21:56 +0900 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> <23753.50345.124026.479885@turnbull.sk.tsukuba.ac.jp> Message-ID: <23753.65508.650064.941497@turnbull.sk.tsukuba.ac.jp> Chris Withers writes: > On 01/05/2019 17:09, Stephen J. Turnbull wrote: > > Executive summary: > > > > "There should be a tool" (sorry, I'm not volunteering any time soon) > > that could be added to $VCS diff (say, "git coverage-diff" or "git > > diff --coverage"). > > That sounds like a very hard problem to solve... I would say "compute-intensive" and "refactor-discouraging". Presumably a detailed coverage report would identify missing coverage by unit (line, function, branch), and if there was a canonical ordering on units those reports could be text diffed. Of course most refactorings will break that -- maybe that would be useless, but most refactorings will break any diff and we still manage to find them useful. I don't know enough to guess: such considerations are why I'm explicitly refusing to volunteer.... > Right, but the more we discuss this, the stronger my feeling that > these should be removed everywhere, rather than just the few in > unittest.test.test_mock that I intend to remove. I personally don't have a problem with that proposal or "Chris's feeling as sufficient condition" in practice, as I don't run tests that way (they frequently don't have an appropriate script interface, so I never developed the habit). My point was more that different people have different feelings about this, and there's some "talking past" going on because the concrete basis for those increasingly strong feelings on both sides seems to mostly be "I haven't heard any concrete reasons to change my mind (and likely more important, my way of doing things) so my original reasons carry the issue". Steve From tjreedy at udel.edu Wed May 1 17:12:42 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 1 May 2019 17:12:42 -0400 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: <05ef1790-dc90-f5a6-a25c-9ba5abc66a3a@withers.org> References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> <05ef1790-dc90-f5a6-a25c-9ba5abc66a3a@withers.org> Message-ID: On 5/1/2019 9:30 AM, Chris Withers wrote: > Agreed, but my focus here is to get to 100% for mock so that it's clear > that all the code is there for a reason; mock is very complicated by > necessity, and having examples of why code needs to be there is what I'm > aiming for most of all. I agree that complete 100.000% test coverage is a nice ideal, but sometimes the last percent can take hours to accomplish, if it is indeed sensibly possible. (If mock has no OS dependencies, then it may be for mock.) It is up to the individual developer to decide what is the priority to spend development time on. >> Is it really that difficult to simply tell coverage to ignore them? I >> thought someone had already pointed to a coveragerc file that let you >> do this. > > It would be if the cpython repo had a coveragerc, but it does not. I have asked more that once what .coveragerc file is being used by CI and whether we can somehow properly customize it for CPython. I have not gotten an answer. The devguide chapter (5) on coverage is deficient in not even mentioning customization. > People maintaining their own ad-hoc coverage configs seems like a pretty > bad idea. I would prefer not having to do that. But it is better than always getting bogus numbers. At least *I* can determine the real single-file test coverage for idlelib files (on Windows), even if the public coverage reports are misleading. Unless I forget, I record it on the first line of text_xxx files. > Right, that's by gut feeling here: I don't want people encountering this > mock codebase to have to wonder whether they should be running the tests > using these blocks versus the way described in the dev guide, The devguide describes the dependable but clumsy way to run tests from a command line. In my opinion, it is the worst way when one is editing a particular file. It is *much* easier to hit one key (F5 in IDLE) in an editor than to switch to a terminal window and type something like > python -m unittest idlelib.idle_test.test_configdialog It is much better to have a SyntaxError marked in the editor than displayed in a terminal or shell. Live tracebacks in an IDE, that can jump to exception locations in an editor are better than a dead traceback in a terminal window. With IDLE there is also the issue that automated unittests cannot completely replace human action and judgment. However, displaying isolated dialogs for human interaction can be automated. The 'if main' blocks for dialog modules do so. For example, for configdialog: if __name__ == '__main__': from unittest import main main('idlelib.idle_test.test_configdialog', verbosity=2, exit=False) from idlelib.idle_test.htest import run run(ConfigDialog) One can either close the box or check the visual appearance and live behavior of the dialog. In this case, running the class is sufficient. For other modules, additional setup code is needed, which should be excluded from coverage. > and stressing about what the differences might be, when there aren't any... Unittest users should know that it has both code and command line APIs. The devguide should mention running tests from code with main or refer to the appropriate section of the unittest doc. It should at least document the use of if __name__ == '__main__': unittest.main(verbosity=2) in test_xxx modules. -- Terry Jan Reedy From jerrywiltse at gmail.com Thu May 2 11:24:37 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Thu, 2 May 2019 11:24:37 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: I've successfully automated the build of the embedded zip on Windows for 3.6 on my internal CI using tools/msi script. Moving onto linux, can you please summarize the primary challenges/differences for creating the embeddable zip for linux? Gerald R. Wiltse jerrywiltse at gmail.com On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse wrote: > Yes, I'll help with this, and I try to get signed up with an account for > the issue tracker ASAP. > > I've just started trying to build create the embeddable package for > windows locally (all from source). Once I have that automated, I'll take a > naive stab at the Linux layout. I'm sure i'll soon hit the additional > complications you're talking about and come back to you with questions. > > Gerald R. Wiltse > jerrywiltse at gmail.com > > > > On Wed, May 1, 2019 at 1:57 PM Steve Dower wrote: > >> On 01May2019 1305, Gerald Wiltse wrote: >> > That does help, thanks so much for the fast response! I now see how >> > it's executed on Azure pipelines also, so that should be enough for me >> > to reverse engineer the rest. >> > >> > Followup questions if you have time: >> > 1. It seems that directory doesn't exist in the 3.6 branches/tags, so >> it >> > must have been done a different way back then. Do you think it's >> > back-portable to 3.6? >> >> Yep, it should work with 3.6. There were scripts under Tools/msi back >> then that did the same thing, but with the Store app I rewrote it to be >> more flexible. >> >> > 2. One of our objectives is to effectively create a nearly identical >> > embeddable package for linux. Do you think the layout code is generic >> > enough to be modified to create a linux equivalent? >> >> Probably not, it's under the "PC" directory because it is entirely >> CPython specific. >> >> I'd love to be able to have a similar embeddable distro for Linux (and >> macOS), and I'll be investing some of my time in working towards this, >> but it's far more complicated than simply laying out the right files. If >> you're interested in helping with this, then we're in the right place to >> discuss it, but we'll likely have to move to the issue tracker at some >> point once we figure out who else wants/needs to be involved. >> >> > Obviously, there are many alternative methods and documents for >> creating >> > "embeddable" pythons on linux, but the windows package has all the >> > characteristics we want, and of course we'd like our Windows and Linux >> > process build and packaging processes to be as similar as possible, so >> > it seems attractive to try to port this layout script to linux. >> >> "make install" already does the same thing, but the install location >> matters at build time, so it's a little more complex to make it work as >> an embedded runtime. I'm sure there are people who know how to make it >> work though. >> >> Cheers, >> Steve >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcepl at cepl.eu Thu May 2 12:24:28 2019 From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Thu, 02 May 2019 18:24:28 +0200 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> Message-ID: On 2019-05-01, 06:46 GMT, Serhiy Storchaka wrote: > These lines were added for purpose. They are needed for > running tests in separate file as a script. > > $ ./python Lib/unittest/test/testmock/testcallable.py -v > test_attributes (__main__.TestCallable) ... ok Isn't the standard way how to run one module just? $ ./python -mtest -v testmock.testcallable Best, Mat?j -- https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 As long as we are thinking of natural values we must say that the sun looks down on nothing half so good as a household laughing together over a meal, or two friends talking over a pint of beer, or a man alone reading a book that interests him; and that all economies, politics, laws, armies, and institutions, save insofar as they prolong and multiply such scenes, are a mere ploughing the sand and sowing the ocean, a meaningless vanity and vexation of the spirit. Collective activities are, of course, necessary, but this is the end to which they are necessary. -- C.S. Lewis, ?Membership? in ?The Weight of Glory? From jerrywiltse at gmail.com Thu May 2 14:56:19 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Thu, 2 May 2019 14:56:19 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: With minimal patching of tools/msi/make_zip.py, I was able to create an equivalent zip package on Linux. It doesn't work out of the box of course. I had to do two commands My biggest concern is that even if the layout is "correct" on linux, the python runtime might not actually support being run like this on Linux. Perhaps the code for using python36.zip as a landmark and source of modules is only implemented on Windows. Do you know if that is the case? tmp$ export LD_LIBRARY_PATH=$(pwd) tmp$ export PYTHONHOME=$(pwd) tmp$ export PYTHONPATH=$(pwd) tmp$ ./python Fatal Python error: Py_Initialize: Unable to get the locale encoding ModuleNotFoundError: No module named 'encodings' Current thread 0x00007fdde9eb1680 (most recent call first): Aborted (core dumped) Here's the full directory structure, error, and patch I created. Any additional feedback would be great. https://gist.github.com/solvingj/87305e0f70e14d738d6989ba2f41352f Gerald R. Wiltse jerrywiltse at gmail.com On Thu, May 2, 2019 at 11:24 AM Gerald Wiltse wrote: > I've successfully automated the build of the embedded zip on Windows for > 3.6 on my internal CI using tools/msi script. > > Moving onto linux, can you please summarize the primary > challenges/differences for creating the embeddable zip for linux? > > Gerald R. Wiltse > jerrywiltse at gmail.com > > > > On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse > wrote: > >> Yes, I'll help with this, and I try to get signed up with an account for >> the issue tracker ASAP. >> >> I've just started trying to build create the embeddable package for >> windows locally (all from source). Once I have that automated, I'll take a >> naive stab at the Linux layout. I'm sure i'll soon hit the additional >> complications you're talking about and come back to you with questions. >> >> Gerald R. Wiltse >> jerrywiltse at gmail.com >> >> >> >> On Wed, May 1, 2019 at 1:57 PM Steve Dower >> wrote: >> >>> On 01May2019 1305, Gerald Wiltse wrote: >>> > That does help, thanks so much for the fast response! I now see how >>> > it's executed on Azure pipelines also, so that should be enough for me >>> > to reverse engineer the rest. >>> > >>> > Followup questions if you have time: >>> > 1. It seems that directory doesn't exist in the 3.6 branches/tags, so >>> it >>> > must have been done a different way back then. Do you think it's >>> > back-portable to 3.6? >>> >>> Yep, it should work with 3.6. There were scripts under Tools/msi back >>> then that did the same thing, but with the Store app I rewrote it to be >>> more flexible. >>> >>> > 2. One of our objectives is to effectively create a nearly identical >>> > embeddable package for linux. Do you think the layout code is generic >>> > enough to be modified to create a linux equivalent? >>> >>> Probably not, it's under the "PC" directory because it is entirely >>> CPython specific. >>> >>> I'd love to be able to have a similar embeddable distro for Linux (and >>> macOS), and I'll be investing some of my time in working towards this, >>> but it's far more complicated than simply laying out the right files. If >>> you're interested in helping with this, then we're in the right place to >>> discuss it, but we'll likely have to move to the issue tracker at some >>> point once we figure out who else wants/needs to be involved. >>> >>> > Obviously, there are many alternative methods and documents for >>> creating >>> > "embeddable" pythons on linux, but the windows package has all the >>> > characteristics we want, and of course we'd like our Windows and Linux >>> > process build and packaging processes to be as similar as possible, so >>> > it seems attractive to try to port this layout script to linux. >>> >>> "make install" already does the same thing, but the install location >>> matters at build time, so it's a little more complex to make it work as >>> an embedded runtime. I'm sure there are people who know how to make it >>> work though. >>> >>> Cheers, >>> Steve >>> >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Thu May 2 15:59:02 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 2 May 2019 15:59:02 -0400 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version Message-ID: Hi, Thanks to Steve Dower's feedback, I enhanced and completed my PEP 587. Main changes: * It is now possible to read the configuration and then modify the read configuration. For example, new directories can be added to PyConfig.module_search_paths (see the example below and the example in the PEP) * PyConfig is now "dynamic" by default: strings are duplicated and PyConfig_Clear() must be called to release memory * PyConfig now only uses wchar_t* for strings (unicode): char* (bytes) is no longer used. I had to hack CPython internals for that :-) * I added a "_config_version" private field to PyPreConfig and PyConfig to prepare the backward compatibility for future changes. * I removed the Open Question section: all known issues have been fixed. During the Language Summit, Brett Cannon said that Steve Dower declined the offer to be the BDFL-delegate for this PEP. Thomas Wouters proposed himself to be the new BDFL-delegate. Example to read the configuration, append a directory to sys.path (module_search_paths) and then initialize Python with this configuration: void init_python(void) { PyInitError err; PyConfig config = PyConfig_INIT; err = PyConfig_Read(&config); if (_Py_INIT_FAILED(err)) { goto fail; } err = PyWideStringList_Append(&config.module_search_paths, L"/path/to/more/modules"); if (_Py_INIT_FAILED(err)) { goto fail; } err = Py_InitializeFromConfig(&config); if (_Py_INIT_FAILED(err)) { goto fail; } PyConfig_Clear(&config); return; fail: PyConfig_Clear(&config); Py_ExitInitError(err); } The HTML version will be online shortly: https://www.python.org/dev/peps/pep-0587/ Full text below. Victor PEP: 587 Title: Python Initialization Configuration Author: Nick Coghlan , Victor Stinner Discussions-To: python-dev at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Mar-2019 Python-Version: 3.8 Abstract ======== Add a new C API to configure the Python Initialization providing finer control on the whole configuration and better error reporting. Rationale ========= Python is highly configurable but its configuration evolved organically: configuration parameters is scattered all around the code using different ways to set them (mostly global configuration variables and functions). A straightforward and reliable way to configure Python is needed. Some configuration parameters are not accessible from the C API, or not easily. The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as input whereas the Python filesystem encoding is set during the initialization. This PEP is a partial implementation of PEP 432 which is the overall design. New fields can be added later to ``PyConfig`` structure to finish the implementation of the PEP 432 (add a new partial initialization which allows to configure Python using Python objects to finish the full initialization). Python Initialization C API =========================== This PEP proposes to add the following new structures, functions and macros. New structures (4): * ``PyConfig`` * ``PyInitError`` * ``PyPreConfig`` * ``PyWideStringList`` New functions (16): * ``Py_PreInitialize(config)`` * ``Py_PreInitializeFromArgs(config, argc, argv)`` * ``Py_PreInitializeFromWideArgs(config, argc, argv)`` * ``PyWideStringList_Append(list, item)`` * ``PyConfig_DecodeLocale(config_str, str)`` * ``PyConfig_SetString(config_str, str)`` * ``PyConfig_Read(config)`` * ``PyConfig_SetArgv(config, argc, argv)`` * ``PyConfig_SetWideArgv(config, argc, argv)`` * ``PyConfig_Clear(config)`` * ``Py_InitializeFromConfig(config)`` * ``Py_InitializeFromArgs(config, argc, argv)`` * ``Py_InitializeFromWideArgs(config, argc, argv)`` * ``Py_UnixMain(argc, argv)`` * ``Py_ExitInitError(err)`` * ``Py_RunMain()`` New macros (9): * ``PyPreConfig_INIT`` * ``PyConfig_INIT`` * ``Py_INIT_OK()`` * ``Py_INIT_ERR(MSG)`` * ``Py_INIT_NO_MEMORY()`` * ``Py_INIT_EXIT(EXITCODE)`` * ``Py_INIT_IS_ERROR(err)`` * ``Py_INIT_IS_EXIT(err)`` * ``Py_INIT_FAILED(err)`` This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type) and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these internal structures. ``PyInterpreterState.config`` becomes the new reference configuration, replacing global configuration variables and other private variables. PyWideStringList ---------------- ``PyWideStringList`` is a list of ``wchar_t*`` strings. Example to initialize a string from C static array:: static wchar_t* argv[2] = { L"-c", L"pass", }; PyWideStringList config_argv = PyWideStringList_INIT; config_argv.length = Py_ARRAY_LENGTH(argv); config_argv.items = argv; ``PyWideStringList`` structure fields: * ``length`` (``Py_ssize_t``) * ``items`` (``wchar_t**``) Methods: * ``PyInitError PyWideStringList_Append(PyWideStringList *list, const wchar_t *item)``: Append *item* to *list*. If *length* is non-zero, *items* must be non-NULL and all strings must be non-NULL. PyInitError ----------- ``PyInitError`` is a structure to store an error message or an exit code for the Python Initialization. For an error, it stores the C function name which created the error. Example:: PyInitError alloc(void **ptr, size_t size) { *ptr = PyMem_RawMalloc(size); if (*ptr == NULL) { return Py_INIT_NO_MEMORY(); } return Py_INIT_OK(); } int main(int argc, char **argv) { void *ptr; PyInitError err = alloc(&ptr, 16); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } PyMem_Free(ptr); return 0; } ``PyInitError`` fields: * ``exitcode`` (``int``): argument passed to ``exit()`` on Unix and to ``ExitProcess()`` on Windows. Only set by ``Py_INIT_EXIT()``. * ``err_msg`` (``const char*``): error message * private ``_func`` field: used by ``Py_INIT_ERR()`` to store the C function name which created the error. * private ``_type`` field: for internal usage only. Macro to create an error: * ``Py_INIT_OK()``: success * ``Py_INIT_ERR(err_msg)``: initialization error with a message * ``Py_INIT_NO_MEMORY()``: memory allocation failure (out of memory) * ``Py_INIT_EXIT(exitcode)``: exit Python with the specified exit code Other macros and functions: * ``Py_INIT_IS_ERROR(err)``: Is the result an error? * ``Py_INIT_IS_EXIT(err)``: Is the result an exit? * ``Py_INIT_FAILED(err)``: Is the result an error or an exit? Similar to ``Py_INIT_IS_ERROR(err) || Py_INIT_IS_EXIT(err)``. * ``Py_ExitInitError(err)``: Call ``exit(exitcode)`` on Unix or ``ExitProcess(exitcode)`` if the result is an exit, call ``Py_FatalError(err_msg)`` if the result is an error. Must not be called if the result is a success. Pre-Initialization with PyPreConfig ----------------------------------- ``PyPreConfig`` structure is used to pre-initialize Python: * Set the memory allocator * Configure the LC_CTYPE locale * Set the UTF-8 mode Example using the pre-initialization to enable the UTF-8 Mode:: PyPreConfig preconfig = PyPreConfig_INIT; preconfig.utf8_mode = 1; PyInitError err = Py_PreInitialize(&preconfig); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* at this point, Python will speak UTF-8 */ Py_Initialize(); /* ... use Python API here ... */ Py_Finalize(); Functions to pre-initialize Python: * ``PyInitError Py_PreInitialize(const PyPreConfig *config)`` * ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *config, int argc, char **argv)`` * ``PyInitError Py_PreInitializeFromWideArgs(const PyPreConfig *config, int argc, wchar_t **argv)`` If Python should be pre-initialized explicitly first and then initialized with command line arguments, it is possible to pass these command line arguments to the pre-initialization since they impact the encodings. For example, ``-X utf8`` enables the UTF-8 Mode. These functions can be called with *config* set to ``NULL``. The caller is responsible to handle error using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. ``PyPreConfig`` fields: * ``allocator`` (``char*``): name of the memory allocator (ex: ``"malloc"``) * ``coerce_c_locale_warn`` (``int``): if non-zero, emit a warning if the C locale is coerced. * ``coerce_c_locale`` (``int``): if equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE to decide if it should be coerced. * ``dev_mode`` (``int``): see ``PyConfig.dev_mode`` * ``isolated`` (``int``): see ``PyConfig.isolated`` * ``legacy_windows_fs_encoding`` (``int``, Windows only): if non-zero, set the Python filesystem encoding to ``"mbcs"``. * ``use_environment`` (``int``): see ``PyConfig.use_environment`` * ``utf8_mode`` (``int``): if non-zero, enable the UTF-8 mode There is also a private field which is for internal-usage only: * ``_config_version`` (``int``): Configuration version, used for ABI compatibility The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``, ``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python enable them depending on the user configuration. Initialization with PyConfig ---------------------------- The ``PyConfig`` structure contains all parameters to configure Python. Example:: PyInitError err; PyConfig config = PyConfig_INIT; err = PyConfig_SetString(&config.program_name, L"my_program"); if (_Py_INIT_FAILED(err)) { Py_ExitInitError(err); } err = Py_InitializeFromConfig(&config); PyConfig_Clear(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } ``PyConfig`` methods: * ``PyInitError PyConfig_SetString(wchar_t **config_str, const wchar_t *str)``: Set a config wide string field from *str* (copy the string) * ``PyInitError PyConfig_DecodeLocale(wchar_t **config_str, const char *str)``: Decode *str* using ``Py_DecodeLocale()`` and set the result into ``*config_str``. Pre-initialize Python if needed to ensure that encodings are properly configured. * ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, char **argv)``: Set command line arguments (decode bytes). Pre-initialize Python if needed to ensure that encodings are properly configured. * ``PyInitError PyConfig_SetWideArgv(PyConfig *config, int argc, wchar_t **argv)``: Set command line arguments (wide characters). * ``PyInitError PyConfig_Read(PyConfig *config)``: Read all Python configuration * ``void PyConfig_Clear(PyConfig *config)``: Release memory Functions to initialize Python: * ``PyInitError Py_InitializeFromConfig(const PyConfig *config)`` These functions can be called with *config* set to ``NULL``. The caller is responsible to handler error using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. PyConfig fields: * ``argv`` (``PyWideStringList``): ``sys.argv`` * ``base_exec_prefix`` (``wchar_t*``): ``sys.base_exec_prefix`` * ``base_prefix`` (``wchar_t*``): ``sys.base_prefix`` * ``buffered_stdio`` (``int``): if equals to 0, enable unbuffered mode, make stdout and stderr streams to be unbuffered. * ``bytes_warning`` (``int``): if equals to 1, issue a warning when comparing ``bytes`` or ``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If equal or greater to 2, raise a ``BytesWarning`` exception. * ``check_hash_pycs_mode`` (``wchar_t*``): ``--check-hash-based-pycs`` command line option value (see PEP 552) * ``dev_mode`` (``int``): Development mode * ``dll_path`` (``wchar_t*``, Windows only): Windows DLL path * ``dump_refs`` (``int``): if non-zero, display all objects still alive at exit * ``exec_prefix`` (``wchar_t*``): ``sys.exec_prefix`` * ``executable`` (``wchar_t*``): ``sys.executable`` * ``faulthandler`` (``int``): if non-zero, call ``faulthandler.enable()`` * ``filesystem_encoding`` (``wchar_t*``): Filesystem encoding, ``sys.getfilesystemencoding()`` * ``filesystem_errors`` (``wchar_t*``): Filesystem encoding errors, ``sys.getfilesystemencodeerrors()`` * ``use_hash_seed`` (``int``), ``hash_seed`` (``unsigned long``): randomized hash function seed * ``home`` (``wchar_t*``): Python home * ``import_time`` (``int``): if non-zero, profile import time * ``inspect`` (``int``): enter interactive mode after executing a script or a command * ``install_signal_handlers`` (``int``): install signal handlers? * ``interactive`` (``int``): interactive mode * ``legacy_windows_stdio`` (``int``, Windows only): if non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. * ``malloc_stats`` (``int``): if non-zero, dump memory allocation statistics at exit * ``module_search_path_env`` (``wchar_t*``): ``PYTHONPATH`` environment variale value * ``use_module_search_paths`` (``int``), ``module_search_paths`` (``PyWideStringList``): ``sys.path`` * ``optimization_level`` (``int``): compilation optimization level * ``parser_debug`` (``int``): if non-zero, turn on parser debugging output (for expert only, depending on compilation options). * ``prefix`` (``wchar_t*``): ``sys.prefix`` * ``program_name`` (``wchar_t*``): Program name * ``program`` (``wchar_t*``): ``argv[0]`` or an empty string * ``pycache_prefix`` (``wchar_t*``): ``.pyc`` cache prefix * ``quiet`` (``int``): quiet mode (ex: don't display the copyright and version messages even in interactive mode) * ``run_command`` (``wchar_t*``): ``-c COMMAND`` argument * ``run_filename`` (``wchar_t*``): ``python3 SCRIPT`` argument * ``run_module`` (``wchar_t*``): ``python3 -m MODULE`` argument * ``show_alloc_count`` (``int``): show allocation counts at exit? * ``show_ref_count`` (``int``): show total reference count at exit? * ``site_import`` (``int``): import the ``site`` module at startup? * ``skip_source_first_line`` (``int``): skip the first line of the source * ``stdio_encoding`` (``wchar_t*``), ``stdio_errors`` (``wchar_t*``): encoding and encoding errors of ``sys.stdin``, ``sys.stdout`` and ``sys.stderr`` * ``tracemalloc`` (``int``): if non-zero, call ``tracemalloc.start(value)`` * ``user_site_directory`` (``int``): if non-zero, add user site directory to ``sys.path`` * ``verbose`` (``int``): if non-zero, enable verbose mode * ``warnoptions`` (``PyWideStringList``): options of the ``warnings`` module to build filters * ``write_bytecode`` (``int``): if non-zero, write ``.pyc`` files * ``xoptions`` (``PyWideStringList``): ``sys._xoptions`` There are also private fields which are for internal-usage only: * ``_config_version`` (``int``): Configuration version, used for ABI compatibility * ``_frozen`` (``int``): Emit warning when computing the path configuration? * ``_install_importlib`` (``int``): Install importlib? More complete commented example modifying the configuration before calling ``PyConfig_Read()`` and then modify the read configuration:: PyInitError init_python(const char *program_name) { PyInitError err; PyConfig config = PyConfig_INIT; /* Set the program name before reading the configuraton (decode byte string from the locale encoding) */ err = PyConfig_DecodeLocale(&config.program_name, program_name); if (_Py_INIT_FAILED(err)) { goto fail; } /* Read all configuration at once */ err = PyConfig_Read(&config); if (_Py_INIT_FAILED(err)) { goto fail; } /* Append our custom search path to sys.path */ err = PyWideStringList_Append(&config.module_search_paths, L"/path/to/more/modules"); if (_Py_INIT_FAILED(err)) { goto fail; } /* Override executable computed by PyConfig_Read() */ err = PyConfig_SetString(&config.executable, L"my_executable"); if (_Py_INIT_FAILED(err)) { goto fail; } err = Py_InitializeFromConfig(&config); /* Py_InitializeFromConfig() copied config which must now be cleared to release memory */ PyConfig_Clear(&config); return err; fail: PyConfig_Clear(&config); Py_ExitInitError(err); } .. note:: ``PyConfig`` does not have any field for extra inittab functions: ``PyImport_AppendInittab()`` and ``PyImport_ExtendInittab()`` functions are still relevant. Initialization with static PyConfig ----------------------------------- When no ``PyConfig`` method is used but only ``Py_InitializeFromConfig()``, the caller is responsible for managing ``PyConfig`` memory which means that static strings and static string lists can be used rather than using dynamically allocated memory. It can be used for most simple configurations. Example of Python initialization enabling the isolated mode:: PyConfig config = PyConfig_INIT; config.isolated = 1; PyInitError err = Py_InitializeFromConfig(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* ... use Python API here ... */ Py_Finalize(); In this example, ``PyConfig_Clear()`` is not needed since ``config`` does not contain any dynamically allocated string: ``Py_InitializeFromConfig`` is responsible for filling other fields and manage the memory. For convenience, two other functions are provided: * ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int argc, char **argv)`` * ``PyInitError Py_InitializeFromWideArgs(const PyConfig *config, int argc, wchar_t **argv)`` These functions can be used with static ``PyConfig``. Pseudo-code of ``Py_InitializeFromArgs()``:: PyInitError init_with_args(const PyConfig *src_config, int argc, char **argv) { PyInitError err; PyConfig config = PyConfig_INIT; /* Copy strings and string lists * (memory dynamically allocated on the heap) */ err = _PyConfig_Copy(&config, src_config); if (Py_INIT_FAILED(err)) { goto exit; } /* Set config.argv: decode argv bytes. Pre-initialize Python if needed to ensure that the encodings are properly configured. */ err = PyConfig_SetArgv(&config, argc, argv); if (Py_INIT_FAILED(err)) { goto exit; } err = Py_InitializeFromConfig(&config); exit: PyConfig_Clear(&config); return err; } where ``_PyConfig_Copy()`` is an internal function. The actual implementation of ``Py_InitializeFromArgs()`` is more complex. Py_UnixMain() ------------- Python 3.7 provides a high-level ``Py_Main()`` function which requires to pass command line arguments as ``wchar_t*`` strings. It is non-trivial to use the correct encoding to decode bytes. Python has its own set of issues with C locale coercion and UTF-8 Mode. This PEP adds a new ``Py_UnixMain()`` function which takes command line arguments as bytes:: int Py_UnixMain(int argc, char **argv) Py_RunMain() ------------ The new ``Py_RunMain()`` function executes the command (``PyConfig.run_command``), the script (``PyConfig.run_filename``) or the module (``PyConfig.run_module``) specified on the command line or in the configuration, and then finalizes Python. It returns an exit status that can be passed to the ``exit()`` function. Example of custom Python executable always running in isolated mode:: #include int main(int argc, char *argv[]) { PyConfig config = PyConfig_INIT; config.isolated = 1; PyInitError err = Py_InitializeFromArgs(&config, argc, argv); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* put more configuration code here if needed */ return Py_RunMain(); } The example is a basic implementation of the "System Python Executable" discussed in PEP 432. Memory allocations and Py_DecodeLocale() ---------------------------------------- Python memory allocation functions like ``PyMem_RawMalloc()`` must not be used before Python pre-initialization. Calling directly ``malloc()`` and ``free()`` is always safe. For ``PyPreConfig`` and static ``PyConfig``, the caller is responsible to manage dynamically allocated strings, but static strings and static string lists are fine. Dynamic ``PyConfig`` requires to call ``PyConfig_Clear()`` to release memory. ``Py_DecodeLocale()`` must not be called before the pre-initialization. When using dynanic configuration, ``PyConfig_DecodeLocale()`` must be used instead of ``Py_DecodeLocale()``. Backwards Compatibility ======================= This PEP only adds a new API: it leaves the existing API unchanged and has no impact on the backwards compatibility. Annex: Python Configuration =========================== Priority and Rules ------------------ Priority of configuration parameters, highest to lowest: * ``PyConfig`` * ``PyPreConfig`` * Configuration files * Command line options * Environment variables * Global configuration variables Priority of warning options, highest to lowest: * ``PyConfig.warnoptions`` * ``PyConfig.dev_mode`` (add ``"default"``) * ``PYTHONWARNINGS`` environment variables * ``-W WARNOPTION`` command line argument * ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater than 1, or add ``"default::BytesWarning``) Rules on ``PyConfig`` and ``PyPreConfig`` parameters: * If ``isolated`` is non-zero, ``use_environment`` and ``user_site_directory`` are set to 0 * If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to 0 * If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``, ``faulthandler`` is set to 1, and ``"default"`` filter is added to ``warnoptions``. But ``PYTHONMALLOC`` has the priority over ``dev_mode`` to set the memory allocator. Configuration Files ------------------- Python configuration files: * ``pyvenv.cfg`` * ``python._pth`` (Windows only) * ``pybuilddir.txt`` (Unix only) Global Configuration Variables ------------------------------ Global configuration variables mapped to ``PyPreConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` ``Py_UTF8Mode`` ``utf8_mode`` ``Py_UTF8Mode`` ``utf8_mode`` ======================================== ================================ Global configuration variables mapped to ``PyConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_BytesWarningFlag`` ``bytes_warning`` ``Py_DebugFlag`` ``parser_debug`` ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_FrozenFlag`` ``_frozen`` ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` ``Py_IgnoreEnvironmentFlag`` ``use_environment`` ``Py_InspectFlag`` ``inspect`` ``Py_InteractiveFlag`` ``interactive`` ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` ``Py_NoSiteFlag`` ``site_import`` ``Py_NoUserSiteDirectory`` ``user_site_directory`` ``Py_OptimizeFlag`` ``optimization_level`` ``Py_QuietFlag`` ``quiet`` ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` ``Py_VerboseFlag`` ``verbose`` ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_BytesWarningFlag`` ``bytes_warning`` ``Py_DebugFlag`` ``parser_debug`` ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_FrozenFlag`` ``_frozen`` ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` ``Py_IgnoreEnvironmentFlag`` ``use_environment`` ``Py_InspectFlag`` ``inspect`` ``Py_InteractiveFlag`` ``interactive`` ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` ``Py_NoSiteFlag`` ``site_import`` ``Py_NoUserSiteDirectory`` ``user_site_directory`` ``Py_OptimizeFlag`` ``optimization_level`` ``Py_QuietFlag`` ``quiet`` ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` ``Py_VerboseFlag`` ``verbose`` ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` ======================================== ================================ ``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are only available on Windows. Command Line Arguments ---------------------- Usage:: python3 [options] python3 [options] -c COMMAND python3 [options] -m MODULE python3 [options] SCRIPT Command line options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyPreConfig`` field ================================ ================================ ``-X dev`` ``dev_mode = 1`` ``-X utf8=N`` ``utf8_mode = N`` ================================ ================================ Command line options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-b`` ``bytes_warning++`` ``-B`` ``write_bytecode = 0`` ``-c COMMAND`` ``run_module = COMMAND`` ``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE`` ``-d`` ``parser_debug++`` ``-E`` ``use_environment = 0`` ``-i`` ``inspect++`` and ``interactive++`` ``-I`` ``isolated = 1`` ``-m MODULE`` ``run_module = MODULE`` ``-O`` ``optimization_level++`` ``-q`` ``quiet++`` ``-R`` ``use_hash_seed = 0`` ``-s`` ``user_site_directory = 0`` ``-S`` ``site_import`` ``-t`` ignored (kept for backwards compatibility) ``-u`` ``buffered_stdio = 0`` ``-v`` ``verbose++`` ``-W WARNING`` add ``WARNING`` to ``warnoptions`` ``-x`` ``skip_source_first_line = 1`` ``-X XOPTION`` add ``XOPTION`` to ``xoptions`` ``-X dev`` ``dev_mode = 1`` ``-X faulthandler`` ``faulthandler = 1`` ``-X importtime`` ``import_time = 1`` ``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX`` ``-X show_alloc_count`` ``show_alloc_count = 1`` ``-X show_ref_count`` ``show_ref_count = 1`` ``-X tracemalloc=N`` ``tracemalloc = N`` ================================ ================================ ``-h``, ``-?`` and ``-V`` options are handled outside ``PyConfig``. Environment Variables --------------------- Environment variables mapped to ``PyPreConfig`` fields: ================================= ============================================= Variable ``PyPreConfig`` field ================================= ============================================= ``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding`` ``PYTHONMALLOC`` ``allocator`` ``PYTHONUTF8`` ``utf8_mode`` ================================= ============================================= Environment variables mapped to ``PyConfig`` fields: ================================= ==================================== Variable ``PyConfig`` field ================================= ==================================== ``PYTHONDEBUG`` ``parser_debug`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONDONTWRITEBYTECODE`` ``write_bytecode`` ``PYTHONDUMPREFS`` ``dump_refs`` ``PYTHONEXECUTABLE`` ``program_name`` ``PYTHONFAULTHANDLER`` ``faulthandler`` ``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed`` ``PYTHONHOME`` ``home`` ``PYTHONINSPECT`` ``inspect`` ``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors`` ``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio`` ``PYTHONMALLOCSTATS`` ``malloc_stats`` ``PYTHONNOUSERSITE`` ``user_site_directory`` ``PYTHONOPTIMIZE`` ``optimization_level`` ``PYTHONPATH`` ``module_search_path_env`` ``PYTHONPROFILEIMPORTTIME`` ``import_time`` ``PYTHONPYCACHEPREFIX,`` ``pycache_prefix`` ``PYTHONTRACEMALLOC`` ``tracemalloc`` ``PYTHONUNBUFFERED`` ``buffered_stdio`` ``PYTHONVERBOSE`` ``verbose`` ``PYTHONWARNINGS`` ``warnoptions`` ================================= ==================================== ``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are specific to Windows. ``PYTHONDEVMODE`` is mapped to ``PyPreConfig.dev_mode`` and ``PyConfig.dev_mode``. Annex: Python 3.7 API ===================== Python 3.7 has 4 functions in its C API to initialize and finalize Python: * ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python * ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python Python can be configured using scattered global configuration variables (like ``Py_IgnoreEnvironmentFlag``) and using the following functions: * ``PyImport_AppendInittab()`` * ``PyImport_ExtendInittab()`` * ``PyMem_SetAllocator()`` * ``PyMem_SetupDebugHooks()`` * ``PyObject_SetArenaAllocator()`` * ``Py_SetPath()`` * ``Py_SetProgramName()`` * ``Py_SetPythonHome()`` * ``Py_SetStandardStreamEncoding()`` * ``PySys_AddWarnOption()`` * ``PySys_AddXOption()`` * ``PySys_ResetWarnOptions()`` There is also a high-level ``Py_Main()`` function. Copyright ========= This document has been placed in the public domain. From edwin at 211mainstreet.net Thu May 2 16:20:10 2019 From: edwin at 211mainstreet.net (Edwin Zimmerman) Date: Thu, 2 May 2019 16:20:10 -0400 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: References: Message-ID: <000001d50124$71ef0d40$55cd27c0$@211mainstreet.net> On Thursday, May 02, 2019 Victor Stinner wrote: > According to this > * ``run_command`` (``wchar_t*``): ``-c COMMAND`` argument > * ``run_filename`` (``wchar_t*``): ``python3 SCRIPT`` argument > * ``run_module`` (``wchar_t*``): ``python3 -m MODULE`` argument this > ``-c COMMAND`` ``run_module = COMMAND`` should read "run_command = COMMAND". Typo, not? From jerrywiltse at gmail.com Thu May 2 16:29:36 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Thu, 2 May 2019 16:29:36 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: Just an update based on testing. If I zip the modules up into python36.zip (whether in .py or .pyc form) the linux binary does not find them. If I copy them into the embedded root, alongside the python executable, it finds them as expected. This just makes it seem more like an issue where Linux binary does not respect .zip file as a source of modules. Gerald R. Wiltse jerrywiltse at gmail.com On Thu, May 2, 2019 at 2:56 PM Gerald Wiltse wrote: > With minimal patching of tools/msi/make_zip.py, I was able to create an > equivalent zip package on Linux. It doesn't work out of the box of > course. I had to do two commands > > My biggest concern is that even if the layout is "correct" on linux, the > python runtime might not actually support being run like this on Linux. > Perhaps the code for using python36.zip as a landmark and source of modules > is only implemented on Windows. Do you know if that is the case? > > tmp$ export LD_LIBRARY_PATH=$(pwd) > tmp$ export PYTHONHOME=$(pwd) > tmp$ export PYTHONPATH=$(pwd) > tmp$ ./python > Fatal Python error: Py_Initialize: Unable to get the locale encoding > ModuleNotFoundError: No module named 'encodings' > > Current thread 0x00007fdde9eb1680 (most recent call first): > Aborted (core dumped) > > Here's the full directory structure, error, and patch I created. Any > additional feedback would be great. > > https://gist.github.com/solvingj/87305e0f70e14d738d6989ba2f41352f > > Gerald R. Wiltse > jerrywiltse at gmail.com > > > > On Thu, May 2, 2019 at 11:24 AM Gerald Wiltse > wrote: > >> I've successfully automated the build of the embedded zip on Windows for >> 3.6 on my internal CI using tools/msi script. >> >> Moving onto linux, can you please summarize the primary >> challenges/differences for creating the embeddable zip for linux? >> >> Gerald R. Wiltse >> jerrywiltse at gmail.com >> >> >> >> On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse >> wrote: >> >>> Yes, I'll help with this, and I try to get signed up with an account for >>> the issue tracker ASAP. >>> >>> I've just started trying to build create the embeddable package for >>> windows locally (all from source). Once I have that automated, I'll take a >>> naive stab at the Linux layout. I'm sure i'll soon hit the additional >>> complications you're talking about and come back to you with questions. >>> >>> Gerald R. Wiltse >>> jerrywiltse at gmail.com >>> >>> >>> >>> On Wed, May 1, 2019 at 1:57 PM Steve Dower >>> wrote: >>> >>>> On 01May2019 1305, Gerald Wiltse wrote: >>>> > That does help, thanks so much for the fast response! I now see how >>>> > it's executed on Azure pipelines also, so that should be enough for >>>> me >>>> > to reverse engineer the rest. >>>> > >>>> > Followup questions if you have time: >>>> > 1. It seems that directory doesn't exist in the 3.6 branches/tags, so >>>> it >>>> > must have been done a different way back then. Do you think it's >>>> > back-portable to 3.6? >>>> >>>> Yep, it should work with 3.6. There were scripts under Tools/msi back >>>> then that did the same thing, but with the Store app I rewrote it to be >>>> more flexible. >>>> >>>> > 2. One of our objectives is to effectively create a nearly identical >>>> > embeddable package for linux. Do you think the layout code is generic >>>> > enough to be modified to create a linux equivalent? >>>> >>>> Probably not, it's under the "PC" directory because it is entirely >>>> CPython specific. >>>> >>>> I'd love to be able to have a similar embeddable distro for Linux (and >>>> macOS), and I'll be investing some of my time in working towards this, >>>> but it's far more complicated than simply laying out the right files. >>>> If >>>> you're interested in helping with this, then we're in the right place >>>> to >>>> discuss it, but we'll likely have to move to the issue tracker at some >>>> point once we figure out who else wants/needs to be involved. >>>> >>>> > Obviously, there are many alternative methods and documents for >>>> creating >>>> > "embeddable" pythons on linux, but the windows package has all the >>>> > characteristics we want, and of course we'd like our Windows and >>>> Linux >>>> > process build and packaging processes to be as similar as possible, >>>> so >>>> > it seems attractive to try to port this layout script to linux. >>>> >>>> "make install" already does the same thing, but the install location >>>> matters at build time, so it's a little more complex to make it work as >>>> an embedded runtime. I'm sure there are people who know how to make it >>>> work though. >>>> >>>> Cheers, >>>> Steve >>>> >>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Thu May 2 16:47:09 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 2 May 2019 16:47:09 -0400 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: <000001d50124$71ef0d40$55cd27c0$@211mainstreet.net> References: <000001d50124$71ef0d40$55cd27c0$@211mainstreet.net> Message-ID: Le jeu. 2 mai 2019 ? 16:20, Edwin Zimmerman a ?crit : > > ``-c COMMAND`` ``run_module = COMMAND`` > should read "run_command = COMMAND". Typo, not? Oops, you're right: it's a typo. Now fixed: ``-c COMMAND`` ``run_command = COMMAND`` Victor -- Night gathers, and now my watch begins. It shall not end until my death. From thatiparthysreenivas at gmail.com Fri May 3 04:46:35 2019 From: thatiparthysreenivas at gmail.com (Srinivas Reddy Thatiparthy) Date: Fri, 3 May 2019 14:16:35 +0530 Subject: [Python-Dev] Please merge : bpo-34848 Message-ID: Hi, The PR - https://github.com/python/cpython/pull/9877 has been approved by Raymond long ago. Please merge or comment if not mergeable. Best Regards, Srinivas Reddy Thatipathy 7768044110. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Fri May 3 07:21:50 2019 From: steve at holdenweb.com (Steve Holden) Date: Fri, 3 May 2019 12:21:50 +0100 Subject: [Python-Dev] Please merge : bpo-34848 In-Reply-To: References: Message-ID: Looks like the failure is due to absence of a News entry. Maybe add a "skip news" label if this doesn't need to be documented? Kind regards. Steve Holden On Fri, May 3, 2019 at 9:48 AM Srinivas Reddy Thatiparthy < thatiparthysreenivas at gmail.com> wrote: > Hi, > The PR - https://github.com/python/cpython/pull/9877 has been > approved by Raymond long ago. Please merge or comment if not mergeable. > > Best Regards, > Srinivas Reddy Thatipathy > 7768044110. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From thatiparthysreenivas at gmail.com Fri May 3 08:14:50 2019 From: thatiparthysreenivas at gmail.com (Srinivas Reddy Thatiparthy) Date: Fri, 3 May 2019 17:44:50 +0530 Subject: [Python-Dev] Please merge : bpo-34848 In-Reply-To: References: Message-ID: Yes, I agree. Please do as I do not have permission. Best Regards, Srinivas Reddy Thatipathy On Fri, 3 May 2019 at 16:52, Steve Holden wrote: > Looks like the failure is due to absence of a News entry. Maybe add a > "skip news" label if this doesn't need to be documented? > > Kind regards. > Steve Holden > > > On Fri, May 3, 2019 at 9:48 AM Srinivas Reddy Thatiparthy < > thatiparthysreenivas at gmail.com> wrote: > >> Hi, >> The PR - https://github.com/python/cpython/pull/9877 has been >> approved by Raymond long ago. Please merge or comment if not mergeable. >> >> Best Regards, >> Srinivas Reddy Thatipathy >> 7768044110. >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Fri May 3 08:24:15 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 3 May 2019 08:24:15 -0400 Subject: [Python-Dev] Please merge : bpo-34848 In-Reply-To: References: Message-ID: Hi Srinivas, I merged your obvious doc fix, thanks. Does someone know how I can list pull requests which has been approved but not merged yet? Victor Le vendredi 3 mai 2019, Srinivas Reddy Thatiparthy < thatiparthysreenivas at gmail.com> a ?crit : > Hi, > The PR - https://github.com/python/cpython/pull/9877 has been approved by Raymond long ago. Please merge or comment if not mergeable. > Best Regards, > Srinivas Reddy Thatipathy > 7768044110. > > -- Night gathers, and now my watch begins. It shall not end until my death. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mhroncok at redhat.com Fri May 3 08:41:17 2019 From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=) Date: Fri, 3 May 2019 14:41:17 +0200 Subject: [Python-Dev] Please merge : bpo-34848 In-Reply-To: References: Message-ID: <10403b62-6025-30ae-74f5-ca096e275b27@redhat.com> On 03. 05. 19 14:24, Victor Stinner wrote: > Does someone know how I can list pull requests which has been approved but not > merged yet? Add review:approved to the search filed. https://github.com/python/cpython/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved Or click on **Reviews** in the top bar and select what you like. -- Miro Hron?ok -- Phone: +420777974800 IRC: mhroncok From bcannon at gmail.com Fri May 3 11:01:49 2019 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 3 May 2019 11:01:49 -0400 Subject: [Python-Dev] Please merge : bpo-34848 In-Reply-To: <10403b62-6025-30ae-74f5-ca096e275b27@redhat.com> References: <10403b62-6025-30ae-74f5-ca096e275b27@redhat.com> Message-ID: The "awaiting merge" was originally created to help with this. On Fri., May 3, 2019, 08:51 Miro Hron?ok, wrote: > On 03. 05. 19 14:24, Victor Stinner wrote: > > Does someone know how I can list pull requests which has been approved > but not > > merged yet? > > Add review:approved to the search filed. > > > https://github.com/python/cpython/pulls?q=is%3Apr+is%3Aopen+review%3Aapproved > > Or click on **Reviews** in the top bar and select what you like. > > -- > Miro Hron?ok > -- > Phone: +420777974800 > IRC: mhroncok > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Fri May 3 11:38:56 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Fri, 3 May 2019 17:38:56 +0200 Subject: [Python-Dev] PEP 580/590 proposals summary Message-ID: <5CCC6090.6010202@UGent.be> Hello all, If we want to have a chance of implementing PEP 580/590 in Python 3.8, we shouldn't wait too long to make a decision on which proposal to accept. As a summary, below I'll write the four proposals together with a star "score" for 3 criteria (there is no obvious best proposal, all have advantages and disadvantages): - complexity: more stars is a protocol which is simpler to document and understand. - implementation: more stars is a simpler implementation of CPython (not just of the protocol itself, but also the code using the protocol). - performance: more stars is better performance for *existing* code. I'm using a minimum of 3 stars here, since the difference is not that big between the proposals. Criteria that I am NOT considering: - The performance for *new* code or the performance of wrappers generated by Argument Clinic: all proposals score excellent here. - Complexity of implementations of external classes: this is hard to judge, since that depends a lot on what those external classes (outside of CPython) want to do. - The work to implement the proposal in CPython: this is a one-time only thing that I'm volunteering to do anyway. - Extensibility of the protocol: first of all, it's hard to define what this means exactly. Second, using Petr's idea of putting the flags inside the instance, every proposal becomes extensible at little cost. Proposals: (A) PEP 580 complexity: * implementation: ***** performance: ***** (B) compromise: PEP 580 with a single calling convention complexity: *** implementation: **** performance: **** (C) PEP 590 with a single bound method class complexity: ***** implementation: *** performance: *** (D) PEP 590 complexity: ***** implementation: * performance: **** I consider Petr's proposal (a more extensible variant of PEP 590 with flags inside the instance) a minor variation of PEP 590 for this purpose and no need to score it differently than "plain" PEP 590. I tried to do this as unbiased as possible, even though I must admit that this is not really possible. I'm considering not just the PEP and the existing implementation as written, but also ideas that haven't been implemented yet such as: - proposals (A)-(C): rationalization of classes, in particular having a single class for bound methods (just like in PyPy). - proposals (B)-(D): Mark Shannon's idea of having a dedicated vectorcall wrapper for each calling convention (one for METH_O, one for METH_VARARGS|METH_KEYWORDS, ...). - using the protocol also for slot wrappers like object.__eq__ I'm NOT considering Petr's proposal of removing support for other calling conventions like METH_VARARGS because that won't happen any time soon. Cheers, Jeroen From J.Demeyer at UGent.be Fri May 3 11:40:55 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Fri, 3 May 2019 17:40:55 +0200 Subject: [Python-Dev] Please merge : bpo-34848 In-Reply-To: References: Message-ID: <5CCC6107.10900@UGent.be> On 2019-05-03 14:24, Victor Stinner wrote: > Hi Srinivas, > > I merged your obvious doc fix, thanks. Can you please do the same for https://github.com/python/cpython/pull/12784 From jerrywiltse at gmail.com Fri May 3 11:55:12 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Fri, 3 May 2019 11:55:12 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: Ok, disregard previous message. I fixed the PYTHONPATH in my command and the .pyc files and . I also saw that python36._pth is specifically only supported by windows. Now i'm moved on to the following error: zipimport.ZipImportError: can't decompress data; zlib not available I'm on Ubuntu. I've definitely got zlib1g-dev package installed, and I've recompiled a number of times trying to get it to work. I've tried Ubuntu 16 and 18. Is this just a limitation on linux? Gerald R. Wiltse jerrywiltse at gmail.com On Thu, May 2, 2019 at 4:29 PM Gerald Wiltse wrote: > Just an update based on testing. If I zip the modules up into > python36.zip (whether in .py or .pyc form) the linux binary does not find > them. If I copy them into the embedded root, alongside the python > executable, it finds them as expected. > > This just makes it seem more like an issue where Linux binary does not > respect .zip file as a source of modules. > > Gerald R. Wiltse > jerrywiltse at gmail.com > > > > On Thu, May 2, 2019 at 2:56 PM Gerald Wiltse > wrote: > >> With minimal patching of tools/msi/make_zip.py, I was able to create an >> equivalent zip package on Linux. It doesn't work out of the box of >> course. I had to do two commands >> >> My biggest concern is that even if the layout is "correct" on linux, the >> python runtime might not actually support being run like this on Linux. >> Perhaps the code for using python36.zip as a landmark and source of modules >> is only implemented on Windows. Do you know if that is the case? >> >> tmp$ export LD_LIBRARY_PATH=$(pwd) >> tmp$ export PYTHONHOME=$(pwd) >> tmp$ export PYTHONPATH=$(pwd) >> tmp$ ./python >> Fatal Python error: Py_Initialize: Unable to get the locale encoding >> ModuleNotFoundError: No module named 'encodings' >> >> Current thread 0x00007fdde9eb1680 (most recent call first): >> Aborted (core dumped) >> >> Here's the full directory structure, error, and patch I created. Any >> additional feedback would be great. >> >> https://gist.github.com/solvingj/87305e0f70e14d738d6989ba2f41352f >> >> Gerald R. Wiltse >> jerrywiltse at gmail.com >> >> >> >> On Thu, May 2, 2019 at 11:24 AM Gerald Wiltse >> wrote: >> >>> I've successfully automated the build of the embedded zip on Windows for >>> 3.6 on my internal CI using tools/msi script. >>> >>> Moving onto linux, can you please summarize the primary >>> challenges/differences for creating the embeddable zip for linux? >>> >>> Gerald R. Wiltse >>> jerrywiltse at gmail.com >>> >>> >>> >>> On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse >>> wrote: >>> >>>> Yes, I'll help with this, and I try to get signed up with an account >>>> for the issue tracker ASAP. >>>> >>>> I've just started trying to build create the embeddable package for >>>> windows locally (all from source). Once I have that automated, I'll take a >>>> naive stab at the Linux layout. I'm sure i'll soon hit the additional >>>> complications you're talking about and come back to you with questions. >>>> >>>> Gerald R. Wiltse >>>> jerrywiltse at gmail.com >>>> >>>> >>>> >>>> On Wed, May 1, 2019 at 1:57 PM Steve Dower >>>> wrote: >>>> >>>>> On 01May2019 1305, Gerald Wiltse wrote: >>>>> > That does help, thanks so much for the fast response! I now see how >>>>> > it's executed on Azure pipelines also, so that should be enough for >>>>> me >>>>> > to reverse engineer the rest. >>>>> > >>>>> > Followup questions if you have time: >>>>> > 1. It seems that directory doesn't exist in the 3.6 branches/tags, >>>>> so it >>>>> > must have been done a different way back then. Do you think it's >>>>> > back-portable to 3.6? >>>>> >>>>> Yep, it should work with 3.6. There were scripts under Tools/msi back >>>>> then that did the same thing, but with the Store app I rewrote it to >>>>> be >>>>> more flexible. >>>>> >>>>> > 2. One of our objectives is to effectively create a nearly identical >>>>> > embeddable package for linux. Do you think the layout code is >>>>> generic >>>>> > enough to be modified to create a linux equivalent? >>>>> >>>>> Probably not, it's under the "PC" directory because it is entirely >>>>> CPython specific. >>>>> >>>>> I'd love to be able to have a similar embeddable distro for Linux (and >>>>> macOS), and I'll be investing some of my time in working towards this, >>>>> but it's far more complicated than simply laying out the right files. >>>>> If >>>>> you're interested in helping with this, then we're in the right place >>>>> to >>>>> discuss it, but we'll likely have to move to the issue tracker at some >>>>> point once we figure out who else wants/needs to be involved. >>>>> >>>>> > Obviously, there are many alternative methods and documents for >>>>> creating >>>>> > "embeddable" pythons on linux, but the windows package has all the >>>>> > characteristics we want, and of course we'd like our Windows and >>>>> Linux >>>>> > process build and packaging processes to be as similar as possible, >>>>> so >>>>> > it seems attractive to try to port this layout script to linux. >>>>> >>>>> "make install" already does the same thing, but the install location >>>>> matters at build time, so it's a little more complex to make it work >>>>> as >>>>> an embedded runtime. I'm sure there are people who know how to make it >>>>> work though. >>>>> >>>>> Cheers, >>>>> Steve >>>>> >>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jerrywiltse at gmail.com Fri May 3 13:02:51 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Fri, 3 May 2019 13:02:51 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: I have it working on linux now. Is now a good time to open a ticket on Issue Tracker? Gerald R. Wiltse jerrywiltse at gmail.com On Fri, May 3, 2019 at 11:55 AM Gerald Wiltse wrote: > Ok, disregard previous message. I fixed the PYTHONPATH in my command > and the .pyc files and . I also saw that python36._pth is specifically > only supported by windows. > > Now i'm moved on to the following error: > > zipimport.ZipImportError: can't decompress data; zlib not available > > I'm on Ubuntu. I've definitely got zlib1g-dev package installed, and I've > recompiled a number of times trying to get it to work. I've tried Ubuntu > 16 and 18. > > Is this just a limitation on linux? > > Gerald R. Wiltse > jerrywiltse at gmail.com > > > > On Thu, May 2, 2019 at 4:29 PM Gerald Wiltse > wrote: > >> Just an update based on testing. If I zip the modules up into >> python36.zip (whether in .py or .pyc form) the linux binary does not find >> them. If I copy them into the embedded root, alongside the python >> executable, it finds them as expected. >> >> This just makes it seem more like an issue where Linux binary does not >> respect .zip file as a source of modules. >> >> Gerald R. Wiltse >> jerrywiltse at gmail.com >> >> >> >> On Thu, May 2, 2019 at 2:56 PM Gerald Wiltse >> wrote: >> >>> With minimal patching of tools/msi/make_zip.py, I was able to create an >>> equivalent zip package on Linux. It doesn't work out of the box of >>> course. I had to do two commands >>> >>> My biggest concern is that even if the layout is "correct" on linux, the >>> python runtime might not actually support being run like this on Linux. >>> Perhaps the code for using python36.zip as a landmark and source of modules >>> is only implemented on Windows. Do you know if that is the case? >>> >>> tmp$ export LD_LIBRARY_PATH=$(pwd) >>> tmp$ export PYTHONHOME=$(pwd) >>> tmp$ export PYTHONPATH=$(pwd) >>> tmp$ ./python >>> Fatal Python error: Py_Initialize: Unable to get the locale encoding >>> ModuleNotFoundError: No module named 'encodings' >>> >>> Current thread 0x00007fdde9eb1680 (most recent call first): >>> Aborted (core dumped) >>> >>> Here's the full directory structure, error, and patch I created. Any >>> additional feedback would be great. >>> >>> https://gist.github.com/solvingj/87305e0f70e14d738d6989ba2f41352f >>> >>> Gerald R. Wiltse >>> jerrywiltse at gmail.com >>> >>> >>> >>> On Thu, May 2, 2019 at 11:24 AM Gerald Wiltse >>> wrote: >>> >>>> I've successfully automated the build of the embedded zip on Windows >>>> for 3.6 on my internal CI using tools/msi script. >>>> >>>> Moving onto linux, can you please summarize the primary >>>> challenges/differences for creating the embeddable zip for linux? >>>> >>>> Gerald R. Wiltse >>>> jerrywiltse at gmail.com >>>> >>>> >>>> >>>> On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse >>>> wrote: >>>> >>>>> Yes, I'll help with this, and I try to get signed up with an account >>>>> for the issue tracker ASAP. >>>>> >>>>> I've just started trying to build create the embeddable package for >>>>> windows locally (all from source). Once I have that automated, I'll take a >>>>> naive stab at the Linux layout. I'm sure i'll soon hit the additional >>>>> complications you're talking about and come back to you with questions. >>>>> >>>>> Gerald R. Wiltse >>>>> jerrywiltse at gmail.com >>>>> >>>>> >>>>> >>>>> On Wed, May 1, 2019 at 1:57 PM Steve Dower >>>>> wrote: >>>>> >>>>>> On 01May2019 1305, Gerald Wiltse wrote: >>>>>> > That does help, thanks so much for the fast response! I now see >>>>>> how >>>>>> > it's executed on Azure pipelines also, so that should be enough for >>>>>> me >>>>>> > to reverse engineer the rest. >>>>>> > >>>>>> > Followup questions if you have time: >>>>>> > 1. It seems that directory doesn't exist in the 3.6 branches/tags, >>>>>> so it >>>>>> > must have been done a different way back then. Do you think it's >>>>>> > back-portable to 3.6? >>>>>> >>>>>> Yep, it should work with 3.6. There were scripts under Tools/msi back >>>>>> then that did the same thing, but with the Store app I rewrote it to >>>>>> be >>>>>> more flexible. >>>>>> >>>>>> > 2. One of our objectives is to effectively create a nearly >>>>>> identical >>>>>> > embeddable package for linux. Do you think the layout code is >>>>>> generic >>>>>> > enough to be modified to create a linux equivalent? >>>>>> >>>>>> Probably not, it's under the "PC" directory because it is entirely >>>>>> CPython specific. >>>>>> >>>>>> I'd love to be able to have a similar embeddable distro for Linux >>>>>> (and >>>>>> macOS), and I'll be investing some of my time in working towards >>>>>> this, >>>>>> but it's far more complicated than simply laying out the right files. >>>>>> If >>>>>> you're interested in helping with this, then we're in the right place >>>>>> to >>>>>> discuss it, but we'll likely have to move to the issue tracker at >>>>>> some >>>>>> point once we figure out who else wants/needs to be involved. >>>>>> >>>>>> > Obviously, there are many alternative methods and documents for >>>>>> creating >>>>>> > "embeddable" pythons on linux, but the windows package has all the >>>>>> > characteristics we want, and of course we'd like our Windows and >>>>>> Linux >>>>>> > process build and packaging processes to be as similar as possible, >>>>>> so >>>>>> > it seems attractive to try to port this layout script to linux. >>>>>> >>>>>> "make install" already does the same thing, but the install location >>>>>> matters at build time, so it's a little more complex to make it work >>>>>> as >>>>>> an embedded runtime. I'm sure there are people who know how to make >>>>>> it >>>>>> work though. >>>>>> >>>>>> Cheers, >>>>>> Steve >>>>>> >>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From jerrywiltse at gmail.com Fri May 3 13:53:00 2019 From: jerrywiltse at gmail.com (Gerald Wiltse) Date: Fri, 3 May 2019 13:53:00 -0400 Subject: [Python-Dev] Location of CI for Windows Embeddable Distro In-Reply-To: References: Message-ID: Now that I have it working, I recall that the runtime challenge for a portable/embeddable distro on Linux is in the system configuration handling. This has always been a complicated fundamental behavior which has prevented or complicated many "portability" cases in the past. There are many projects which have managed to ship portable/relocatable python installations in the past. Maybe looking at them will be helpful in deciding how to deal with the situation here. Gerald R. Wiltse jerrywiltse at gmail.com On Fri, May 3, 2019 at 1:02 PM Gerald Wiltse wrote: > I have it working on linux now. Is now a good time to open a ticket on > Issue Tracker? > > Gerald R. Wiltse > jerrywiltse at gmail.com > > > > On Fri, May 3, 2019 at 11:55 AM Gerald Wiltse > wrote: > >> Ok, disregard previous message. I fixed the PYTHONPATH in my command >> and the .pyc files and . I also saw that python36._pth is specifically >> only supported by windows. >> >> Now i'm moved on to the following error: >> >> zipimport.ZipImportError: can't decompress data; zlib not available >> >> I'm on Ubuntu. I've definitely got zlib1g-dev package installed, and >> I've recompiled a number of times trying to get it to work. I've tried >> Ubuntu 16 and 18. >> >> Is this just a limitation on linux? >> >> Gerald R. Wiltse >> jerrywiltse at gmail.com >> >> >> >> On Thu, May 2, 2019 at 4:29 PM Gerald Wiltse >> wrote: >> >>> Just an update based on testing. If I zip the modules up into >>> python36.zip (whether in .py or .pyc form) the linux binary does not find >>> them. If I copy them into the embedded root, alongside the python >>> executable, it finds them as expected. >>> >>> This just makes it seem more like an issue where Linux binary does not >>> respect .zip file as a source of modules. >>> >>> Gerald R. Wiltse >>> jerrywiltse at gmail.com >>> >>> >>> >>> On Thu, May 2, 2019 at 2:56 PM Gerald Wiltse >>> wrote: >>> >>>> With minimal patching of tools/msi/make_zip.py, I was able to create an >>>> equivalent zip package on Linux. It doesn't work out of the box of >>>> course. I had to do two commands >>>> >>>> My biggest concern is that even if the layout is "correct" on linux, >>>> the python runtime might not actually support being run like this on >>>> Linux. Perhaps the code for using python36.zip as a landmark and source of >>>> modules is only implemented on Windows. Do you know if that is the case? >>>> >>>> tmp$ export LD_LIBRARY_PATH=$(pwd) >>>> tmp$ export PYTHONHOME=$(pwd) >>>> tmp$ export PYTHONPATH=$(pwd) >>>> tmp$ ./python >>>> Fatal Python error: Py_Initialize: Unable to get the locale encoding >>>> ModuleNotFoundError: No module named 'encodings' >>>> >>>> Current thread 0x00007fdde9eb1680 (most recent call first): >>>> Aborted (core dumped) >>>> >>>> Here's the full directory structure, error, and patch I created. Any >>>> additional feedback would be great. >>>> >>>> https://gist.github.com/solvingj/87305e0f70e14d738d6989ba2f41352f >>>> >>>> Gerald R. Wiltse >>>> jerrywiltse at gmail.com >>>> >>>> >>>> >>>> On Thu, May 2, 2019 at 11:24 AM Gerald Wiltse >>>> wrote: >>>> >>>>> I've successfully automated the build of the embedded zip on Windows >>>>> for 3.6 on my internal CI using tools/msi script. >>>>> >>>>> Moving onto linux, can you please summarize the primary >>>>> challenges/differences for creating the embeddable zip for linux? >>>>> >>>>> Gerald R. Wiltse >>>>> jerrywiltse at gmail.com >>>>> >>>>> >>>>> >>>>> On Wed, May 1, 2019 at 2:47 PM Gerald Wiltse >>>>> wrote: >>>>> >>>>>> Yes, I'll help with this, and I try to get signed up with an account >>>>>> for the issue tracker ASAP. >>>>>> >>>>>> I've just started trying to build create the embeddable package for >>>>>> windows locally (all from source). Once I have that automated, I'll take a >>>>>> naive stab at the Linux layout. I'm sure i'll soon hit the additional >>>>>> complications you're talking about and come back to you with questions. >>>>>> >>>>>> Gerald R. Wiltse >>>>>> jerrywiltse at gmail.com >>>>>> >>>>>> >>>>>> >>>>>> On Wed, May 1, 2019 at 1:57 PM Steve Dower >>>>>> wrote: >>>>>> >>>>>>> On 01May2019 1305, Gerald Wiltse wrote: >>>>>>> > That does help, thanks so much for the fast response! I now see >>>>>>> how >>>>>>> > it's executed on Azure pipelines also, so that should be enough >>>>>>> for me >>>>>>> > to reverse engineer the rest. >>>>>>> > >>>>>>> > Followup questions if you have time: >>>>>>> > 1. It seems that directory doesn't exist in the 3.6 branches/tags, >>>>>>> so it >>>>>>> > must have been done a different way back then. Do you think it's >>>>>>> > back-portable to 3.6? >>>>>>> >>>>>>> Yep, it should work with 3.6. There were scripts under Tools/msi >>>>>>> back >>>>>>> then that did the same thing, but with the Store app I rewrote it to >>>>>>> be >>>>>>> more flexible. >>>>>>> >>>>>>> > 2. One of our objectives is to effectively create a nearly >>>>>>> identical >>>>>>> > embeddable package for linux. Do you think the layout code is >>>>>>> generic >>>>>>> > enough to be modified to create a linux equivalent? >>>>>>> >>>>>>> Probably not, it's under the "PC" directory because it is entirely >>>>>>> CPython specific. >>>>>>> >>>>>>> I'd love to be able to have a similar embeddable distro for Linux >>>>>>> (and >>>>>>> macOS), and I'll be investing some of my time in working towards >>>>>>> this, >>>>>>> but it's far more complicated than simply laying out the right >>>>>>> files. If >>>>>>> you're interested in helping with this, then we're in the right >>>>>>> place to >>>>>>> discuss it, but we'll likely have to move to the issue tracker at >>>>>>> some >>>>>>> point once we figure out who else wants/needs to be involved. >>>>>>> >>>>>>> > Obviously, there are many alternative methods and documents for >>>>>>> creating >>>>>>> > "embeddable" pythons on linux, but the windows package has all the >>>>>>> > characteristics we want, and of course we'd like our Windows and >>>>>>> Linux >>>>>>> > process build and packaging processes to be as similar as >>>>>>> possible, so >>>>>>> > it seems attractive to try to port this layout script to linux. >>>>>>> >>>>>>> "make install" already does the same thing, but the install location >>>>>>> matters at build time, so it's a little more complex to make it work >>>>>>> as >>>>>>> an embedded runtime. I'm sure there are people who know how to make >>>>>>> it >>>>>>> work though. >>>>>>> >>>>>>> Cheers, >>>>>>> Steve >>>>>>> >>>>>> -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri May 3 14:07:47 2019 From: status at bugs.python.org (Python tracker) Date: Fri, 3 May 2019 18:07:47 +0000 (UTC) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20190503180747.44D9052B035@bugs.ams1.psf.io> ACTIVITY SUMMARY (2019-04-26 - 2019-05-03) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7081 ( -2) closed 41454 (+52) total 48535 (+50) Open issues with patches: 2820 Issues opened (41) ================== #36356: Failure to build with address sanitizer https://bugs.python.org/issue36356 reopened by vstinner #36707: The "m" ABI flag of SOABI for pymalloc is no longer needed https://bugs.python.org/issue36707 reopened by ned.deily #36737: Warnings operate out of global runtime state. https://bugs.python.org/issue36737 opened by eric.snow #36738: Add 'array_hook' for json module https://bugs.python.org/issue36738 opened by matomatical #36739: "4.6. Defining Functions" should mention nonlocal https://bugs.python.org/issue36739 opened by pbhd0815 #36743: Docs: Descript __get__ signature defined differently across th https://bugs.python.org/issue36743 opened by jdufresne #36744: functools.singledispatch: Shouldn't require a positional argum https://bugs.python.org/issue36744 opened by KevinG #36746: Create test for fcntl.lockf() https://bugs.python.org/issue36746 opened by nanjekyejoannah #36748: Optimize textio write buffering https://bugs.python.org/issue36748 opened by inada.naoki #36749: PPC64 AIX 3.x: compilation issue, linker fails to locate symbo https://bugs.python.org/issue36749 opened by vstinner #36750: test_socket leaks file descriptors on macOS https://bugs.python.org/issue36750 opened by xtreak #36751: Changes in the inspect module for PEP 570 https://bugs.python.org/issue36751 opened by pablogsal #36752: test multiprocessing: test_rapid_restart() crash on AIX https://bugs.python.org/issue36752 opened by vstinner #36753: Python modules not linking to libpython causes issues for RTLD https://bugs.python.org/issue36753 opened by reimar #36754: Remove smart quotes in pydoc text https://bugs.python.org/issue36754 opened by steve.dower #36755: [2.7] test_httplib leaked [8, 8, 8] references with OpenSSL 1. https://bugs.python.org/issue36755 opened by vstinner #36756: tkinter tk.createcommand memory leak https://bugs.python.org/issue36756 opened by WKraus #36757: uuid constructor accept invalid strings (extra dash) https://bugs.python.org/issue36757 opened by C??dric Cabessa #36758: configured libdir not correctly passed to Python executable https://bugs.python.org/issue36758 opened by michael-o #36759: datetime: astimezone() results in OSError: [Errno 22] Invalid https://bugs.python.org/issue36759 opened by Snidhi #36760: subprocess.run fails with capture_output=True and stderr=STDOU https://bugs.python.org/issue36760 opened by Joe.Borg #36761: Extended slice assignment + iterable unpacking https://bugs.python.org/issue36761 opened by wim.glenn #36762: Teach "import *" to warn when overwriting globals or builtins https://bugs.python.org/issue36762 opened by rhettinger #36763: PEP 587: Rework initialization API to prepare second version o https://bugs.python.org/issue36763 opened by vstinner #36766: Typos in docs and code comments https://bugs.python.org/issue36766 opened by xtreak #36767: Segmentation fault when running c extension on macOS https://bugs.python.org/issue36767 opened by Senhui Guo #36768: distutils.util.convert_path mangles windows paths with forward https://bugs.python.org/issue36768 opened by LordAro #36769: doc Document that fnmatch.filter supports any kind of iterable https://bugs.python.org/issue36769 opened by adelfino #36770: stdlib - shutil.make_archive - add support for different ZIP c https://bugs.python.org/issue36770 opened by owenchia #36771: Feature Request: An option to os.walk() to return os.DirEntry https://bugs.python.org/issue36771 opened by apocalyptech #36772: Let lru_cache be used as a decorator with no arguments https://bugs.python.org/issue36772 opened by rhettinger #36773: Race condition during pickle.load() https://bugs.python.org/issue36773 opened by papad #36774: f-strings: Add a !d conversion for ease of debugging https://bugs.python.org/issue36774 opened by eric.smith #36777: unittest discover throws TypeError on empty packages https://bugs.python.org/issue36777 opened by xtreak #36778: test_site.StartupImportTests.test_startup_imports fails if def https://bugs.python.org/issue36778 opened by Paul Monson #36779: time.tzname returns empty string on Windows if default codepag https://bugs.python.org/issue36779 opened by Paul Monson #36780: Interpreter exit blocks waiting for futures of shut-down Threa https://bugs.python.org/issue36780 opened by hniksic #36781: Optimize sum() for bools https://bugs.python.org/issue36781 opened by serhiy.storchaka #36782: Add tests for the datetime C API https://bugs.python.org/issue36782 opened by p-ganssle #36783: No documentation for _FromXandFold C API functions https://bugs.python.org/issue36783 opened by p-ganssle #36784: __import__ with empty folder after importlib.invalidate_caches https://bugs.python.org/issue36784 opened by xtreak Most recent 15 issues with no replies (15) ========================================== #36784: __import__ with empty folder after importlib.invalidate_caches https://bugs.python.org/issue36784 #36781: Optimize sum() for bools https://bugs.python.org/issue36781 #36780: Interpreter exit blocks waiting for futures of shut-down Threa https://bugs.python.org/issue36780 #36779: time.tzname returns empty string on Windows if default codepag https://bugs.python.org/issue36779 #36770: stdlib - shutil.make_archive - add support for different ZIP c https://bugs.python.org/issue36770 #36769: doc Document that fnmatch.filter supports any kind of iterable https://bugs.python.org/issue36769 #36768: distutils.util.convert_path mangles windows paths with forward https://bugs.python.org/issue36768 #36766: Typos in docs and code comments https://bugs.python.org/issue36766 #36758: configured libdir not correctly passed to Python executable https://bugs.python.org/issue36758 #36755: [2.7] test_httplib leaked [8, 8, 8] references with OpenSSL 1. https://bugs.python.org/issue36755 #36752: test multiprocessing: test_rapid_restart() crash on AIX https://bugs.python.org/issue36752 #36748: Optimize textio write buffering https://bugs.python.org/issue36748 #36738: Add 'array_hook' for json module https://bugs.python.org/issue36738 #36732: test_asyncio: test_huge_content_recvinto() fails randomly https://bugs.python.org/issue36732 #36728: Remove PyEval_ReInitThreads() from the public C API https://bugs.python.org/issue36728 Most recent 15 issues waiting for review (15) ============================================= #36778: test_site.StartupImportTests.test_startup_imports fails if def https://bugs.python.org/issue36778 #36774: f-strings: Add a !d conversion for ease of debugging https://bugs.python.org/issue36774 #36772: Let lru_cache be used as a decorator with no arguments https://bugs.python.org/issue36772 #36771: Feature Request: An option to os.walk() to return os.DirEntry https://bugs.python.org/issue36771 #36770: stdlib - shutil.make_archive - add support for different ZIP c https://bugs.python.org/issue36770 #36769: doc Document that fnmatch.filter supports any kind of iterable https://bugs.python.org/issue36769 #36763: PEP 587: Rework initialization API to prepare second version o https://bugs.python.org/issue36763 #36751: Changes in the inspect module for PEP 570 https://bugs.python.org/issue36751 #36749: PPC64 AIX 3.x: compilation issue, linker fails to locate symbo https://bugs.python.org/issue36749 #36748: Optimize textio write buffering https://bugs.python.org/issue36748 #36746: Create test for fcntl.lockf() https://bugs.python.org/issue36746 #36743: Docs: Descript __get__ signature defined differently across th https://bugs.python.org/issue36743 #36739: "4.6. Defining Functions" should mention nonlocal https://bugs.python.org/issue36739 #36738: Add 'array_hook' for json module https://bugs.python.org/issue36738 #36725: Reference leak regression with Python3.8a3 https://bugs.python.org/issue36725 Top 10 most discussed issues (10) ================================= #36774: f-strings: Add a !d conversion for ease of debugging https://bugs.python.org/issue36774 14 msgs #36751: Changes in the inspect module for PEP 570 https://bugs.python.org/issue36751 12 msgs #30458: [security][CVE-2019-9740][CVE-2019-9947] HTTP Header Injection https://bugs.python.org/issue30458 9 msgs #36763: PEP 587: Rework initialization API to prepare second version o https://bugs.python.org/issue36763 9 msgs #26493: Bad formatting in WinError 193 when using subprocess.check_cal https://bugs.python.org/issue26493 8 msgs #36710: Pass _PyRuntimeState as an argument rather than using the _PyR https://bugs.python.org/issue36710 8 msgs #34616: implement "Async exec" https://bugs.python.org/issue34616 7 msgs #35184: Makefile is not correctly generated when compiling pyextat wit https://bugs.python.org/issue35184 5 msgs #36356: Failure to build with address sanitizer https://bugs.python.org/issue36356 5 msgs #36540: PEP 570: Python Positional-Only Parameters https://bugs.python.org/issue36540 5 msgs Issues closed (51) ================== #5166: ElementTree and minidom don't prevent creation of not well-for https://bugs.python.org/issue5166 closed by scoder #8583: Hardcoded namespace_separator in the cElementTree.XMLParser https://bugs.python.org/issue8583 closed by scoder #10328: re.sub[n] doesn't seem to handle /Z replacements correctly in https://bugs.python.org/issue10328 closed by serhiy.storchaka #13611: Integrate ElementC14N module into xml.etree package https://bugs.python.org/issue13611 closed by scoder #14546: lll.py can't handle multiple parameters correctly https://bugs.python.org/issue14546 closed by vstinner #15388: SAX parse (ExpatParser) leaks file handle when given filename https://bugs.python.org/issue15388 closed by berker.peksag #16425: minidom replaceChild(new_child, old_child) removes new_child e https://bugs.python.org/issue16425 closed by scoder #18675: Daemon Threads can seg fault https://bugs.python.org/issue18675 closed by asvetlov #19895: Cryptic error when subclassing multiprocessing classes https://bugs.python.org/issue19895 closed by SilentGhost #21403: cElementTree's Element creation handles attrib argument differ https://bugs.python.org/issue21403 closed by scoder #24638: asyncio "loop argument must agree with future" error message c https://bugs.python.org/issue24638 closed by asvetlov #27682: wsgiref BaseHandler / SimpleHandler can raise additional error https://bugs.python.org/issue27682 closed by berker.peksag #28460: Minidom, order of attributes, datachars https://bugs.python.org/issue28460 closed by scoder #28608: Support creating hardlink using `pathlib` https://bugs.python.org/issue28608 closed by xtreak #32235: test_xml_etree test_xml_etree_c failures with 2.7 and 3.6 bran https://bugs.python.org/issue32235 closed by scoder #34442: zlib module not built on windows https://bugs.python.org/issue34442 closed by zach.ware #34602: python3 resource.setrlimit strange behaviour under macOS https://bugs.python.org/issue34602 closed by ned.deily #35329: Documentation - capitalization issue https://bugs.python.org/issue35329 closed by benjamin.peterson #35502: Memory leak in xml.etree.ElementTree.iterparse https://bugs.python.org/issue35502 closed by scoder #35726: QueueHandler formatting affects other handlers https://bugs.python.org/issue35726 closed by ned.deily #35947: Update libffi_msvc to current version of libffi https://bugs.python.org/issue35947 closed by Paul Monson #35952: test.pythoninfo prints a stack trace and exits with 1 when the https://bugs.python.org/issue35952 closed by vstinner #36004: Add datetime.fromisocalendar https://bugs.python.org/issue36004 closed by vstinner #36025: Breaking change in PyDate_FromTimeStamp API https://bugs.python.org/issue36025 closed by berker.peksag #36341: bind() on AF_UNIX socket may fail in tests run as non-root https://bugs.python.org/issue36341 closed by asvetlov #36432: Running python test suite fails on macOS 10.14.4 with resource https://bugs.python.org/issue36432 closed by ned.deily #36475: PyEval_AcquireLock() and PyEval_AcquireThread() do not handle https://bugs.python.org/issue36475 closed by vstinner #36509: Add iot layout for windows iot containers https://bugs.python.org/issue36509 closed by Paul Monson #36613: asyncio._wait() don't remove callback in case of exception https://bugs.python.org/issue36613 closed by asvetlov #36673: Comment/PI parsing support for ElementTree https://bugs.python.org/issue36673 closed by scoder #36676: Make ET.XMLParser target aware of namespace prefixes https://bugs.python.org/issue36676 closed by scoder #36699: building for riscv multilib (patch attached) https://bugs.python.org/issue36699 closed by Andreas K. H??ttel #36715: Dictionary initialization https://bugs.python.org/issue36715 closed by rhettinger #36716: Embedded Python fails to import module files with version_plat https://bugs.python.org/issue36716 closed by ecosatto #36719: regrtest --findleaks should fail if an uncollectable object is https://bugs.python.org/issue36719 closed by vstinner #36729: Delete unused text variable on tests https://bugs.python.org/issue36729 closed by serhiy.storchaka #36730: Change outdated references to macOS https://bugs.python.org/issue36730 closed by berker.peksag #36734: Modules/faulthandler.c does not compile on HP-UX due to bpo-35 https://bugs.python.org/issue36734 closed by vstinner #36735: minimize disk size of cross-compiled python3.6 https://bugs.python.org/issue36735 closed by michalgr #36736: Python crashes when calling win32file.LockFileEx https://bugs.python.org/issue36736 closed by eryksun #36740: zipimporter misses namespace packages for implicit dirs https://bugs.python.org/issue36740 closed by jaraco #36741: Variable about function and list https://bugs.python.org/issue36741 closed by xtreak #36742: urlsplit doesn't accept a NFKD hostname with a port number https://bugs.python.org/issue36742 closed by steve.dower #36745: A possible reference leak in PyObject_SetAttr() https://bugs.python.org/issue36745 closed by asvetlov #36747: Tools/scripts/setup.py is missing https://bugs.python.org/issue36747 closed by vstinner #36764: Types module doesn't have a type for _abc_data https://bugs.python.org/issue36764 closed by serhiy.storchaka #36765: Invalid grammar for f_expression https://bugs.python.org/issue36765 closed by eric.smith #36775: Rework filesystem codec implementation https://bugs.python.org/issue36775 closed by vstinner #36776: test_tools: test_lll_multiple_dirs() failed on AMD64 Windows7 https://bugs.python.org/issue36776 closed by vstinner #1613500: Write mode option for fileinput module. https://bugs.python.org/issue1613500 closed by berker.peksag #1191964: add non-blocking read and write methods to subprocess.Popen https://bugs.python.org/issue1191964 closed by josiahcarlson From solipsis at pitrou.net Fri May 3 17:40:00 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 3 May 2019 23:40:00 +0200 Subject: [Python-Dev] PEP 574 ready for review In-Reply-To: References: <20190430101428.204457e6@fsol> Message-ID: <20190503234000.267faa69@fsol> On Wed, 1 May 2019 12:49:24 -0700 Nick Coghlan wrote: > Thanks Antoine. > > As BDFL-Delegate I'm happy with this version of the PEP, so it's my > pleasure to accept it for inclusion in Python 3.8. Thank you Nick! The implementation has been posted for review at https://github.com/python/cpython/pull/7076 Regards Antoine. From eric at trueblade.com Fri May 3 22:46:03 2019 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 3 May 2019 22:46:03 -0400 Subject: [Python-Dev] Unicode identifiers in test files? Message-ID: Is there a policy against using Unicode identifiers in test files? As part of adding !d to f-strings, there's a code path that's only executed if the text of the expression is non-ascii. The easiest way to exercise it, and the way I found a bug, is by using an identifier with Unicode chars. I know we have a policy against this in Lib/, but what about Lib/test/? I could work around this with exec-ing some strings, but that seems like added confusion that I'd avoid with a real Unicode identifier. Eric From greg at krypto.org Fri May 3 23:27:00 2019 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 3 May 2019 23:27:00 -0400 Subject: [Python-Dev] Unicode identifiers in test files? In-Reply-To: References: Message-ID: On Fri, May 3, 2019 at 10:46 PM Eric V. Smith wrote: > Is there a policy against using Unicode identifiers in test files? > > As part of adding !d to f-strings, there's a code path that's only > executed if the text of the expression is non-ascii. The easiest way to > exercise it, and the way I found a bug, is by using an identifier with > Unicode chars. I know we have a policy against this in Lib/, but what > about Lib/test/? > > I could work around this with exec-ing some strings, but that seems like > added confusion that I'd avoid with a real Unicode identifier. > At this point I'd just do it, make sure the file has an explicit coding utf-8 comment up top just so it is clear it is intentional but everything should be happy with it. -gps > > Eric > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri May 3 23:50:35 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 4 May 2019 13:50:35 +1000 Subject: [Python-Dev] Unicode identifiers in test files? In-Reply-To: References: Message-ID: <20190504035034.GE5010@ando.pearwood.info> On Fri, May 03, 2019 at 10:46:03PM -0400, Eric V. Smith wrote: > Is there a policy against using Unicode identifiers in test files? [...] > I could work around this with exec-ing some strings, but that seems like > added confusion that I'd avoid with a real Unicode identifier. "Look, that's why there's rules, understand? So that you *think* before you break 'em." -- Terry Pratchett, "The Thief Of Time" -- Steven From storchaka at gmail.com Sat May 4 02:48:04 2019 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 4 May 2019 09:48:04 +0300 Subject: [Python-Dev] Unicode identifiers in test files? In-Reply-To: References: Message-ID: 04.05.19 05:46, Eric V. Smith ????: > Is there a policy against using Unicode identifiers in test files? > > As part of adding !d to f-strings, there's a code path that's only > executed if the text of the expression is non-ascii. The easiest way to > exercise it, and the way I found a bug, is by using an identifier with > Unicode chars. I know we have a policy against this in Lib/, but what > about Lib/test/? > > I could work around this with exec-ing some strings, but that seems like > added confusion that I'd avoid with a real Unicode identifier. Could you use string literals with non-ascii characters? They are more used in tests. From eric at trueblade.com Sat May 4 03:54:04 2019 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 4 May 2019 03:54:04 -0400 Subject: [Python-Dev] Unicode identifiers in test files? In-Reply-To: References: Message-ID: On 5/4/19 2:48 AM, Serhiy Storchaka wrote: > 04.05.19 05:46, Eric V. Smith ????: >> Is there a policy against using Unicode identifiers in test files? >> >> As part of adding !d to f-strings, there's a code path that's only >> executed if the text of the expression is non-ascii. The easiest way >> to exercise it, and the way I found a bug, is by using an identifier >> with Unicode chars. I know we have a policy against this in Lib/, but >> what about Lib/test/? >> >> I could work around this with exec-ing some strings, but that seems >> like added confusion that I'd avoid with a real Unicode identifier. > > Could you use string literals with non-ascii characters? They are more > used in tests. Hi, Serhiy. I could and will, yes: thanks for the suggestion. But for this specific feature, I also want to test with simple variable-only expressions. I'm either going to use a unicode identifier in the code, or eval one in a string. Doing the eval dance just seems like extra work for some sort of purity that I don't think is important in a test, unless someone can think of a good reason for it. Eric From eric at trueblade.com Sat May 4 04:10:32 2019 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 4 May 2019 04:10:32 -0400 Subject: [Python-Dev] Unicode identifiers in test files? In-Reply-To: References: Message-ID: <64330e01-d4c0-c048-a2fb-74810cf8d930@trueblade.com> On 5/4/19 3:54 AM, Eric V. Smith wrote: > On 5/4/19 2:48 AM, Serhiy Storchaka wrote: >> 04.05.19 05:46, Eric V. Smith ????: >>> Is there a policy against using Unicode identifiers in test files? >>> >>> As part of adding !d to f-strings, there's a code path that's only >>> executed if the text of the expression is non-ascii. The easiest way >>> to exercise it, and the way I found a bug, is by using an identifier >>> with Unicode chars. I know we have a policy against this in Lib/, but >>> what about Lib/test/? >>> >>> I could work around this with exec-ing some strings, but that seems >>> like added confusion that I'd avoid with a real Unicode identifier. >> >> Could you use string literals with non-ascii characters? They are more >> used in tests. > > Hi, Serhiy. > > I could and will, yes: thanks for the suggestion. But for this specific > feature, I also want to test with simple variable-only expressions. I'm > either going to use a unicode identifier in the code, or eval one in a > string. Doing the eval dance just seems like extra work for some sort of > purity that I don't think is important in a test, unless someone can > think of a good reason for it. And I just noticed that PEP 3131 has an exception for tests in its section that says the stdlib can't contain unicode identifiers: https://www.python.org/dev/peps/pep-3131/#policy-specification So, since it's the most direct and probably safest thing to do for a test, I'm going to use a unicode identier. Thanks, all, for your ideas. Especially Greg for reminding me to add the coding comment. Eric From songofacandy at gmail.com Sat May 4 04:45:38 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Sat, 4 May 2019 17:45:38 +0900 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: References: Message-ID: 2019?5?3?(?) 4:59 Victor Stinner : > > * PyConfig now only uses wchar_t* for strings (unicode): char* (bytes) > is no longer used. I had to hack CPython internals for that :-) I prefer char* to wchar_t* on Unix. Since UTF-8 dominated Unix world in these decades, wchar_t* is less usable on Unix nowadays. Is it impossible to use just char* on Unix and wchar_t* on Windows? -- Inada Naoki From vstinner at redhat.com Sat May 4 11:45:49 2019 From: vstinner at redhat.com (Victor Stinner) Date: Sat, 4 May 2019 11:45:49 -0400 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: References: Message-ID: Hi INADA-san, This PEP is the result of 2 years of refactoring to *simplify* the *implementation*. I agree that bytes string is the native type on Unix. But. On Windows, Unicode is the native type. On Python 3, Unicode is the native type. One key of the simplified implementation is the unique PyConfig structure. It means that all platforms have to use the same types. I love the idea of using only wchar_t* for PyConfig because it makes Python initialization more reliable. The question of the encoding used to decode byte strings and any possible decoding error (very unlikely thanks to surrogateescape) is better defined: it occurs when you set the parameter, not "later during init". The PEP adds Py_UnixMain() for most trivial use cases, and PyConfig_DecodeLocale() and PyConfig_SetArgs() for more advanced cases. Victor Le samedi 4 mai 2019, Inada Naoki a ?crit : > 2019?5?3?(?) 4:59 Victor Stinner : >> >> * PyConfig now only uses wchar_t* for strings (unicode): char* (bytes) >> is no longer used. I had to hack CPython internals for that :-) > > I prefer char* to wchar_t* on Unix. Since UTF-8 dominated Unix world > in these decades, wchar_t* is less usable on Unix nowadays. > > Is it impossible to use just char* on Unix and wchar_t* on Windows? > > -- > Inada Naoki > -- Night gathers, and now my watch begins. It shall not end until my death. -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Sat May 4 17:02:47 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 5 May 2019 06:02:47 +0900 Subject: [Python-Dev] "if __name__ == '__main__'" at the bottom of python unittest files In-Reply-To: References: <01ebef8d-d370-22c5-cb7a-194704a3906c@withers.org> <20190501124831.7faadb2b@fsol> <05ef1790-dc90-f5a6-a25c-9ba5abc66a3a@withers.org> Message-ID: <23757.65015.549415.193161@turnbull.sk.tsukuba.ac.jp> Terry Reedy writes: > I agree that complete 100.000% test coverage is a nice ideal, but > sometimes the last percent can take hours to accomplish, if it is > indeed sensibly possible. 100% test coverage is an ideal. Reports *claiming* 100% coverage, however, are of practical benefit. The point is to identify a regression. It's best to have 100% coverage, because it's possible that improvements in the environment allow a test that wasn't reliable (== deterministic) or maybe not even feasible before to become feasible and reliable, and a coveragerc that says "oh, that line is OK" will obscure that possibility. But a *claim* (albeit somewhat undermined by a non-trivial coveragerc) of 100% coverage means it's *easy to identify regressions in coverage*. I think that's a bigger deal, at least at this time. > I have asked more that once what .coveragerc file is being used by > CI and whether we can somehow properly customize it for CPython. Seconded. Thank you for pushing this, and for all the other efforts you're making here. Steve From ericsnowcurrently at gmail.com Sun May 5 09:47:07 2019 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Sun, 5 May 2019 09:47:07 -0400 Subject: [Python-Dev] PEP 574 ready for review In-Reply-To: References: <20190430101428.204457e6@fsol> Message-ID: On Wed, May 1, 2019, 15:49 Nick Coghlan wrote: > As BDFL-Delegate I'm happy with this version of the PEP, so it's my > pleasure to accept it for inclusion in Python 3.8. > Congratulations, Antoine! This is a big deal and a lot of folks won't realize it at first. :) -eric > -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Sun May 5 18:04:06 2019 From: encukou at gmail.com (Petr Viktorin) Date: Sun, 5 May 2019 18:04:06 -0400 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: Hello! Sorry for the delay; PyCon is keeping me busy. On the other hand, I did get to talk to a lot of smart people here! I'm leaning toward accepting PEP 590 (with some changes still). Let's start focusing on it. As for the changes, I have these 4 points: I feel that the API needs some contact with real users before it's set in stone. That was the motivation behind my proposal for PEP 590 with additional flags. At PyCon, Nick Coghlan suggested another option make the API "provisional": make it formally private. Py_TPFLAGS_HAVE_VECTORCALL would be underscore-prefixed, and the docs would say that it can change. in Python 3.9, the semantics will be finalized and the underscore removed. This would allow high-maintenance projects (like Cython) to start using it and give their feedback, and we'd have a chance to respond to the feedback. tp_vectorcall_offset should be what's replacing tp_print in the struct. The current implementation has tp_vectorcall there. This way, Cython can create vectorcall callables for older Pythons. (See PEP 580: https://www.python.org/dev/peps/pep-0580/#replacing-tp-print). Subclassing should not be forbidden. Jeroen, do you want write a section for how subclassing should work? Given Jeroen's research and ideas that went into the PEP (and hopefully, we'll incorporate some PEP 580 text as well), it seems fair to list him as co-author of the accepted PEP, instead of just listing PEP 580 in the acknowledgement section. On some other points: - Single bound method class for all kinds of function classes: This would be a cleaner design, yes, but I don't see a pressing need. As PEP 579 says, "this is a compounding issue", not a goal. As I recall, that is the only major reason for CCALL_DEFARG. PEP 590 says that x64 Windows passes 4 arguments in registers. Admittedly, I haven't checked this, nor the performance implications (so this would be a good point to argue!), but it seems like a good reason to keep the argument count down. So, no CCALL_DEFARG. - In reply to this Mark's note: > PEP 590 is fully universal, it supports callables that can do anything with anything. There is no need for it to be extended because it already supports any possible behaviour. I don't buy this point. The current tp_call also supports any possible behavior. Here we want to support any behavior *efficiently*. As a specific example: for calling PEP 590 callable with a kwarg dict, there'll need to be an extra allocation. That's inefficient relative to PEP 580 (or PEP 590 plus allowing a dict in "kwnames"). But I'm willing to believe the inefficiency is acceptable. From J.Demeyer at UGent.be Mon May 6 03:43:46 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 6 May 2019 09:43:46 +0200 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: <5CCFE5B2.3020602@UGent.be> On 2019-05-06 00:04, Petr Viktorin wrote: > - Single bound method class for all kinds of function classes: This > would be a cleaner design, yes, but I don't see a pressing need. As > PEP 579 says, "this is a compounding issue", not a goal. As I recall, > that is the only major reason for CCALL_DEFARG. Just a minor correction here: I guess that you mean CCALL_SELFARG. The flag CCALL_DEFARG is for passing the PyCCallDef* in PEP 580, which is mostly equivalent to passing the callable object in PEP 590. The signature of PEP 580 is func(const PyCCallDef *def, PyObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) And with PEP 590 it is func(PyObject *callable, PyObject *const *args, Py_ssize_t nargs, PyObject *kwnames) with the additional special role for the PY_VECTORCALL_ARGUMENTS_OFFSET bit (which is meant to solve the problem of "self" in a different way). From J.Demeyer at UGent.be Mon May 6 04:24:44 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 6 May 2019 10:24:44 +0200 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: <5CCFEF4C.50404@UGent.be> Hello Petr, Thanks for your time. I suggest you (or somebody else) to officially reject PEP 580. I start working on reformulating PEP 590, adding some elements from PEP 580. At the same time, I work on the implementation of PEP 590. I want to implement Mark's idea of having a separate wrapper for each old-style calling convention. In the mean time, we can continue the discussion about the details, for example whether to store the flags inside the instance (I don't have an answer for that right now, I'll need to think about it). Petr, did you discuss with the Steering Council? It would be good to have some kind of pre-approval that PEP 590 and its implementation will be accepted. I want to work on PEP 590, but I'm not the right person to "defend" it (I know that it's worse in some ways than PEP 580). Jeroen. From J.Demeyer at UGent.be Mon May 6 09:55:03 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 6 May 2019 15:55:03 +0200 Subject: [Python-Dev] Stable ABI or not for PyTypeObject? Message-ID: <5CD03CB7.1050106@UGent.be> Hello, I have a simple question for which there doesn't seem to be a good answer: is the layout of PyTypeObject considered to be part of the stable ABI? Officially, the answer is certainly "no" (see PEP 384). However, unofficially the answer might be "yes". At least, the last time that an incompatible change was made to PyTypeObject (adding tp_finalize in Python 3.4, PEP 442), care was taken not to break the ABI by using the Py_TPFLAGS_HAVE_FINALIZE flag. There is some discussion about this on https://bugs.python.org/issue32388 The implementation of PEP 590 is going to make another ABI-breaking change. So should we add a new Py_TFLAGS_HAVE_... flag for that or not? Jeroen. From encukou at gmail.com Mon May 6 10:47:06 2019 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 6 May 2019 10:47:06 -0400 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: <5CCFEF4C.50404@UGent.be> References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> <5CCFEF4C.50404@UGent.be> Message-ID: <39140e82-b1b1-e7ef-e77e-0b57fa2d022b@gmail.com> On 5/6/19 4:24 AM, Jeroen Demeyer wrote: > Hello Petr, > > Thanks for your time. I suggest you (or somebody else) to officially > reject PEP 580. I'll do that shortly. I hope that you are not taking this personally. PEP 580 is a good design. PEP 590 even says that it's built on your ideas. > I start working on reformulating PEP 590, adding some elements from PEP > 580. At the same time, I work on the implementation of PEP 590. I want > to implement Mark's idea of having a separate wrapper for each old-style > calling convention. > > In the mean time, we can continue the discussion about the details, for > example whether to store the flags inside the instance (I don't have an > answer for that right now, I'll need to think about it). I'm abandoning per-instance flag proposal. It's an unnecessary complication; per-type flags are fine. > Petr, did you discuss with the Steering Council? It would be good to > have some kind of pre-approval that PEP 590 and its implementation will > be accepted. I want to work on PEP 590, but I'm not the right person to > "defend" it (I know that it's worse in some ways than PEP 580). As BDFL-delegate, I'm "pre-approving" PEP 590. I mentioned some details of PEP 590 that still need attention. If there are any more, now's the time to bring them up. And yes, I know that in some ways it's worse than PEP 580. That's what makes it a hard decision. From solipsis at pitrou.net Mon May 6 11:34:22 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 6 May 2019 17:34:22 +0200 Subject: [Python-Dev] Stable ABI or not for PyTypeObject? References: <5CD03CB7.1050106@UGent.be> Message-ID: <20190506173422.445a0933@fsol> On Mon, 6 May 2019 15:55:03 +0200 Jeroen Demeyer wrote: > Hello, > > I have a simple question for which there doesn't seem to be a good > answer: is the layout of PyTypeObject considered to be part of the > stable ABI? > > Officially, the answer is certainly "no" (see PEP 384). > > However, unofficially the answer might be "yes". At least, the last time > that an incompatible change was made to PyTypeObject (adding tp_finalize > in Python 3.4, PEP 442), care was taken not to break the ABI by using > the Py_TPFLAGS_HAVE_FINALIZE flag. > > There is some discussion about this on https://bugs.python.org/issue32388 The discussion there was more or less conclusive. At the time it was too late to go into 3.7 and then I (and others) forgot about it. We should revive that PR in time for 3.8. Regards Antoine. From vstinner at redhat.com Mon May 6 12:12:51 2019 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 6 May 2019 12:12:51 -0400 Subject: [Python-Dev] Stable ABI or not for PyTypeObject? In-Reply-To: <5CD03CB7.1050106@UGent.be> References: <5CD03CB7.1050106@UGent.be> Message-ID: PyType_FromSpec() looks like a better approach for ABI compatibility. My notes on types and ABI: https://pythoncapi.readthedocs.io/type_object.html Victor Le lun. 6 mai 2019 ? 09:57, Jeroen Demeyer a ?crit : > > Hello, > > I have a simple question for which there doesn't seem to be a good > answer: is the layout of PyTypeObject considered to be part of the > stable ABI? > > Officially, the answer is certainly "no" (see PEP 384). > > However, unofficially the answer might be "yes". At least, the last time > that an incompatible change was made to PyTypeObject (adding tp_finalize > in Python 3.4, PEP 442), care was taken not to break the ABI by using > the Py_TPFLAGS_HAVE_FINALIZE flag. > > There is some discussion about this on https://bugs.python.org/issue32388 > > The implementation of PEP 590 is going to make another ABI-breaking > change. So should we add a new Py_TFLAGS_HAVE_... flag for that or not? > > > Jeroen. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com -- Night gathers, and now my watch begins. It shall not end until my death. From encukou at gmail.com Mon May 6 12:25:34 2019 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 6 May 2019 12:25:34 -0400 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: <5CCFE5B2.3020602@UGent.be> References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> <5CCFE5B2.3020602@UGent.be> Message-ID: <1ffcaa84-1569-4212-00c4-fc4272ae4176@gmail.com> On 5/6/19 3:43 AM, Jeroen Demeyer wrote: > On 2019-05-06 00:04, Petr Viktorin wrote: >> - Single bound method class for all kinds of function classes: This >> would be a cleaner design, yes, but I don't see a pressing need. As >> PEP 579 says, "this is a compounding issue", not a goal. As I recall, >> that is the only major reason for CCALL_DEFARG. > > Just a minor correction here: I guess that you mean CCALL_SELFARG. The > flag CCALL_DEFARG is for passing the PyCCallDef* in PEP 580, which is > mostly equivalent to passing the callable object in PEP 590. > > The signature of PEP 580 is > > func(const PyCCallDef *def, PyObject *self, PyObject *const *args, > Py_ssize_t nargs, PyObject *kwnames) > > And with PEP 590 it is > > func(PyObject *callable, PyObject *const *args, Py_ssize_t nargs, > PyObject *kwnames) > > with the additional special role for the PY_VECTORCALL_ARGUMENTS_OFFSET > bit (which is meant to solve the problem of "self" in a different way). I worded that badly, sorry. From PEP 590's `callable`, the called function can get any of these if it needs to (and if they're stored somewhere). But you can't write generic code would get them from any callable. If we're not going for the "single bound method class" idea, that is OK; `def` & `self` can be implementation details of the callables that need them. From eric at trueblade.com Mon May 6 20:39:41 2019 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 6 May 2019 20:39:41 -0400 Subject: [Python-Dev] Easier debugging with f-strings Message-ID: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> Last fall Larry Hastings made a suggestion for adding a way to make so-called "print-based debugging" easier with f-strings. Basically the approach is that f-strings would be able to produce the text of the expression and the value of that expression, without repeating the expression in the f-sting. No more writing f'foo={foo}, bar={bar}'. foo and bar should each only be in there once each! At PyCon US 2019 I did a lightning talk about this, suggesting the syntax of !d, so that if foo="Hello", then f"{foo!d}" would produce "foo='Hello'". That is, it's the text of the expression, followed by an equal sign, followed by the repr of the expression. I have implemented this and a PR exists. Arbitrary expressions are allowed. I heard from core devs and end users after this talk, and all were positive. After that lightning talk, Larry and I talked about it some more, and for a number of reasons decided that it would make more sense if the syntax used an = sign. So we came up with f"{foo=}", which would also produce "foo='Hello'". The reasons for the change are: - Having '=' in the expression is a better mnemonic than !d. - By not using a conversion starting with !, we can compose = with the existing ! conversions, !r, !s, and the rarely used !a. - We can let the user have a little more control of the resulting string. Another thing I like about this approach over !d is that the patch is simpler, because there are fewer special cases. And because there are fewer special cases in the code, I think the proposal is easier to explain than !d (in particular how it interacts (or doesn't!) with format specifiers). There's a good rationale here, along with a PR: https://bugs.python.org/issue36817. My plan is to commit this change before 3.8b1. If anyone would like to discuss it at PyCon, I'll be around until about 10:30 am on Tuesday. I'll be in the CPython sprint room, and I'll be watching bpo, too. Eric From v+python at g.nevcal.com Mon May 6 21:29:44 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 6 May 2019 18:29:44 -0700 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> Message-ID: <15377535-338f-bd70-8221-52a6509c58a4@g.nevcal.com> On 5/6/2019 5:39 PM, Eric V. Smith wrote: > Last fall Larry Hastings made a suggestion for adding a way to make > so-called "print-based debugging" easier with f-strings. Basically the > approach is that f-strings would be able to produce the text of the > expression and the value of that expression, without repeating the > expression in the f-sting. No more writing f'foo={foo}, bar={bar}'. > foo and bar should each only be in there once each! > My plan is to commit this change before 3.8b1. If anyone would like to > discuss it at PyCon, I'll be around until about 10:30 am on Tuesday. > I'll be in the CPython sprint room, and I'll be watching bpo, too. Yes, I'd like this yesterday, please :)? Thanks! -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Tue May 7 03:52:37 2019 From: steve at holdenweb.com (Steve Holden) Date: Tue, 7 May 2019 08:52:37 +0100 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: <15377535-338f-bd70-8221-52a6509c58a4@g.nevcal.com> References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> <15377535-338f-bd70-8221-52a6509c58a4@g.nevcal.com> Message-ID: What's not to like? On Tue, May 7, 2019 at 2:31 AM Glenn Linderman wrote: > On 5/6/2019 5:39 PM, Eric V. Smith wrote: > > Last fall Larry Hastings made a suggestion for adding a way to make > so-called "print-based debugging" easier with f-strings. Basically the > approach is that f-strings would be able to produce the text of the > expression and the value of that expression, without repeating the > expression in the f-sting. No more writing f'foo={foo}, bar={bar}'. foo and > bar should each only be in there once each! > > > My plan is to commit this change before 3.8b1. If anyone would like to > discuss it at PyCon, I'll be around until about 10:30 am on Tuesday. I'll > be in the CPython sprint room, and I'll be watching bpo, too. > > > Yes, I'd like this yesterday, please :) Thanks! > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue May 7 04:02:13 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 7 May 2019 09:02:13 +0100 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> Message-ID: On Tue, 7 May 2019 at 01:43, Eric V. Smith wrote: > After that lightning talk, Larry and I talked about it some more, and > for a number of reasons decided that it would make more sense if the > syntax used an = sign. So we came up with f"{foo=}", which would also > produce "foo='Hello'". Works for me. Whatever the spelling, I expect I'll use this a lot - thanks :-) Paul From barry at python.org Tue May 7 09:49:46 2019 From: barry at python.org (Barry Warsaw) Date: Tue, 7 May 2019 09:49:46 -0400 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> <15377535-338f-bd70-8221-52a6509c58a4@g.nevcal.com> Message-ID: <556F2AB8-F595-4700-9663-FB9066489369@python.org> On May 7, 2019, at 03:52, Steve Holden wrote: > > What's not to like? My only complaint is that you steadfastly refuse use Guido?s time machine keys to make this available in 3.7. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From larry at hastings.org Tue May 7 10:21:51 2019 From: larry at hastings.org (Larry Hastings) Date: Tue, 7 May 2019 10:21:51 -0400 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> Message-ID: Guido just stopped by--we're all at the PyCon 2019 dev sprints--and we had a chat about it.? Guido likes it but wanted us to restore a little of the magical behavior we had in "!d": now, = in f-strings will default to repr (!r), /unless/ you specify a format spec. If you specify a format spec it will always default to format.? And naturally if you specify an explicit conversion function (!r !s !a) it will use that. This makes !f irrelevant, so we're removing it. Here's the thinking: 99% of the time the user will just use {foo=}, and for that you want repr.? After that, 0.99% of the time the user will want a format spec that applies directly the value. It's exceedingly unlikely that someone will want a format spec, but want it to apply to repr(value) and not value itself.? So that's /possible/ (f'{foo=!r:20}').? But the default behavior is the most common case at every step. "So, is it checked in?" --GvR, //arry/ On 5/6/19 8:39 PM, Eric V. Smith wrote: > Last fall Larry Hastings made a suggestion for adding a way to make > so-called "print-based debugging" easier with f-strings. Basically the > approach is that f-strings would be able to produce the text of the > expression and the value of that expression, without repeating the > expression in the f-sting. No more writing f'foo={foo}, bar={bar}'. > foo and bar should each only be in there once each! > > At PyCon US 2019 I did a lightning talk about this, suggesting the > syntax of !d, so that if foo="Hello", then f"{foo!d}" would produce > "foo='Hello'". That is, it's the text of the expression, followed by > an equal sign, followed by the repr of the expression. I have > implemented this and a PR exists. Arbitrary expressions are allowed. I > heard from core devs and end users after this talk, and all were > positive. > > After that lightning talk, Larry and I talked about it some more, and > for a number of reasons decided that it would make more sense if the > syntax used an = sign. So we came up with f"{foo=}", which would also > produce "foo='Hello'". > > The reasons for the change are: > - Having '=' in the expression is a better mnemonic than !d. > - By not using a conversion starting with !, we can compose = with the > existing ! conversions, !r, !s, and the rarely used !a. > - We can let the user have a little more control of the resulting string. > > Another thing I like about this approach over !d is that the patch is > simpler, because there are fewer special cases. And because there are > fewer special cases in the code, I think the proposal is easier to > explain than !d (in particular how it interacts (or doesn't!) with > format specifiers). > > There's a good rationale here, along with a PR: > https://bugs.python.org/issue36817. > > My plan is to commit this change before 3.8b1. If anyone would like to > discuss it at PyCon, I'll be around until about 10:30 am on Tuesday. > I'll be in the CPython sprint room, and I'll be watching bpo, too. > > Eric > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/larry%40hastings.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From lukasz at langa.pl Tue May 7 10:58:27 2019 From: lukasz at langa.pl (=?utf-8?Q?=C5=81ukasz_Langa?=) Date: Tue, 7 May 2019 10:58:27 -0400 Subject: [Python-Dev] [RELEASE] Python 3.8.0a4 is now available for testing Message-ID: <27210321-6B27-4629-9643-2A64F62DA85D@langa.pl> It's time for the LAST alpha of Python 3.8.0. Go get it here: https://www.python.org/downloads/release/python-380a4/ Python 3.8.0a4 is the fourth and final alpha release of Python 3.8, the next feature release of Python. During the alpha phase, Python 3.8 remains under heavy development: additional features will be added and existing features may be modified or deleted. Please keep in mind that this is a preview release and its use is not recommended for production environments. The first beta release, 3.8.0b1, is planned for 2019-05-31. The release has slipped a week because of me being overwhelmed with PyCon US this year. There was also a release blocker and a breaking change to ElementTree. Anyway, sorry for the wait! I moved the planned date of beta1 a few days to make up for it. If you have a feature you're working on and you'd like to see it in 3.8.0, NOW IS THE TIME TO ACT. Please don't wait until May 30th, get a proper review and land your change as soon as possible. Q: Can I get my feature in after that date if I ask nicely? A: Yes, of course. I will release it in Python 3.9. - ? -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From skip.montanaro at gmail.com Tue May 7 11:01:39 2019 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 7 May 2019 10:01:39 -0500 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: <556F2AB8-F595-4700-9663-FB9066489369@python.org> References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> <15377535-338f-bd70-8221-52a6509c58a4@g.nevcal.com> <556F2AB8-F595-4700-9663-FB9066489369@python.org> Message-ID: > My only complaint is that you steadfastly refuse use Guido?s time machine keys to make this available in 3.7. Wait a minute, Barry. You mean you don't already have an Emacs function to do the rewriting as a pre-save-hook? Skip From eric at trueblade.com Tue May 7 11:33:10 2019 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 7 May 2019 11:33:10 -0400 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: <556F2AB8-F595-4700-9663-FB9066489369@python.org> References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> <15377535-338f-bd70-8221-52a6509c58a4@g.nevcal.com> <556F2AB8-F595-4700-9663-FB9066489369@python.org> Message-ID: <7e6a8718-d74e-9f24-b0ae-5405f9d2e49d@trueblade.com> On 5/7/19 9:49 AM, Barry Warsaw wrote: > On May 7, 2019, at 03:52, Steve Holden wrote: >> >> What's not to like? > > My only complaint is that you steadfastly refuse use Guido?s time machine keys to make this available in 3.7. Open a feature request here: https://github.com/asottile/future-fstrings ! Eric From tim.peters at gmail.com Tue May 7 12:50:54 2019 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 7 May 2019 11:50:54 -0500 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> Message-ID: [Larry Hastings ] > Guido just stopped by--we're all at the PyCon 2019 dev sprints--and we had > a chat about it. Guido likes it but wanted us to restore a little of the magical > behavior we had in "!d": now, = in f-strings will default to repr (!r), unless > you specify a format spec. If you specify a format spec it will always default > to format. And naturally if you specify an explicit conversion function (!r !s !a) > it will use that. > > This makes !f irrelevant, so we're removing it. > > Here's the thinking: 99% of the time the user will just use {foo=}, and for that > you want repr. After that, 0.99% of the time the user will want a format spec > that applies directly the value. It's exceedingly unlikely that someone will > want a format spec, but want it to apply to repr(value) and not value itself. So > that's possible (f'{foo=!r:20}'). But the default behavior is the most common > case at every step. +1. Perfect! I'm one of the 0.99% who will frequently use this to display floats, and really wants them to show as "0.99%" rather than "0.9913499340289%". BTW, that Guido person has made enough decent contributions by now that I think he should be asked whether he wants to become a core dev! From eric at trueblade.com Tue May 7 12:59:52 2019 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 7 May 2019 12:59:52 -0400 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> Message-ID: <43D94CC8-4FEB-45E2-A141-C5C0DA9E8C65@trueblade.com> Hi, Tim. Your name came up frequently as a target for this use case. I think we?ve come up with a pretty good solution. Now if I could only convince my clients to upgrade from 2.7! I?ve heard about these f-string things, and I think I would enjoy using them. Eric > On May 7, 2019, at 12:50 PM, Tim Peters wrote: > > [Larry Hastings ] >> Guido just stopped by--we're all at the PyCon 2019 dev sprints--and we had >> a chat about it. Guido likes it but wanted us to restore a little of the magical >> behavior we had in "!d": now, = in f-strings will default to repr (!r), unless >> you specify a format spec. If you specify a format spec it will always default >> to format. And naturally if you specify an explicit conversion function (!r !s !a) >> it will use that. >> >> This makes !f irrelevant, so we're removing it. >> >> Here's the thinking: 99% of the time the user will just use {foo=}, and for that >> you want repr. After that, 0.99% of the time the user will want a format spec >> that applies directly the value. It's exceedingly unlikely that someone will >> want a format spec, but want it to apply to repr(value) and not value itself. So >> that's possible (f'{foo=!r:20}'). But the default behavior is the most common >> case at every step. > > +1. Perfect! I'm one of the 0.99% who will frequently use this to > display floats, and really wants them to show as "0.99%" rather than > "0.9913499340289%". > > BTW, that Guido person has made enough decent contributions by now > that I think he should be asked whether he wants to become a core dev! > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com From solipsis at pitrou.net Tue May 7 14:09:15 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 7 May 2019 20:09:15 +0200 Subject: [Python-Dev] PEP 574 ready for review In-Reply-To: References: <20190430101428.204457e6@fsol> Message-ID: <20190507200915.54a39451@fsol> Should I submit a PR to change the PEP status or would you like to do it? Regards Antoine. On Wed, 1 May 2019 12:49:24 -0700 Nick Coghlan wrote: > Thanks Antoine. > > As BDFL-Delegate I'm happy with this version of the PEP, so it's my > pleasure to accept it for inclusion in Python 3.8. > > Regards, > Nick. > From paul at ganssle.io Tue May 7 15:42:56 2019 From: paul at ganssle.io (Paul Ganssle) Date: Tue, 7 May 2019 15:42:56 -0400 Subject: [Python-Dev] Adding a tzidx cache to datetime Message-ID: Greetings all, I have one last feature request that I'd like added to datetime for Python 3.8, and this one I think could use some more discussion, the addition of a "time zone index cache" to the /datetime/ object. The rationale is laid out in detail in bpo-35723 . The general problem is that currently, /every/ invocation of utcoffset, tzname and dst needs to do full, independent calculations of the time zone offsets, even for time zones where the mapping is guaranteed to be stable because datetimes are immutable. I have a proof of concept implementation: PR #11529 . I'm envisioning that the `datetime` class will add a private `_tzidx` single-byte member (it seems that this does not increase the size of the datetime object, because it's just using an unused alignment byte). `datetime` will also add a `tzidx()` method, which will return `_tzidx` if it's been set and otherwise it will call `self.tzinfo.tzidx()`.? If `self.tzinfo.tzidx()` returns a number between 0 and 254 (inclusive), it sets `_tzidx` to this value. tzidx() then returns whatever self.tzinfo.tzidx() returned. The value of this is that as far as I can tell, nearly all non-trivial tzinfo implementations construct a list of possible offsets, and implement utcoffset(), tzname() and dst() by calculating an index into that list and returning it. There are almost always less than 255 distinct offsets. By adding this cache /on the datetime/, we're using a small amount of currently-unused memory to prevent unnecessary calculations about a given datetime. The feature is entirely opt-in, and has no downsides if it goes unused, and it makes it possible to write tzinfo implementations that are both lazy and as fast as the "eager calculation" mode that pytz uses (and that causes many problems for pytz's users). I have explored the idea of using an lru cache of some sort on the tzinfo object itself, but there are two problems with this: 1. Calculating the hash of a datetime calls .utcoffset(), which means that it is necessary to, at minimum, do a `replace` on the datetime (and constructing a new datetime is a pretty considerable speed hit) 2. It will be a much bigger memory cost, since my current proposal uses approximately zero additional memory (not sure if the alignment stuff is platform-dependent or something, but it doesn't use additional memory on my linux computer). I realize this proposal is somewhat difficult to wrap your head around, so if anyone would like to chat with me about it in person, I'll be at PyCon sprints until Thursday morning. Best, Paul -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From robb at datalogics.com Tue May 7 16:09:34 2019 From: robb at datalogics.com (Rob Boehne) Date: Tue, 7 May 2019 20:09:34 +0000 Subject: [Python-Dev] Can't build 3.7.3 on AIX Message-ID: Hello, I?m attempting to get a recent version of Python3 running on my UNIX build machines, but having a problem under AIX. This seems like it might be since 3.6.5, but I?ve not actually attempted anything else other than 3.6.8 which fails similarly. I?m building with the native C compiler in 64-bit mode, using OBJECT_MODE=64 exported in the environment. I have attached config.log. Is this a known issue? TIA - Rob ../../Python-3.7.3/config.guess powerpc-ibm-aix7.1.4.0 ../../Python-3.7.3/Modules/makexp_aix Modules/python.exp . libpython3.7m.a; xlc_r -L/raid/checkouts-raid/robb/zlib-dl/Release/rs6000aix_64/lib -Wl,-blibpath:/opt/IBM/xlmass/8.1.3/lib/aix61:/opt/IBM/xlc/13.1.3/lib:/usr/lib:/lib:/opt/Python-3.7/lib -q64 -Wl,-bE:Modules/python.exp -lld -o python Programs/python.o libpython3.7m.so -lintl -ldl -lm LIBPATH=/raid/checkouts-raid/robb/nepal/build-py37 ./python -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi Fatal Python error: _Py_HashRandomization_Init: failed to get random numbers to initialize Python generate-posix-vars failed Makefile:603: recipe for target 'pybuilddir.txt' failed gmake: *** [pybuilddir.txt] Error 1 [cid:image002.png at 01D3D0E3.DCFE6710] Rob Boehne Senior Software Architect | Datalogics, Inc. +1.312.853.8351 | robb at datalogics.com datalogics.com | blogs.datalogics.com Connect with us: Facebook | Twitter | LinkedIn | YouTube -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 124449 bytes Desc: image001.png URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: config.log Type: application/octet-stream Size: 1065198 bytes Desc: config.log URL: From bbk1524 at gmail.com Tue May 7 15:47:52 2019 From: bbk1524 at gmail.com (Ben Kane) Date: Tue, 7 May 2019 15:47:52 -0400 Subject: [Python-Dev] Hello Message-ID: My name is Ben Kane. I'm joining this mailing list and introducing myself as asked in the "welcome to python-dev" email. I've been using Python for maybe 7 years now. I started to learn it in college to help with accounting homework and I continue to use it now for personal projects ranging from small scripts to larger applications and in my day job as a LinkedIn Site Reliability Engineer. I'd like to contribute back to the language that's helped me so much; and talking with core devs at PyCon 2019 has really encouraged me. So I'll be lurking and trying to pick things up as I see them. Thanks! -- Ben Kane -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 7 16:26:17 2019 From: christian at python.org (Christian Heimes) Date: Tue, 7 May 2019 16:26:17 -0400 Subject: [Python-Dev] PEP 578 accepted Message-ID: <2f1e4bef-d845-c006-537f-d81abc4c980c@python.org> Hi, Steve and I worked on the final touch of the PEP during PyCon. PEP 578 [1] is now ready. As the BDFL delegate I'm honoured to accept Steve's PEP. There is a very slim possibility that the PEP implementation might need adjustment for Linux. I haven't got feedback from a Red Hat engineer that works on Linux Kernel Auditing team. Everybody is busy with RHEL 8 release and the Red Hat Summit in Boston. I don't anticipate any changes to the PEP itself. After some consideration I decided to accept PEP now to give Steve enough time to push and test his implementation. There is sufficient time to address issues until the first release candidate of 3.8.0. Thanks Steve, it's been a privilege to work with you! Christian [1] https://www.python.org/dev/peps/pep-0578/ From vstinner at redhat.com Tue May 7 16:42:20 2019 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 7 May 2019 16:42:20 -0400 Subject: [Python-Dev] Can't build 3.7.3 on AIX In-Reply-To: References: Message-ID: Please report the bug at https://bugs.python.org/ ;-) I'm not sure how _PyOS_URandom() / pyurandom() is implemented on AIX. I guess that it reads from /dev/urandom. Please attach config.log to your bug report. Victor Le mar. 7 mai 2019 ? 16:30, Rob Boehne a ?crit : > Hello, > > > > I?m attempting to get a recent version of Python3 running on my UNIX build > machines, but having a problem under AIX. > > This seems like it might be since 3.6.5, but I?ve not actually attempted > anything else other than 3.6.8 which fails similarly. > > I?m building with the native C compiler in 64-bit mode, using > OBJECT_MODE=64 exported in the environment. I have attached config.log. > > Is this a known issue? TIA - Rob > > > > > > ../../Python-3.7.3/config.guess > > powerpc-ibm-aix7.1.4.0 > > > > > > ../../Python-3.7.3/Modules/makexp_aix Modules/python.exp . libpython3.7m.a; > xlc_r -L/raid/checkouts-raid/robb/zlib-dl/Release/rs6000aix_64/lib -Wl,-blibpath:/opt/IBM/xlmass/8.1.3/lib/aix61:/opt/IBM/xlc/13.1.3/lib:/usr/lib:/lib:/opt/Python-3.7/lib > -q64 -Wl,-bE:Modules/python.exp -lld -o python Programs/python.o > libpython3.7m.so -lintl -ldl -lm > > LIBPATH=/raid/checkouts-raid/robb/nepal/build-py37 ./python -E -S -m > sysconfig --generate-posix-vars ;\ > > if test $? -ne 0 ; then \ > > echo "generate-posix-vars failed" ; \ > > rm -f ./pybuilddir.txt ; \ > > exit 1 ; \ > > fi > > Fatal Python error: _Py_HashRandomization_Init: failed to get random > numbers to initialize Python > > > > generate-posix-vars failed > > Makefile:603: recipe for target 'pybuilddir.txt' failed > > gmake: *** [pybuilddir.txt] Error 1 > > > > > > > > [image: cid:image002.png at 01D3D0E3.DCFE6710] > > *Rob Boehne* > > Senior Software Architect | Datalogics, Inc. > > +1.312.853.8351 <(312)%20853-8351> | robb at datalogics.com > > datalogics.com | blogs.datalogics.com > > Connect with us: Facebook | > Twitter | LinkedIn > > | YouTube > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > -- Night gathers, and now my watch begins. It shall not end until my death. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 124449 bytes Desc: not available URL: From vstinner at redhat.com Tue May 7 16:47:30 2019 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 7 May 2019 16:47:30 -0400 Subject: [Python-Dev] Hello In-Reply-To: References: Message-ID: Welcome Ben! Le mar. 7 mai 2019 ? 16:28, Ben Kane a ?crit : > My name is Ben Kane. I'm joining this mailing list and introducing myself as asked in the "welcome to python-dev" email. I've been using Python for maybe 7 years now. I started to learn it in college to help with accounting homework and I continue to use it now for personal projects ranging from small scripts to larger applications and in my day job as a LinkedIn Site Reliability Engineer. > > I'd like to contribute back to the language that's helped me so much; and talking with core devs at PyCon 2019 has really encouraged me. So I'll be lurking and trying to pick things up as I see them. python-dev is not the proper mail to introduce yourself to contribute. I suggest you to start reading http://devguide.python.org/ and please send the same email to core-mentorship mailing list: https://mail.python.org/mailman3/lists/core-mentorship.python.org/?x=22 It would help if you can elaborate on which parts of Python you would be interested to contribute ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death. From vstinner at redhat.com Tue May 7 16:59:49 2019 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 7 May 2019 16:59:49 -0400 Subject: [Python-Dev] PEP 574 ready for review In-Reply-To: <20190507200915.54a39451@fsol> References: <20190430101428.204457e6@fsol> <20190507200915.54a39451@fsol> Message-ID: You can do it. I like when the acceptance email is linked from the PEP ;-) Victor Le mar. 7 mai 2019 ? 14:12, Antoine Pitrou a ?crit : > > > Should I submit a PR to change the PEP status or would you like to do > it? > > Regards > > Antoine. > > > On Wed, 1 May 2019 12:49:24 -0700 > Nick Coghlan wrote: > > Thanks Antoine. > > > > As BDFL-Delegate I'm happy with this version of the PEP, so it's my > > pleasure to accept it for inclusion in Python 3.8. > > > > Regards, > > Nick. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com -- Night gathers, and now my watch begins. It shall not end until my death. From robb at datalogics.com Tue May 7 17:03:17 2019 From: robb at datalogics.com (Rob Boehne) Date: Tue, 7 May 2019 21:03:17 +0000 Subject: [Python-Dev] Can't build 3.7.3 on AIX In-Reply-To: References: Message-ID: <5038F465-FB30-4FA5-8B58-1A5CBE1B79B2@datalogics.com> I?ve created the issue, but was unable to attach config.log ? 413 Request Entity Too Large Kept coming up if I attempted to attach the log file. https://bugs.python.org/issue36843 From: Victor Stinner Date: Tuesday, May 7, 2019 at 3:42 PM To: Rob Boehne Cc: "python-dev at python.org" Subject: Re: [Python-Dev] Can't build 3.7.3 on AIX Please report the bug at https://bugs.python.org/ ;-) I'm not sure how _PyOS_URandom() / pyurandom() is implemented on AIX. I guess that it reads from /dev/urandom. Please attach config.log to your bug report. Victor Le mar. 7 mai 2019 ? 16:30, Rob Boehne > a ?crit : Hello, I?m attempting to get a recent version of Python3 running on my UNIX build machines, but having a problem under AIX. This seems like it might be since 3.6.5, but I?ve not actually attempted anything else other than 3.6.8 which fails similarly. I?m building with the native C compiler in 64-bit mode, using OBJECT_MODE=64 exported in the environment. I have attached config.log. Is this a known issue? TIA - Rob ../../Python-3.7.3/config.guess powerpc-ibm-aix7.1.4.0 ../../Python-3.7.3/Modules/makexp_aix Modules/python.exp . libpython3.7m.a; xlc_r -L/raid/checkouts-raid/robb/zlib-dl/Release/rs6000aix_64/lib -Wl,-blibpath:/opt/IBM/xlmass/8.1.3/lib/aix61:/opt/IBM/xlc/13.1.3/lib:/usr/lib:/lib:/opt/Python-3.7/lib -q64 -Wl,-bE:Modules/python.exp -lld -o python Programs/python.o libpython3.7m.so -lintl -ldl -lm LIBPATH=/raid/checkouts-raid/robb/nepal/build-py37 ./python -E -S -m sysconfig --generate-posix-vars ;\ if test $? -ne 0 ; then \ echo "generate-posix-vars failed" ; \ rm -f ./pybuilddir.txt ; \ exit 1 ; \ fi Fatal Python error: _Py_HashRandomization_Init: failed to get random numbers to initialize Python generate-posix-vars failed Makefile:603: recipe for target 'pybuilddir.txt' failed gmake: *** [pybuilddir.txt] Error 1 [cid:image002.png at 01D3D0E3.DCFE6710] Rob Boehne Senior Software Architect | Datalogics, Inc. +1.312.853.8351 | robb at datalogics.com datalogics.com | blogs.datalogics.com Connect with us: Facebook | Twitter | LinkedIn | YouTube _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com -- Night gathers, and now my watch begins. It shall not end until my death. -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: image001.png Type: image/png Size: 124450 bytes Desc: image001.png URL: From jordan.m.adler at gmail.com Tue May 7 17:05:57 2019 From: jordan.m.adler at gmail.com (Jordan Adler) Date: Tue, 7 May 2019 17:05:57 -0400 Subject: [Python-Dev] Definition of equality check behavior Message-ID: Hey folks! Through the course of work on the future polyfills that mimic the behavior of Py3 builtins across versions of Python, we've discovered that the equality check behavior of at least some builtin types do not match the documented core data model . Specifically, a comparison between a primitive (int, str, float were tested) and an object of a different type always return False, instead of raising a NotImplementedError. Consider `1 == '1'` as a test case. Should the data model be adjusted to declare that primitive types are expected to fallback to False, or should the cpython primitive type's __eq__ implementation fallback to raise NotImplementedError? Reasonable people could disagree about the right approach, but my distaste for silent failures leads me to recommend that the implementation be adjusted to return NotImplementedError as a fallback, and to document that the operands should not be coerced to the same type prior to comparison (enforcing a stricter equality check). This will of course require a deprecation protocol. Alternatively some new equality operator could be used to specify the level of coercion/type checking desired (currently Python has 'is' and '=='). Jordan -------------- next part -------------- An HTML attachment was scrubbed... URL: From max.vogler at gmail.com Tue May 7 18:05:07 2019 From: max.vogler at gmail.com (Max Vogler) Date: Wed, 8 May 2019 00:05:07 +0200 Subject: [Python-Dev] Definition of equality check behavior In-Reply-To: References: Message-ID: Thanks for starting this discussion and addressing my issue, Jordan. I would like to clarify the intentions behind my original issue : It does not concern coercion in Python's __eq__. Instead, the issue concerns the return of False instead of NotImplemented in future.types.newstr.__eq__. As a result, in special cases this breaks the symmetry of equality. Also, please be aware of the difference between NotImplemented (a singleton that is used as indicator for an unimplemented equality operation) and NotImplementedError , a RuntimeError used for example in abstract classes. To quote from the docs: > NotImplementedError and NotImplemented are not interchangeable, even though they have similar names and purposes. On Tue, May 7, 2019 at 11:06 PM Jordan Adler wrote: > Hey folks! > > Through the course of work on the future polyfills > that mimic > the behavior of Py3 builtins across versions of Python, we've discovered > that the equality check behavior of at least some builtin types do not > match the documented core data model > . > > Specifically, a comparison between a primitive (int, str, float were > tested) and an object of a different type always return False, instead of > raising a NotImplementedError. Consider `1 == '1'` as a test case. > > Should the data model be adjusted to declare that primitive types are > expected to fallback to False, or should the cpython primitive type's > __eq__ implementation fallback to raise NotImplementedError? > > Reasonable people could disagree about the right approach, but my distaste > for silent failures leads me to recommend that the implementation be > adjusted to return NotImplementedError as a fallback, and to document that > the operands should not be coerced to the same type prior to comparison > (enforcing a stricter equality check). This will of course require a > deprecation protocol. > > Alternatively some new equality operator could be used to specify the > level of coercion/type checking desired (currently Python has 'is' and > '=='). > > Jordan > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 7 19:17:24 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 8 May 2019 09:17:24 +1000 Subject: [Python-Dev] Definition of equality check behavior In-Reply-To: References: Message-ID: <20190507231724.GW5010@ando.pearwood.info> On Tue, May 07, 2019 at 05:05:57PM -0400, Jordan Adler wrote: [...] > Specifically, a comparison between a primitive (int, str, float were > tested) and an object of a different type always return False, instead of > raising a NotImplementedError. Consider `1 == '1'` as a test case. I think you may be labouring under a few misapprehensions here. 1. Comparisons between builtins such as ints and objects of different types do not always return False: py> class X: ... def __eq__(self, other): ... return True ... py> 123 == X() True You don't need a custom class to demonstrate this fact, you just need values which actually are equal: py> 123 == 1.23e2 # int compared to float True 2. Comparisons are not supposed to raise NotImplementedError as part of the core data model, they are supposed to return (not raise) NotImplemented. Note that NotImplementedError is a completely different thing). As the documentation you linked to says: A rich comparison method may return the singleton NotImplemented if it does not implement the operation for a given pair of arguments. 3. Equality does not suppress exceptions and lead to silent failure: py> class Y: ... def __eq__(self, other): ... raise NotImplementedError ... py> 123 == Y() Traceback (most recent call last): File "", line 1, in File "", line 3, in __eq__ NotImplementedError Hope this helps. -- Steven From steve at pearwood.info Tue May 7 19:30:09 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 8 May 2019 09:30:09 +1000 Subject: [Python-Dev] Definition of equality check behavior In-Reply-To: References: Message-ID: <20190507233009.GX5010@ando.pearwood.info> On Wed, May 08, 2019 at 12:05:07AM +0200, Max Vogler wrote: > Thanks for starting this discussion and addressing my issue, Jordan. > > I would like to clarify the intentions behind my original issue > : It does not > concern coercion in Python's __eq__. Instead, the issue concerns the return > of False instead of NotImplemented in future.types.newstr.__eq__. Unless there is something special about newstr that it should force an unequal result, it should return NotImplemented when comparing to types it knows nothing about. I'm reluctant to outright declare a class I know nothing about in a library I don't understand is buggy, but on the face of it, if newstr.__eq__ doesn't return NotImplemented, it probably is buggy. If so, newstr is in good company: failure to return NotImplemented is a very common bug in third-party code. I've written heaps of classes that wrongly force a False result. But I've learned better and don't do it any more :-) -- Steven From ethan at stoneleaf.us Tue May 7 20:23:09 2019 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 7 May 2019 17:23:09 -0700 Subject: [Python-Dev] Definition of equality check behavior In-Reply-To: References: Message-ID: <945db57c-3b27-0d2f-f8c5-c10e69fd3a0c@stoneleaf.us> On 05/07/2019 02:05 PM, Jordan Adler wrote: > Specifically, a comparison between a primitive (int, str, float were > tested) and an object of a different type always return False, > instead of raising a NotImplementedError. Consider `1 == '1'` as a > test case. If the object of a different type doesn't support comparing to an int, str, or float then False is the correct answer. On the other hand, if the object of a different type wants to compare equal to, say, ints then it will have to supply its own __eq__ method to return False/True as appropriate. > Should the data model be adjusted to declare that primitive types are > expected to fallback to False No, because they aren't. > or should the cpython primitive type's __eq__ implementation fallback > to raise NotImplementedError? No, because raising an error is not appropriate. Did you mean `return NotImplemented`? Because empirical evidence suggests that they do: ------- class MyCustomInt(): def __init__(self, value): self.value = value def __eq__(self, other): if isinstance(other, int): return self.value == other else: return NotImplemented def __ne__(self, other): if isinstance(other, int): return self.value != other else: return NotImplemented core_int = 7 my_int = MyCustomInt(7) print(core_int == my_int) # True print(my_int == core_int) # True ------- If the core types were not returning NotImplemented then the above would be False on the `core_int == my_int` line. Hopefully this is clearer now? -- ~Ethan~ From steve at pearwood.info Tue May 7 20:39:37 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 8 May 2019 10:39:37 +1000 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> Message-ID: <20190508003937.GA5010@ando.pearwood.info> Disclaimer: this topic seems to have been split over at least two issues on the bug tracker, a Python-Ideas thread from 2018, Discourse (I think...) and who knows what else. I haven't read it all, so excuse me if I'm raising something already discussed. On Mon, May 06, 2019 at 08:39:41PM -0400, Eric V. Smith wrote: > After that lightning talk, Larry and I talked about it some more, and > for a number of reasons decided that it would make more sense if the > syntax used an = sign. So we came up with f"{foo=}", which would also > produce "foo='Hello'". > > The reasons for the change are: > - Having '=' in the expression is a better mnemonic than !d. > - By not using a conversion starting with !, we can compose = with the > existing ! conversions, !r, !s, and the rarely used !a. > - We can let the user have a little more control of the resulting string. You're going to hate me for bike-shedding, but I really don't like using = as a defacto unary postfix operator. To me, spam= is always going to look like it was meant to be spam== and the second half got accidentally deleted. I don't have a better suggestion, sorry. In an earlier draft, back when this was spelled !d, you specifically talked about whitespace. Does this still apply? spam = 42 f'{spam=}' # returns 'spam=42' f'{spam =}' # returns 'spam =42' f'{spam = }' # returns 'spam = 42' I guess? f'{spam+1=}' # returns 'spam+1=41' I guess? -- Steven From bbk1524 at gmail.com Tue May 7 21:08:41 2019 From: bbk1524 at gmail.com (Ben Kane) Date: Tue, 7 May 2019 21:08:41 -0400 Subject: [Python-Dev] Hello In-Reply-To: References: Message-ID: Sorry Victor, I must have misinterpreted this snippet from the mailbot. Maybe it would be a good idea to re-word it? I'll be happy to open an issue for that in an appropriate mailing list if you think that would help. > Welcome to the Python-Dev at python.org mailing list! If you are a new subscriber, please take the time to introduce yourself briefly in your first post. It is appreciated if you lurk around for a while before posting! :-) I talked to Barry Warsaw about https://bugs.python.org/issue36837 and he said I should post to python-dev to get feedback. I'll work on that email and post back here once I'm happy with it. Thanks, Ben On Tue, May 7, 2019 at 4:47 PM Victor Stinner wrote: > Welcome Ben! > > Le mar. 7 mai 2019 ? 16:28, Ben Kane a ?crit : > > My name is Ben Kane. I'm joining this mailing list and introducing > myself as asked in the "welcome to python-dev" email. I've been using > Python for maybe 7 years now. I started to learn it in college to help with > accounting homework and I continue to use it now for personal projects > ranging from small scripts to larger applications and in my day job as a > LinkedIn Site Reliability Engineer. > > > > I'd like to contribute back to the language that's helped me so much; > and talking with core devs at PyCon 2019 has really encouraged me. So I'll > be lurking and trying to pick things up as I see them. > > python-dev is not the proper mail to introduce yourself to contribute. > I suggest you to start reading http://devguide.python.org/ and please > send the same email to core-mentorship mailing list: > https://mail.python.org/mailman3/lists/core-mentorship.python.org/?x=22 > > It would help if you can elaborate on which parts of Python you would > be interested to contribute ;-) > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > -- Ben Kane -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue May 7 21:23:35 2019 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 7 May 2019 21:23:35 -0400 Subject: [Python-Dev] Easier debugging with f-strings In-Reply-To: <20190508003937.GA5010@ando.pearwood.info> References: <133ec1d8-cee5-a287-b786-304a7ed0789f@trueblade.com> <20190508003937.GA5010@ando.pearwood.info> Message-ID: On 5/7/19 8:39 PM, Steven D'Aprano wrote: > Disclaimer: this topic seems to have been split over at least two issues > on the bug tracker, a Python-Ideas thread from 2018, Discourse (I > think...) and who knows what else. I haven't read it all, so excuse me > if I'm raising something already discussed. Yeah, it's all over the place. At this stage, I think discussing it here is best. > > On Mon, May 06, 2019 at 08:39:41PM -0400, Eric V. Smith wrote: > >> After that lightning talk, Larry and I talked about it some more, and >> for a number of reasons decided that it would make more sense if the >> syntax used an = sign. So we came up with f"{foo=}", which would also >> produce "foo='Hello'". >> >> The reasons for the change are: >> - Having '=' in the expression is a better mnemonic than !d. >> - By not using a conversion starting with !, we can compose = with the >> existing ! conversions, !r, !s, and the rarely used !a. >> - We can let the user have a little more control of the resulting string. > > You're going to hate me for bike-shedding, but I really don't like using > = as a defacto unary postfix operator. To me, spam= is always going to > look like it was meant to be spam== and the second half got > accidentally deleted. > > I don't have a better suggestion, sorry. I think = makes a great mnemonic for what's happening. I realize it looks a little funny at first, but just wait until you start confusing it with := (hah!). Everyone else I've discussed it with likes =. > In an earlier draft, back when this was spelled !d, you specifically > talked about whitespace. Does this still apply? Yes, it does. > spam = 42 > f'{spam=}' # returns 'spam=42' > f'{spam =}' # returns 'spam =42' > f'{spam = }' # returns 'spam = 42' I guess? All correct. > f'{spam+1=}' # returns 'spam+1=41' I guess? I'm not proposing to redefine addition by one, so this is actually 'spam+1=43' :) Eric From tim.peters at gmail.com Tue May 7 21:23:35 2019 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 7 May 2019 20:23:35 -0500 Subject: [Python-Dev] Definition of equality check behavior In-Reply-To: References: Message-ID: [Jordan Adler ] > Through the course of work on the future polyfills that mimic the behavior > of Py3 builtins across versions of Python, we've discovered that the > equality check behavior of at least some builtin types do not match the > documented core data model. > > Specifically, a comparison between a primitive (int, str, float were tested) > and an object of a different type always return False, instead of raising > a NotImplementedError. Consider `1 == '1'` as a test case. > > Should the data model be adjusted to declare that primitive types are > expected to fallback to False, or should the cpython primitive type's > __eq__ implementation fallback to raise NotImplementedError? Nope ;-) This isn't a "data model" issue. Look instead at the Standard Library manual's section on Built-In Types, under heading Comparisons: """ Objects of different types, except different numeric types, never compare equal. ... The <, <=, > and >= operators will raise a TypeError exception when comparing a complex number with another built-in numeric type, when the objects are of different types that cannot be compared, or in other cases where there is no defined ordering. """ It's not an object's responsibility to arrange for that. It's done for them by default, and objects only need to supply their own rich comparison methods if they don't want the defaults. For example, when comparing an int with another type, all the int rich comparison methods _do_ return NotImplemented: >>> f = 4 >>> f.__eq__("abc") NotImplemented It's at a higher level that comparison logic says "OK, I gave both comparands a chance, and they both returned NotImplemented. So one last chance (from object.c's do_richcompare())": /* If neither object implements it, provide a sensible default for == and !=, but raise an exception for ordering. */ switch (op) { case Py_EQ: res = (v == w) ? Py_True : Py_False; break; case Py_NE: res = (v != w) ? Py_True : Py_False; break; default: PyErr_Format(PyExc_TypeError, "'%s' not supported between instances of '%.100s' and '%.100s'", opstrings[op], v->ob_type->tp_name, w->ob_type->tp_name); return NULL; } Then the Py_EQ case of that delivers: >>> f == "abc" False and the Py_NE case: >>> f != "abc" True despite that (or because of that ;-) ): >>> f.__eq__("abc") NotImplemented >>> "abc".__eq__(f) NotImplemented Note that there's nothing special about builtin types here. All types are treated alike. From mark at hotpy.org Wed May 8 10:07:46 2019 From: mark at hotpy.org (Mark Shannon) Date: Wed, 8 May 2019 10:07:46 -0400 Subject: [Python-Dev] Redoing failed PR checks Message-ID: Hi, How do I redo a failed PR check? The appveyor failure for https://github.com/python/cpython/pull/13181 appears to be spurious, but there is no obvious way to redo it. BTW, this is not the first time I've seen a PR blocked by a spurious appveyor failure. Cheers, Mark. From encukou at gmail.com Wed May 8 10:13:58 2019 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 8 May 2019 10:13:58 -0400 Subject: [Python-Dev] Hello In-Reply-To: References: Message-ID: <76bd3d71-ac71-727d-7748-46d5aa4c0cd1@gmail.com> On 5/7/19 9:08 PM, Ben Kane wrote: > Sorry Victor, I must have misinterpreted this snippet from the mailbot. > Maybe it would be a good idea to re-word it? I'll be happy to open an > issue for that in an appropriate mailing list if you think that would help. > > > Welcome to the Python-Dev at python.org > mailing list! If you are a new subscriber, please take the time to > introduce yourself briefly in your first post. It is appreciated if you > lurk around for a while before posting! :-) Not sure what the original intent was, but I read that as: "Whenever you get to sending your first post, include an introduction in it. But, before you start actively participating, take your time to hang around and absorb the culture." Perhaps we want to reverse the two sentences to put them in chronological order? I'm not sure what the proper forum for this discussion would be. > I talked to Barry Warsaw about https://bugs.python.org/issue36837 and he > said I should post to python-dev to get feedback. I'll work on that > email and post back here once I'm happy with it. > > Thanks, > Ben > > > On Tue, May 7, 2019 at 4:47 PM Victor Stinner > wrote: > > Welcome Ben! > > Le mar. 7 mai 2019 ? 16:28, Ben Kane > a ?crit : > > My name is Ben Kane. I'm joining this mailing list and > introducing myself as asked in the "welcome to python-dev" email. > I've been using Python for maybe 7 years now. I started to learn it > in college to help with accounting homework and I continue to use it > now for personal projects ranging from small scripts to larger > applications and in my day job as a LinkedIn Site Reliability Engineer. > > > > I'd like to contribute back to the language that's helped me so > much; and talking with core devs at PyCon 2019 has really encouraged > me. So I'll be lurking and trying to pick things up as I see them. > > python-dev is not the proper mail to introduce yourself to contribute. > I suggest you to start reading http://devguide.python.org/ and please > send the same email to core-mentorship mailing list: > https://mail.python.org/mailman3/lists/core-mentorship.python.org/?x=22 > > It would help if you can elaborate on which parts of Python you would > be interested to contribute ;-) > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > > > > -- > > Ben Kane > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com > From eric at trueblade.com Wed May 8 10:23:35 2019 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 8 May 2019 10:23:35 -0400 Subject: [Python-Dev] Redoing failed PR checks In-Reply-To: References: Message-ID: I think you can close and reopen the PR. That?s what I?m trying on my blocked PR. -- Eric V. Smith True Blade Systems, Inc (301) 859-4544 > On May 8, 2019, at 10:07 AM, Mark Shannon wrote: > > Hi, > > How do I redo a failed PR check? > The appveyor failure for https://github.com/python/cpython/pull/13181 appears to be spurious, but there is no obvious way to redo it. > > BTW, this is not the first time I've seen a PR blocked by a spurious appveyor failure. > > Cheers, > Mark. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com From barry at python.org Wed May 8 10:25:50 2019 From: barry at python.org (Barry Warsaw) Date: Wed, 8 May 2019 10:25:50 -0400 Subject: [Python-Dev] Hello In-Reply-To: <76bd3d71-ac71-727d-7748-46d5aa4c0cd1@gmail.com> References: <76bd3d71-ac71-727d-7748-46d5aa4c0cd1@gmail.com> Message-ID: <18245C2E-EC7B-48D0-8320-05C15E5C27D3@python.org> On May 8, 2019, at 10:13, Petr Viktorin wrote: > > Not sure what the original intent was, but I read that as: "Whenever you get to sending your first post, include an introduction in it. But, before you start actively participating, take your time to hang around and absorb the culture." > > Perhaps we want to reverse the two sentences to put them in chronological order? > > I'm not sure what the proper forum for this discussion would be. Here or -committers. Then postmasters to list owner to change the welcome text. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From levkivskyi at gmail.com Wed May 8 10:39:02 2019 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Wed, 8 May 2019 15:39:02 +0100 Subject: [Python-Dev] Definition of equality check behavior In-Reply-To: References: Message-ID: On Tue, 7 May 2019 at 22:31, Jordan Adler wrote: > Hey folks! > > Through the course of work on the future polyfills > that mimic > the behavior of Py3 builtins across versions of Python, we've discovered > that the equality check behavior of at least some builtin types do not > match the documented core data model > . > > Specifically, a comparison between a primitive (int, str, float were > tested) and an object of a different type always return False, instead of > raising a NotImplementedError. Consider `1 == '1'` as a test case. > > Should the data model be adjusted to declare that primitive types are > expected to fallback to False, or should the cpython primitive type's > __eq__ implementation fallback to raise NotImplementedError? > > Reasonable people could disagree about the right approach, but my distaste > for silent failures leads me to recommend that the implementation be > adjusted to return NotImplementedError as a fallback, and to document that > the operands should not be coerced to the same type prior to comparison > (enforcing a stricter equality check). This will of course require a > deprecation protocol. > > Alternatively some new equality operator could be used to specify the > level of coercion/type checking desired (currently Python has 'is' and > '=='). > I don't think there is a chance this can be changed at runtime. OTOH, mypy has a (pretty recent, so better use master) flag --strict-equality that uses some heuristics to detect suspicious comparisons, identity checks, and container checks (all of these may return False at runtime while some people want them to be an error, e.g. b'abc' == 'abc', '1' in [1, 2, 3], etc). -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Wed May 8 11:36:45 2019 From: larry at hastings.org (Larry Hastings) Date: Wed, 8 May 2019 11:36:45 -0400 Subject: [Python-Dev] Farewell, Python 3.4 Message-ID: It's with a note of sadness that I announce the final retirement of Python 3.4.? The final release was back in March, but I didn't get around to actually closing and deleting the 3.4 branch until this morning. Python 3.4 introduced many features we all enjoy in modern Python--the asyncio, ensurepip, and enum packages, just to name three.? It's a release I hope we all remember fondly. My eternal thanks to all the members of the release team that worked on Python 3.4: Georg Brandl Julien Palard Martin von L?wis Ned Deily Steve Dower Terry Reedy and all the engineers of the Python infrastructure team. Special thanks to Benjamin Peterson and Ned Deily, who frequently scurried around behind the scenes cleaning up the messes I cluelessly left in my wake. Having closed 3.4, I am now retired as Python 3.4 Release Manager.? I regret to inform all of you that you're still stuck with me as Python 3.5 Release Manager until sometime next year. My very best wishes, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Wed May 8 11:46:05 2019 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 8 May 2019 17:46:05 +0200 Subject: [Python-Dev] Farewell, Python 3.4 In-Reply-To: References: Message-ID: Thank you for having been 3.4 release manager, Larry ! On 08.05.2019 17:36, Larry Hastings wrote: > > It's with a note of sadness that I announce the final retirement of > Python 3.4.? The final release was back in March, but I didn't get > around to actually closing and deleting the 3.4 branch until this morning. > > Python 3.4 introduced many features we all enjoy in modern Python--the > asyncio, ensurepip, and enum packages, just to name three.? It's a > release I hope we all remember fondly. > > My eternal thanks to all the members of the release team that worked on > Python 3.4: > > Georg Brandl > > Julien Palard > > Martin von L?wis > > Ned Deily > > Steve Dower > > Terry Reedy > > and all the engineers of the Python infrastructure team. > > Special thanks to Benjamin Peterson and Ned Deily, who frequently > scurried around behind the scenes cleaning up the messes I cluelessly > left in my wake. > > Having closed 3.4, I am now retired as Python 3.4 Release Manager.? I > regret to inform all of you that you're still stuck with me as Python > 3.5 Release Manager until sometime next year. > > > My very best wishes, > > > //arry/ > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 08 2019) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From jordan.m.adler at gmail.com Wed May 8 11:51:58 2019 From: jordan.m.adler at gmail.com (Jordan Adler) Date: Wed, 8 May 2019 11:51:58 -0400 Subject: [Python-Dev] Definition of equality check behavior In-Reply-To: References: Message-ID: Ahh, I didn't locate the documentation on the NotImplemented constant as I was checking 2.7's docs, so I wasn't aware of the layer of indirection and fallback behavior for the operator. Sorry about that! Based on that, the right call here is to adjust newstr to return that singleton where appropriate. On Tue, May 7, 2019 at 9:23 PM Tim Peters wrote: > [Jordan Adler ] > > Through the course of work on the future polyfills that mimic the > behavior > > of Py3 builtins across versions of Python, we've discovered that the > > equality check behavior of at least some builtin types do not match the > > documented core data model. > > > > Specifically, a comparison between a primitive (int, str, float were > tested) > > and an object of a different type always return False, instead of raising > > a NotImplementedError. Consider `1 == '1'` as a test case. > > > > Should the data model be adjusted to declare that primitive types are > > expected to fallback to False, or should the cpython primitive type's > > __eq__ implementation fallback to raise NotImplementedError? > > Nope ;-) This isn't a "data model" issue. Look instead at the > Standard Library manual's section on Built-In Types, under heading > Comparisons: > > """ > Objects of different types, except different numeric types, never > compare equal. ... > The <, <=, > and >= operators will raise a TypeError exception when > comparing a complex number with another built-in numeric type, when > the objects are of different types that cannot be compared, or in > other cases where there is no defined ordering. > """ > > It's not an object's responsibility to arrange for that. It's done > for them by default, and objects only need to supply their own rich > comparison methods if they don't want the defaults. For example, when > comparing an int with another type, all the int rich comparison > methods _do_ return NotImplemented: > > >>> f = 4 > >>> f.__eq__("abc") > NotImplemented > > It's at a higher level that comparison logic says "OK, I gave both > comparands a chance, and they both returned NotImplemented. So one > last chance (from object.c's do_richcompare())": > > /* If neither object implements it, provide a sensible default > for == and !=, but raise an exception for ordering. */ > switch (op) { > case Py_EQ: > res = (v == w) ? Py_True : Py_False; > break; > case Py_NE: > res = (v != w) ? Py_True : Py_False; > break; > default: > PyErr_Format(PyExc_TypeError, > "'%s' not supported between instances of '%.100s' > and '%.100s'", > opstrings[op], > v->ob_type->tp_name, > w->ob_type->tp_name); > return NULL; > } > > Then the Py_EQ case of that delivers: > > >>> f == "abc" > False > > and the Py_NE case: > > >>> f != "abc" > True > > despite that (or because of that ;-) ): > > >>> f.__eq__("abc") > NotImplemented > >>> "abc".__eq__(f) > NotImplemented > > Note that there's nothing special about builtin types here. All types > are treated alike. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed May 8 15:47:18 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 8 May 2019 15:47:18 -0400 Subject: [Python-Dev] Redoing failed PR checks In-Reply-To: References: Message-ID: On 5/8/2019 10:23 AM, Eric V. Smith wrote: > I think you can close and reopen the PR. That?s what I?m trying on my blocked PR. That works but reruns all the CI checks, including the ones already passed. Some bots allow individual reruns, but it is not as clear as it should be. -- Terry Jan Reedy From vano at mail.mipt.ru Wed May 8 16:15:02 2019 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 8 May 2019 23:15:02 +0300 Subject: [Python-Dev] Redoing failed PR checks In-Reply-To: References: Message-ID: <5fee6656-1d33-e478-1018-8ac23c4cc433@mail.mipt.ru> On 08.05.2019 22:47, Terry Reedy wrote: > On 5/8/2019 10:23 AM, Eric V. Smith wrote: >> I think you can close and reopen the PR. That?s what I?m trying on my blocked PR. > > That works but reruns all the CI checks, including the ones already passed.? Some bots allow individual reruns, but it is not as clear as > it should be. > > I think rerunning PR checks is intentionally blocked to prevent submitters from silently smuggling unreliable code in. Whatever the case, you can make an empty commit with `git commit --allow-empty` (credit goes to https://coderwall.com/p/vkdekq/git-commit-allow-empty). This will rerun things, but all the failures will be visible in the PR. -- Regards, Ivan From ericsnowcurrently at gmail.com Wed May 8 18:19:05 2019 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 8 May 2019 18:19:05 -0400 Subject: [Python-Dev] My PyCon talk on the GIL, the C-API, and subinterpreters Message-ID: I just wanted to share links to my talk about the above, which I gave a week ago at PyCon. Enjoy! :) -eric video: https://youtu.be/7RlqbHCCVyc slides: https://bit.ly/2UMMJey project: https://github.com/ericsnowcurrently/multi-core-python -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Wed May 8 22:23:17 2019 From: benjamin at python.org (Benjamin Peterson) Date: Wed, 08 May 2019 22:23:17 -0400 Subject: [Python-Dev] Farewell, Python 3.4 In-Reply-To: References: Message-ID: <7bca5961-b8cd-42d8-8a00-1384e31de010@www.fastmail.com> Thank you for your service! On Wed, May 8, 2019, at 08:37, Larry Hastings wrote: > > > It's with a note of sadness that I announce the final retirement of > Python 3.4. The final release was back in March, but I didn't get > around to actually closing and deleting the 3.4 branch until this > morning. > > Python 3.4 introduced many features we all enjoy in modern Python--the > asyncio, ensurepip, and enum packages, just to name three. It's a > release I hope we all remember fondly. > > My eternal thanks to all the members of the release team that worked on > Python 3.4: > > > Georg Brandl > > > Julien Palard > > > Martin von L?wis > > > Ned Deily > > > Steve Dower > > > Terry Reedy > > > and all the engineers of the Python infrastructure team. > > Special thanks to Benjamin Peterson and Ned Deily, who frequently > scurried around behind the scenes cleaning up the messes I cluelessly > left in my wake. > > Having closed 3.4, I am now retired as Python 3.4 Release Manager. I > regret to inform all of you that you're still stuck with me as Python > 3.5 Release Manager until sometime next year. > > > > My very best wishes, > > > > */arry* > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org > From jake.tesler at gmail.com Thu May 9 12:15:41 2019 From: jake.tesler at gmail.com (Jake Tesler) Date: Thu, 9 May 2019 09:15:41 -0700 Subject: [Python-Dev] PR Review request Message-ID: Hello Python Dev team! I submitted a Bug+PR back in February and unfortunately, nobody has given it a review yet. :/ I'm was hoping that someone could take a look and suggest changes (if needed). I realize that there are a ton of PRs and things to review across the board, but here's to hoping someone has a little time to take a look at mine! Links: Bug: https://bugs.python.org/issue36084 PR: https://github.com/python/cpython/pull/11993 Thanks so much! Cheers, -Jake Tesler -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Thu May 9 14:30:23 2019 From: encukou at gmail.com (Petr Viktorin) Date: Thu, 9 May 2019 14:30:23 -0400 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: PEP 590 is on its way to be accepted, with some details still to be discussed. I've rejected PEP 580 so we can focus on one place. Here are things we discussed on GitHub but now seem to agree on: * The vectorcall function's kwname argument can be NULL. * Let's use `vectorcallfunc`, not `vectorcall`, and stop the bikeshedding. * `tp_vectorcall_offset` can be `Py_ssize_t` (The discussions around signedness and C standards and consistency are interesting, but ultimately irrelevant here.) * `PyCall_MakeTpCall` can be removed. * `PyVectorcall_Function` (for getting the `vectorcallfunc` of an object) can be an internal helper. External code should go through `PyCall_Vectorcall` (whatever we name it). * `PY_VECTORCALL_ARGUMENTS_OFFSET` is OK, bikeshedding over variants like `PY_VECTORCALL_PREPEND` won't bring much benefit. Anyone against, make your point now :) The following have discussion PRs open: * `PyCall_MakeVectorCall` name: https://github.com/python/peps/pull/1037 * Passing a dict to `PyObject_Vectorcall`: https://github.com/python/peps/pull/1038 * Type of the kwnames argument (PyObject/PyTupleObject): https://github.com/python/peps/pull/1039 The remaining points are: ### Making things private For Python 3.8, the public API should be private, so the API can get some contact with the real world. I'd especially like to be able to learn from Cython's experience using it. That would mean: * _PyObject_Vectorcall * _PyCall_MakeVectorCall * _PyVectorcall_NARGS * _METH_VECTORCALL * _Py_TPFLAGS_HAVE_VECTORCALL * _Py_TPFLAGS_METHOD_DESCRIPTOR ### Can the kwnames tuple be empty? Disallowing empty tuples means it's easier for the *callee* to detect the case of no keyword arguments. Instead of: if (kwnames != NULL && PyTuple_GET_SIZE(kwnames)) you have: if (kwnames != NULL) On the other hand, the *caller* would now be responsible for handling the no-kwarg case specially. Jeroen points out: > The side of the caller (which should ensure not to send an empty tuple) > is CPython and there the issue of people implementing the protocol wrongly > doesn't arise. > External C code is not expected to manually use tp_vectorcall_offset to make > vectorcalls: it is expected to use an API like PyCall_Vectorcall() and that > API will ensure to replace an empty tuple with NULL. > > I see it as an application of https://en.wikipedia.org/wiki/Robustness_principle > (Be conservative in what you send, be liberal in what you accept): > PyCall_Vectorcall should accept an empty tuple but it should not send an > empty tuple to the vectorcall function. But, if you apply the robustness principle to vectorcallfunc, it should accept empty tuples. ### `METH_VECTORCALL` function type Jeroen suggested changing this from: `PyObject *(*call) (PyObject *self, PyObject *const *args, Py_ssize_t nargs, PyObject *kwname)` to `vectorcallfunc`, i.e.: `PyObject *(*call) (PyObject *callable, Py_ssize_t n, PyObject *const *args, PyObject *kwnames)` Mark argues that this is a major change and prevents the interpreter from sanity checking the return value of PyMethodDef defined functions. (Since the functions are defined by extension code, they need to be sanity-checked, and this will be done by PyCFunction's vectorcall adapter. Tools like Cython can bypass the check if needed.) The underlying C function should not need to know how to extract "self" from the function object, or how to handle the argument offsetting. Those should be implementation details. I see the value in having METH_VECTORCALL equivalent to the existing METH_FASTCALL|METH_KEYWORDS. (Even though PEP 573 will need to add to the calling convention.) From brett at python.org Thu May 9 17:09:39 2019 From: brett at python.org (Brett Cannon) Date: Thu, 9 May 2019 14:09:39 -0700 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: On Thu, May 9, 2019 at 11:31 AM Petr Viktorin wrote: > PEP 590 is on its way to be accepted, with some details still to be > discussed. I've rejected PEP 580 so we can focus on one place. > > Here are things we discussed on GitHub but now seem to agree on: > > * The vectorcall function's kwname argument can be NULL. > * Let's use `vectorcallfunc`, not `vectorcall`, and stop the bikeshedding. > * `tp_vectorcall_offset` can be `Py_ssize_t` (The discussions around > signedness and C standards and consistency are interesting, but > ultimately irrelevant here.) > * `PyCall_MakeTpCall` can be removed. > * `PyVectorcall_Function` (for getting the `vectorcallfunc` of an > object) can be an internal helper. External code should go through > `PyCall_Vectorcall` (whatever we name it). > * `PY_VECTORCALL_ARGUMENTS_OFFSET` is OK, bikeshedding over variants > like `PY_VECTORCALL_PREPEND` won't bring much benefit. > > Anyone against, make your point now :) > Any reason the above are all "Vectorcall" and not "VectorCall"? You seem to potentially have that capitalization for "PyCall_MakeVectorCall" as mentioned below which seems to be asking for typos if there's going to be two ways to do it. :) -Brett > > The following have discussion PRs open: > > * `PyCall_MakeVectorCall` name: https://github.com/python/peps/pull/1037 > * Passing a dict to `PyObject_Vectorcall`: > https://github.com/python/peps/pull/1038 > * Type of the kwnames argument (PyObject/PyTupleObject): > https://github.com/python/peps/pull/1039 > > > The remaining points are: > > > ### Making things private > > For Python 3.8, the public API should be private, so the API can get > some contact with the real world. I'd especially like to be able to > learn from > Cython's experience using it. > That would mean: > > * _PyObject_Vectorcall > * _PyCall_MakeVectorCall > * _PyVectorcall_NARGS > * _METH_VECTORCALL > * _Py_TPFLAGS_HAVE_VECTORCALL > * _Py_TPFLAGS_METHOD_DESCRIPTOR > > > ### Can the kwnames tuple be empty? > > Disallowing empty tuples means it's easier for the *callee* to detect > the case of no keyword arguments. Instead of: > > if (kwnames != NULL && PyTuple_GET_SIZE(kwnames)) > > you have: > > if (kwnames != NULL) > > On the other hand, the *caller* would now be responsible for handling > the no-kwarg case specially. > > Jeroen points out: > > The side of the caller (which should ensure not to send an empty tuple) > > is CPython and there the issue of people implementing the protocol > wrongly > > doesn't arise. > > External C code is not expected to manually use tp_vectorcall_offset to > make > > vectorcalls: it is expected to use an API like PyCall_Vectorcall() and > that > > API will ensure to replace an empty tuple with NULL. > > > > I see it as an application of > https://en.wikipedia.org/wiki/Robustness_principle > > (Be conservative in what you send, be liberal in what you accept): > > PyCall_Vectorcall should accept an empty tuple but it should not send an > > empty tuple to the vectorcall function. > > But, if you apply the robustness principle to vectorcallfunc, it > should accept empty tuples. > > > ### `METH_VECTORCALL` function type > > Jeroen suggested changing this from: > > `PyObject *(*call) (PyObject *self, PyObject *const *args, > Py_ssize_t nargs, PyObject *kwname)` > > to `vectorcallfunc`, i.e.: > > `PyObject *(*call) (PyObject *callable, Py_ssize_t n, PyObject > *const *args, PyObject *kwnames)` > > Mark argues that this is a major change and prevents the interpreter > from sanity checking the return value of PyMethodDef defined > functions. > (Since the functions are defined by extension code, they need to be > sanity-checked, and this will be done by PyCFunction's vectorcall > adapter. Tools like Cython can bypass the check if needed.) > > The underlying C function should not need to know how to extract > "self" from the function object, or how to handle the argument > offsetting. > Those should be implementation details. > > I see the value in having METH_VECTORCALL equivalent to the existing > METH_FASTCALL|METH_KEYWORDS. > (Even though PEP 573 will need to add to the calling convention.) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Thu May 9 17:14:02 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 9 May 2019 23:14:02 +0200 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: <5CD4981A.4010402@UGent.be> On 2019-05-09 20:30, Petr Viktorin wrote: > ### Making things private > > For Python 3.8, the public API should be private, so the API can get > some contact with the real world. I'd especially like to be able to > learn from > Cython's experience using it. > That would mean: > > * _PyObject_Vectorcall > * _PyCall_MakeVectorCall > * _PyVectorcall_NARGS > * _METH_VECTORCALL > * _Py_TPFLAGS_HAVE_VECTORCALL > * _Py_TPFLAGS_METHOD_DESCRIPTOR Do we really have to underscore the names? Would there be a way to mark this API as provisional and subject to change without changing the names? If it turns out that PEP 590 was perfect after all, then we're just breaking stuff in Python 3.9 (when removing the underscores) for no reason. Alternatively, could we keep the underscored names as official API in Python 3.9? From J.Demeyer at UGent.be Thu May 9 17:17:06 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 9 May 2019 23:17:06 +0200 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: <5CD498D2.4030600@UGent.be> On 2019-05-09 23:09, Brett Cannon wrote: > Any reason the above are all "Vectorcall" and not "VectorCall"? You seem > to potentially have that capitalization for "PyCall_MakeVectorCall" as > mentioned below which seems to be asking for typos if there's going to > be two ways to do it. :) "PyCall_MakeVectorCall" is a typo for "PyVectorcall_Call" (https://github.com/python/peps/pull/1037) Everything else uses "Vectorcall" or "VECTORCALL". In text, we use "vectorcall" without a space. From J.Demeyer at UGent.be Thu May 9 17:33:47 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 9 May 2019 23:33:47 +0200 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: <5CD49CBB.2040508@UGent.be> On 2019-05-09 20:30, Petr Viktorin wrote: > The underlying C function should not need to know how to extract > "self" from the function object, or how to handle the argument > offsetting. > Those should be implementation details. Maybe you misunderstood my proposal. I want to allow both for extra flexibility: - METH_FASTCALL (possibly combined with METH_KEYWORDS) continues to work as before. If you don't want to care about the implementation details of vectorcall, this is the right thing to use. - METH_VECTORCALL (using exactly the vectorcallfunc signature) is a new calling convention for applications that want the lowest possible overhead at the cost of being slightly harder to use. Personally, I consider the discussion about who is supposed to check that a function returns NULL if and if an error occurred a tiny detail which shouldn't dictate the design. There are two solutions for this: either we move that check one level up and do it for all vectorcall functions. Or, we keep the existing checks in place but we don't do that check for METH_VECTORCALL (this is already more specialized anyway, so dropping that check doesn't hurt much). We could also decide to enable this check only for debug builds, especially if debug builds are going to be easier to use thank to Victor Stinner's work. > I see the value in having METH_VECTORCALL equivalent to the existing > METH_FASTCALL|METH_KEYWORDS. But why invent a new name for that? METH_FASTCALL|METH_KEYWORDS already works. The alias METH_VECTORCALL could only make things more confusing (having two ways to specify exactly the same thing). Or am I missing something? Jeroen. From J.Demeyer at UGent.be Thu May 9 17:54:14 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 9 May 2019 23:54:14 +0200 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> Message-ID: <5CD4A186.1010105@UGent.be> On 2019-05-09 20:30, Petr Viktorin wrote: > But, if you apply the robustness principle to vectorcallfunc, it > should accept empty tuples. Sure, if the callee wants to accept empty tuples anyway, it can do that. That's the robustness principle. But us *forcing* the callee to accept empty tuples is certainly not. Basically my point is: with a little bit of effort in CPython we can make things simpler for all users of vectorcall. Why not do that? Seriously, what's the argument for *not* applying this change? Jeroen. From encukou at gmail.com Thu May 9 18:07:06 2019 From: encukou at gmail.com (Petr Viktorin) Date: Thu, 9 May 2019 18:07:06 -0400 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: <5CD49CBB.2040508@UGent.be> References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> <5CD49CBB.2040508@UGent.be> Message-ID: On 5/9/19 5:33 PM, Jeroen Demeyer wrote: > On 2019-05-09 20:30, Petr Viktorin wrote: >> The underlying C function should not need to know how to extract >> "self" from the function object, or how to handle the argument >> offsetting. >> Those should be implementation details. > > Maybe you misunderstood my proposal. I want to allow both for extra > flexibility: > > - METH_FASTCALL (possibly combined with METH_KEYWORDS) continues to work > as before. If you don't want to care about the implementation details of > vectorcall, this is the right thing to use. > > - METH_VECTORCALL (using exactly the vectorcallfunc signature) is a new > calling convention for applications that want the lowest possible > overhead at the cost of being slightly harder to use. Then we can, in the spirit of minimalism, not add METH_VECTORCALL at all. > Personally, I consider the discussion about who is supposed to check > that a function returns NULL if and if an error occurred a tiny detail > which shouldn't dictate the design. There are two solutions for this: > either we move that check one level up and do it for all vectorcall > functions. Or, we keep the existing checks in place but we don't do that > check for METH_VECTORCALL (this is already more specialized anyway, so > dropping that check doesn't hurt much). We could also decide to enable > this check only for debug builds, especially if debug builds are going > to be easier to use thank to Victor Stinner's work. > >> I see the value in having METH_VECTORCALL equivalent to the existing >> METH_FASTCALL|METH_KEYWORDS. > > But why invent a new name for that? METH_FASTCALL|METH_KEYWORDS already > works. The alias METH_VECTORCALL could only make things more confusing > (having two ways to specify exactly the same thing). Or am I missing > something? METH_FASTCALL is currently not documented, and it should be renamed before it's documented. Names with "fast" or "new" generally don't age well. From vstinner at redhat.com Thu May 9 20:14:56 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 10 May 2019 02:14:56 +0200 Subject: [Python-Dev] Adding a tzidx cache to datetime In-Reply-To: References: Message-ID: Hi Paul, The change is basically an optimization. I'm uncomfortable to design it "only" for dateutil. What if tomorrow someone has to store an arbitrary Python object, rather than just an integer (in range [0; 254]), into a datetime for a different optimization? Moreover, I dislike adding a *public* method for an *internal* cache. Right now, it is not possible to create a weak reference to a datetime. If we make it possible, it would be possible to have an external cache implemented with weakref.WeakSet to clear old entries when a datetime object is detroyed. What do you think of adding a private "_cache" attribute which would be an arbitrary Python object? (None by default) Victor Le mar. 7 mai 2019 ? 21:46, Paul Ganssle a ?crit : > > Greetings all, > > I have one last feature request that I'd like added to datetime for Python 3.8, and this one I think could use some more discussion, the addition of a "time zone index cache" to the datetime object. The rationale is laid out in detail in bpo-35723. The general problem is that currently, every invocation of utcoffset, tzname and dst needs to do full, independent calculations of the time zone offsets, even for time zones where the mapping is guaranteed to be stable because datetimes are immutable. I have a proof of concept implementation: PR #11529. > > I'm envisioning that the `datetime` class will add a private `_tzidx` single-byte member (it seems that this does not increase the size of the datetime object, because it's just using an unused alignment byte). `datetime` will also add a `tzidx()` method, which will return `_tzidx` if it's been set and otherwise it will call `self.tzinfo.tzidx()`. If `self.tzinfo.tzidx()` returns a number between 0 and 254 (inclusive), it sets `_tzidx` to this value. tzidx() then returns whatever self.tzinfo.tzidx() returned. > > The value of this is that as far as I can tell, nearly all non-trivial tzinfo implementations construct a list of possible offsets, and implement utcoffset(), tzname() and dst() by calculating an index into that list and returning it. There are almost always less than 255 distinct offsets. By adding this cache on the datetime, we're using a small amount of currently-unused memory to prevent unnecessary calculations about a given datetime. The feature is entirely opt-in, and has no downsides if it goes unused, and it makes it possible to write tzinfo implementations that are both lazy and as fast as the "eager calculation" mode that pytz uses (and that causes many problems for pytz's users). > > I have explored the idea of using an lru cache of some sort on the tzinfo object itself, but there are two problems with this: > > 1. Calculating the hash of a datetime calls .utcoffset(), which means that it is necessary to, at minimum, do a `replace` on the datetime (and constructing a new datetime is a pretty considerable speed hit) > > 2. It will be a much bigger memory cost, since my current proposal uses approximately zero additional memory (not sure if the alignment stuff is platform-dependent or something, but it doesn't use additional memory on my linux computer). > > I realize this proposal is somewhat difficult to wrap your head around, so if anyone would like to chat with me about it in person, I'll be at PyCon sprints until Thursday morning. > > Best, > Paul > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com -- Night gathers, and now my watch begins. It shall not end until my death. From paul at ganssle.io Thu May 9 20:58:58 2019 From: paul at ganssle.io (Paul Ganssle) Date: Thu, 9 May 2019 20:58:58 -0400 Subject: [Python-Dev] Adding a tzidx cache to datetime In-Reply-To: References: Message-ID: This is only "only" for dateutil in the sense that no one other than dateutil implements tzinfo with the interface provided. If dateutil were /not/ already implemented with a list of offsets and their indexes, I would still propose this, and just re-write dateutil to take advantage of it. From a cursory glance at pendulum, it seems that they could take advantage of it as well (though they use their own datetime subclass, so they have always had the ability to add this). > What do you think of adding a private "_cache" attribute which would > be an arbitrary Python object? (None by default) We cannot use a private attribute (other than to do the actual storage, since the thing that gets stored is not directly accessible anyway and is instead mediated by a layer that manages the cache) because this is a feature explicitly being added for use by tzinfo, /not/ by datetime. If it's private then it's not safe for implementations of tzinfo to actually use it, which defeats the purpose. Regarding the use of an arbitrary Python object: What I'm proposing is that we offer a bit of the "free" storage space in the alignment bits to tzinfo objects to use as a cache. In /most/ cases this will be very useful to someone implementing a tzinfo, because there are really only so many ways to accomplish this task, and most time zones are expressible as a very short list of offset/name/dst combinations, plus some rule for which applies when, which is why a small integer cache is sufficient and more or less universal (i.e. not specific to dateutil's implementation). I will also note that in my design, it is still possible for `tzinfo` to return something other than [0, 254], it's just that that information will not be cached, so it won't get the benefit of any optimization, but the same interface / implementation can be used. In my test with gcc, adding an additional PyObject* to the end of the PyDateTime_DateTime struct increased the size of the `datetime.datetime` object from 64 to 72 bytes, whereas adding an `unsigned char` after the `fold` leaves it unchanged. Given that the expansion to arbitrary Python objects is speculative and doesn't have any particular use case, I would prefer to leave the feature as is, and reconsider the possibility of storing arbitrary Python objects on the datetime if there's some compelling reason to do so (it would be a backwards-compatible change at that point anyway). On 5/9/19 8:14 PM, Victor Stinner wrote: > Hi Paul, > > The change is basically an optimization. I'm uncomfortable to design > it "only" for dateutil. What if tomorrow someone has to store an > arbitrary Python object, rather than just an integer (in range [0; > 254]), into a datetime for a different optimization? > > Moreover, I dislike adding a *public* method for an *internal* cache. > > Right now, it is not possible to create a weak reference to a > datetime. If we make it possible, it would be possible to have an > external cache implemented with weakref.WeakSet to clear old entries > when a datetime object is detroyed. > > What do you think of adding a private "_cache" attribute which would > be an arbitrary Python object? (None by default) > > Victor > > Le mar. 7 mai 2019 ? 21:46, Paul Ganssle a ?crit : >> Greetings all, >> >> I have one last feature request that I'd like added to datetime for Python 3.8, and this one I think could use some more discussion, the addition of a "time zone index cache" to the datetime object. The rationale is laid out in detail in bpo-35723. The general problem is that currently, every invocation of utcoffset, tzname and dst needs to do full, independent calculations of the time zone offsets, even for time zones where the mapping is guaranteed to be stable because datetimes are immutable. I have a proof of concept implementation: PR #11529. >> >> I'm envisioning that the `datetime` class will add a private `_tzidx` single-byte member (it seems that this does not increase the size of the datetime object, because it's just using an unused alignment byte). `datetime` will also add a `tzidx()` method, which will return `_tzidx` if it's been set and otherwise it will call `self.tzinfo.tzidx()`. If `self.tzinfo.tzidx()` returns a number between 0 and 254 (inclusive), it sets `_tzidx` to this value. tzidx() then returns whatever self.tzinfo.tzidx() returned. >> >> The value of this is that as far as I can tell, nearly all non-trivial tzinfo implementations construct a list of possible offsets, and implement utcoffset(), tzname() and dst() by calculating an index into that list and returning it. There are almost always less than 255 distinct offsets. By adding this cache on the datetime, we're using a small amount of currently-unused memory to prevent unnecessary calculations about a given datetime. The feature is entirely opt-in, and has no downsides if it goes unused, and it makes it possible to write tzinfo implementations that are both lazy and as fast as the "eager calculation" mode that pytz uses (and that causes many problems for pytz's users). >> >> I have explored the idea of using an lru cache of some sort on the tzinfo object itself, but there are two problems with this: >> >> 1. Calculating the hash of a datetime calls .utcoffset(), which means that it is necessary to, at minimum, do a `replace` on the datetime (and constructing a new datetime is a pretty considerable speed hit) >> >> 2. It will be a much bigger memory cost, since my current proposal uses approximately zero additional memory (not sure if the alignment stuff is platform-dependent or something, but it doesn't use additional memory on my linux computer). >> >> I realize this proposal is somewhat difficult to wrap your head around, so if anyone would like to chat with me about it in person, I'll be at PyCon sprints until Thursday morning. >> >> Best, >> Paul >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From vstinner at redhat.com Thu May 9 21:01:41 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 10 May 2019 03:01:41 +0200 Subject: [Python-Dev] Redoing failed PR checks In-Reply-To: References: Message-ID: Hi, Le mer. 8 mai 2019 ? 16:10, Mark Shannon a ?crit : > How do I redo a failed PR check? > The appveyor failure for https://github.com/python/cpython/pull/13181 > appears to be spurious, but there is no obvious way to redo it. First, please make sure that a test failure is known in the bug tracker. Nobody here checked and no: it wasn't reported, whereas I also saw this bug once last week! (but I was too busy to report it, sorry!) I reported the bug: "test_asyncio: test_drain_raises() fails randomly on Windows" https://bugs.python.org/issue36870 > BTW, this is not the first time I've seen a PR blocked by a spurious > appveyor failure. Please check if every single test failure is known in the bug tracker, and if it's not the case: report it! More info about Python CI: https://pythondev.readthedocs.io/ci.html I'm trying to fix all tests which fail randomly for 5 years now (hopefully, with the help of other core devs like Pablo!). It's getting better, but sometimes new flaky tests pop up. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From vstinner at redhat.com Thu May 9 21:04:46 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 10 May 2019 03:04:46 +0200 Subject: [Python-Dev] Please merge : bpo-34848 In-Reply-To: <5CCC6107.10900@UGent.be> References: <5CCC6107.10900@UGent.be> Message-ID: Le ven. 3 mai 2019 ? 17:47, Jeroen Demeyer a ?crit : > On 2019-05-03 14:24, Victor Stinner wrote: > > Hi Srinivas, > > > > I merged your obvious doc fix, thanks. > > Can you please do the same for > > https://github.com/python/cpython/pull/12784 This PR is modying critical change. I don't have the bandwidth right now to review it. Victor From vstinner at redhat.com Thu May 9 21:05:30 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 10 May 2019 03:05:30 +0200 Subject: [Python-Dev] Please merge : bpo-34848 In-Reply-To: <5CCC6107.10900@UGent.be> References: <5CCC6107.10900@UGent.be> Message-ID: Le ven. 3 mai 2019 ? 17:47, Jeroen Demeyer a ?crit : > > On 2019-05-03 14:24, Victor Stinner wrote: > > Hi Srinivas, > > > > I merged your obvious doc fix, thanks. > > Can you please do the same for > https://github.com/python/cpython/pull/12784 This PR is modying critical code (signal). I don't have the bandwidth right now to review it. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From mal at egenix.com Fri May 10 03:18:43 2019 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 10 May 2019 09:18:43 +0200 Subject: [Python-Dev] Adding a tzidx cache to datetime In-Reply-To: References: Message-ID: <619825e1-be2e-b8e7-6eef-07921baebc7f@egenix.com> On 10.05.2019 02:58, Paul Ganssle wrote: > This is only "only" for dateutil in the sense that no one other than > dateutil implements tzinfo with the interface provided. If dateutil were > /not/ already implemented with a list of offsets and their indexes, I > would still propose this, and just re-write dateutil to take advantage > of it. From a cursory glance at pendulum, it seems that they could take > advantage of it as well (though they use their own datetime subclass, so > they have always had the ability to add this). > >> What do you think of adding a private "_cache" attribute which would >> be an arbitrary Python object? (None by default) > > We cannot use a private attribute (other than to do the actual storage, > since the thing that gets stored is not directly accessible anyway and > is instead mediated by a layer that manages the cache) because this is a > feature explicitly being added for use by tzinfo, /not/ by datetime. If > it's private then it's not safe for implementations of tzinfo to > actually use it, which defeats the purpose. > > Regarding the use of an arbitrary Python object: What I'm proposing is > that we offer a bit of the "free" storage space in the alignment bits to > tzinfo objects to use as a cache. In /most/ cases this will be very > useful to someone implementing a tzinfo, because there are really only > so many ways to accomplish this task, and most time zones are > expressible as a very short list of offset/name/dst combinations, plus > some rule for which applies when, which is why a small integer cache is > sufficient and more or less universal (i.e. not specific to dateutil's > implementation). > > I will also note that in my design, it is still possible for `tzinfo` to > return something other than [0, 254], it's just that that information > will not be cached, so it won't get the benefit of any optimization, but > the same interface / implementation can be used. > > In my test with gcc, adding an additional PyObject* to the end of the > PyDateTime_DateTime struct increased the size of the `datetime.datetime` > object from 64 to 72 bytes, whereas adding an `unsigned char` after the > `fold` leaves it unchanged. Given that the expansion to arbitrary Python > objects is speculative and doesn't have any particular use case, I would > prefer to leave the feature as is, and reconsider the possibility of > storing arbitrary Python objects on the datetime if there's some > compelling reason to do so (it would be a backwards-compatible change at > that point anyway). Given that many datetime objects in practice don't use timezones (e.g. in large data stores you typically use UTC and naive datetime objects), I think that making the object itself larger to accommodate for a cache, which will only be used a smaller percentage of the use cases, isn't warranted. Going from 64 bytes to 72 bytes also sounds like this could have negative effects on cache lines. If you need a per object cache, you can either use weakref objects or maintain a separate dictionary in dateutil or other timezone helpers which indexes objects by id(obj). That said, if you only add a byte field which doesn't make the object larger in practice (you merely use space that alignments would use anyway), this shouldn't be a problem. The use of that field should be documented, though, so that other implementations can use/provide it as well. Thanks, -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, May 10 2019) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ > On 5/9/19 8:14 PM, Victor Stinner wrote: >> Hi Paul, >> >> The change is basically an optimization. I'm uncomfortable to design >> it "only" for dateutil. What if tomorrow someone has to store an >> arbitrary Python object, rather than just an integer (in range [0; >> 254]), into a datetime for a different optimization? >> >> Moreover, I dislike adding a *public* method for an *internal* cache. >> >> Right now, it is not possible to create a weak reference to a >> datetime. If we make it possible, it would be possible to have an >> external cache implemented with weakref.WeakSet to clear old entries >> when a datetime object is detroyed. >> >> What do you think of adding a private "_cache" attribute which would >> be an arbitrary Python object? (None by default) >> >> Victor >> >> Le mar. 7 mai 2019 ? 21:46, Paul Ganssle a ?crit : >>> Greetings all, >>> >>> I have one last feature request that I'd like added to datetime for Python 3.8, and this one I think could use some more discussion, the addition of a "time zone index cache" to the datetime object. The rationale is laid out in detail in bpo-35723. The general problem is that currently, every invocation of utcoffset, tzname and dst needs to do full, independent calculations of the time zone offsets, even for time zones where the mapping is guaranteed to be stable because datetimes are immutable. I have a proof of concept implementation: PR #11529. >>> >>> I'm envisioning that the `datetime` class will add a private `_tzidx` single-byte member (it seems that this does not increase the size of the datetime object, because it's just using an unused alignment byte). `datetime` will also add a `tzidx()` method, which will return `_tzidx` if it's been set and otherwise it will call `self.tzinfo.tzidx()`. If `self.tzinfo.tzidx()` returns a number between 0 and 254 (inclusive), it sets `_tzidx` to this value. tzidx() then returns whatever self.tzinfo.tzidx() returned. >>> >>> The value of this is that as far as I can tell, nearly all non-trivial tzinfo implementations construct a list of possible offsets, and implement utcoffset(), tzname() and dst() by calculating an index into that list and returning it. There are almost always less than 255 distinct offsets. By adding this cache on the datetime, we're using a small amount of currently-unused memory to prevent unnecessary calculations about a given datetime. The feature is entirely opt-in, and has no downsides if it goes unused, and it makes it possible to write tzinfo implementations that are both lazy and as fast as the "eager calculation" mode that pytz uses (and that causes many problems for pytz's users). >>> >>> I have explored the idea of using an lru cache of some sort on the tzinfo object itself, but there are two problems with this: >>> >>> 1. Calculating the hash of a datetime calls .utcoffset(), which means that it is necessary to, at minimum, do a `replace` on the datetime (and constructing a new datetime is a pretty considerable speed hit) >>> >>> 2. It will be a much bigger memory cost, since my current proposal uses approximately zero additional memory (not sure if the alignment stuff is platform-dependent or something, but it doesn't use additional memory on my linux computer). >>> >>> I realize this proposal is somewhat difficult to wrap your head around, so if anyone would like to chat with me about it in person, I'll be at PyCon sprints until Thursday morning. >>> >>> Best, >>> Paul >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > From stefan_ml at behnel.de Fri May 10 03:57:06 2019 From: stefan_ml at behnel.de (Stefan Behnel) Date: Fri, 10 May 2019 09:57:06 +0200 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> <5CD49CBB.2040508@UGent.be> Message-ID: Petr Viktorin schrieb am 10.05.19 um 00:07: > On 5/9/19 5:33 PM, Jeroen Demeyer wrote: >> Maybe you misunderstood my proposal. I want to allow both for extra >> flexibility: >> >> - METH_FASTCALL (possibly combined with METH_KEYWORDS) continues to work >> as before. If you don't want to care about the implementation details of >> vectorcall, this is the right thing to use. >> >> - METH_VECTORCALL (using exactly the vectorcallfunc signature) is a new >> calling convention for applications that want the lowest possible >> overhead at the cost of being slightly harder to use. > > Then we can, in the spirit of minimalism, not add METH_VECTORCALL at all. > [...] > METH_FASTCALL is currently not documented, and it should be renamed before > it's documented. Names with "fast" or "new" generally don't age well. I personally don't see an advantage in having both, apart from helping code that wants to be fast also on Py3.7, for example. It unnecessarily complicates the CPython implementation and C-API. I'd be ok with removing FASTCALL in favour of VECTORCALL. That's more code to generate for Cython in order to adapt to Py<3.6, Py3.6, Py3.7 and then Py>=3.[89], but well, seeing the heap of code that we *already* generate, it's not going to hurt our users much. It would, however, be (selfishly) helpful if FASTCALL could still go through a deprecation period, because we'd like to keep the current Cython 0.29.x release series compatible with Python 3.8, and I'd like to avoid adding support for VECTORCALL and compiling out FASTCALL in a point release. Removing it in Py3.9 seems ok to me. Stefan From J.Demeyer at UGent.be Fri May 10 04:20:20 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Fri, 10 May 2019 10:20:20 +0200 Subject: [Python-Dev] PEP 580/590 discussion In-Reply-To: References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> <5CC1C765.1020608@UGent.be> <1309c111-359e-1af6-2655-a16ffaad2994@redhat.com> <5CD49CBB.2040508@UGent.be> Message-ID: <5CD53444.1050301@UGent.be> On 2019-05-10 00:07, Petr Viktorin wrote: > METH_FASTCALL is currently not documented, and it should be renamed > before it's documented. Names with "fast" or "new" generally don't age > well. Just to make sure that we're understanding correctly, is your proposal to do the following: - remove the name METH_FASTCALL - remove the calling convention METH_FASTCALL without METH_KEYWORDS - rename METH_FASTCALL|METH_KEYWORDS -> METH_VECTORCALL From nad at python.org Fri May 10 13:52:08 2019 From: nad at python.org (Ned Deily) Date: Fri, 10 May 2019 13:52:08 -0400 Subject: [Python-Dev] Ignore spurious 3.4 checkin messages! Message-ID: Eek! I was just doing a bit of branch cleanup in the cpython repo and managed to trigger a bunch (30-ish) of duplicate checkin messages to bugs.python.org for old commits. I will remove them from b.p.o. Please ignore any 3.4 spam email. Sorry! -- Ned Deily nad at python.org -- [] From status at bugs.python.org Fri May 10 14:08:20 2019 From: status at bugs.python.org (Python tracker) Date: Fri, 10 May 2019 18:08:20 +0000 (UTC) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20190510180820.4C70E52B200@bugs.ams1.psf.io> ACTIVITY SUMMARY (2019-05-03 - 2019-05-10) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7084 ( +3) closed 41542 (+88) total 48626 (+91) Open issues with patches: 2851 Issues opened (65) ================== #15987: Provide a way to compare AST nodes for equality recursively https://bugs.python.org/issue15987 reopened by vstinner #36785: Implement PEP 574 https://bugs.python.org/issue36785 opened by pitrou #36786: "make install" should run compileall in parallel https://bugs.python.org/issue36786 opened by pitrou #36788: Add clamp() function to builtins https://bugs.python.org/issue36788 opened by TheComet #36789: Unicode HOWTO incorrectly states that UTF-8 contains no zero b https://bugs.python.org/issue36789 opened by mbiggs #36790: test_asyncio fails with application verifier! https://bugs.python.org/issue36790 opened by Alexander Riccio #36791: sum() relies on C signed overflow behaviour https://bugs.python.org/issue36791 opened by serhiy.storchaka #36792: [Windows] time: crash on formatting time with de_DE locale https://bugs.python.org/issue36792 opened by CharlieClark #36793: Do not define unneeded __str__ equal to __repr__ https://bugs.python.org/issue36793 opened by serhiy.storchaka #36794: asyncio.Lock documentation in Py3.8 lacks parts presented in d https://bugs.python.org/issue36794 opened by germn #36795: "make venv" failed in Docs https://bugs.python.org/issue36795 opened by pitrou #36796: Error handling cleanup in _testcapimodule.c https://bugs.python.org/issue36796 opened by ZackerySpytz #36797: Cull more oudated distutils information https://bugs.python.org/issue36797 opened by ncoghlan #36799: Typo in ctypes documentation https://bugs.python.org/issue36799 opened by rrt #36803: Getting a lot of runtime misaligned address error while buildi https://bugs.python.org/issue36803 opened by dibya ranjan mishra #36804: Python for Windows installer Repair option does not repair PIP https://bugs.python.org/issue36804 opened by Mark Green #36805: Don't close subprocess stream if it's stdin is closed https://bugs.python.org/issue36805 opened by asvetlov #36807: IDLE doesn't call os.fsync() https://bugs.python.org/issue36807 opened by gvanrossum #36809: Crash for test test_importlib https://bugs.python.org/issue36809 opened by furzoom #36810: Recursive type annotations do not work in documentation tests https://bugs.python.org/issue36810 opened by lovasoa #36813: QueueListener not calling task_done upon termination https://bugs.python.org/issue36813 opened by bar.harel #36816: self-signed.pythontest.net TLS certificate key is too weak https://bugs.python.org/issue36816 opened by gregory.p.smith #36818: Add PyInterpreterState.runtime. https://bugs.python.org/issue36818 opened by eric.snow #36819: Crash during encoding using UTF-16/32 and custom error handler https://bugs.python.org/issue36819 opened by atalaba #36820: Captured exceptions are keeping user objects alive unnecessari https://bugs.python.org/issue36820 opened by mariocj89 #36821: Termios module largely untested https://bugs.python.org/issue36821 opened by anthonypjshaw #36822: Minor grammatical fix in glossary.rst https://bugs.python.org/issue36822 opened by CuriousLearner #36823: shutil.copytree copies directories and files but fails with th https://bugs.python.org/issue36823 opened by rbavery #36824: Refactor str tests to reflect that str and unicode are merged https://bugs.python.org/issue36824 opened by dfortunov #36825: Make TestCase aware of the command line arguments given to Tes https://bugs.python.org/issue36825 opened by remi.lapeyre #36826: ast_unparser.c doesn't handle := expressions https://bugs.python.org/issue36826 opened by eric.smith #36827: Overriding __new__ method with itself changes behaviour of the https://bugs.python.org/issue36827 opened by alexey-muranov #36829: CLI option to make PyErr_WriteUnraisable abort the current pro https://bugs.python.org/issue36829 opened by graingert #36833: Add tests for Datetime C API Macros https://bugs.python.org/issue36833 opened by p-ganssle #36834: mock.patch.object does not persist __module__ name for functio https://bugs.python.org/issue36834 opened by spark #36837: Make il8n tools available from `python -m` https://bugs.python.org/issue36837 opened by bbkane #36839: Support the buffer protocol in code objects https://bugs.python.org/issue36839 opened by dino.viehland #36840: Add stream.abort() async method https://bugs.python.org/issue36840 opened by asvetlov #36841: Supporting customization of float encoding in JSON https://bugs.python.org/issue36841 opened by mitar #36842: Implement PEP 578 https://bugs.python.org/issue36842 opened by steve.dower #36843: AIX build fails with failure to get random numbers https://bugs.python.org/issue36843 opened by Robert Boehne #36845: ipaddres.IPv4Network and ipaddress.IPv6Network tuple construct https://bugs.python.org/issue36845 opened by niconorsk #36848: autospec fails with AttributeError when mocked class has __sig https://bugs.python.org/issue36848 opened by xtreak #36849: Native libcurses on HP-UX not properly detected https://bugs.python.org/issue36849 opened by michael-o #36850: shutil.copy2 fails with even with source network filesystem no https://bugs.python.org/issue36850 opened by hans-meine #36852: Python3.7.2 fails to cross-compile (yocto / openembedded) when https://bugs.python.org/issue36852 opened by mschoepf #36853: inconsistencies in docs builds (Sphinx 2) https://bugs.python.org/issue36853 opened by jaraco #36854: GC operates out of global runtime state. https://bugs.python.org/issue36854 opened by eric.snow #36855: update tests relying on a copy of a TLS certificate (pem) to s https://bugs.python.org/issue36855 opened by gregory.p.smith #36856: faulthandler._stack_overflow doesn't work on x86-linux with KP https://bugs.python.org/issue36856 opened by xry111 #36858: f-string '=' debugging output needs to be documented https://bugs.python.org/issue36858 opened by eric.smith #36859: sqlite3 dml statement detection does not account for CTEs https://bugs.python.org/issue36859 opened by coleifer #36860: Inconsistent/Undocumented behavior with pathlib resolve and ab https://bugs.python.org/issue36860 opened by alexjacobson95 #36862: Add Visual Studio 2019 support to PCBuild/env.bat https://bugs.python.org/issue36862 opened by anthonypjshaw #36863: argparse doesn't like options in the middle of arguments https://bugs.python.org/issue36863 opened by rhubarbdog x #36865: FileInput does not allow 'rt' mode, but all its existing deleg https://bugs.python.org/issue36865 opened by natgaertner #36866: Certificate verification errors in urllib.request become URLEr https://bugs.python.org/issue36866 opened by sivel #36867: Make semaphore_tracker track other system resources https://bugs.python.org/issue36867 opened by pierreglaser #36868: New behavior of OpenSSL hostname verification not exposed, inc https://bugs.python.org/issue36868 opened by josh.r #36869: Avoid warning of unused variables https://bugs.python.org/issue36869 opened by eamanu #36870: test_asyncio: test_drain_raises() fails randomly on Windows https://bugs.python.org/issue36870 opened by vstinner #36871: Misleading error from unittest.mock's assert_has_calls https://bugs.python.org/issue36871 opened by gregory.p.smith #36873: http.server: Document explicitly that symbolic links are follo https://bugs.python.org/issue36873 opened by vstinner #36874: Support CDATA by xml.etree.(c)ElementTree https://bugs.python.org/issue36874 opened by Pierre van de Laar #36875: argparse does not ship with translations https://bugs.python.org/issue36875 opened by carmenbianca Most recent 15 issues with no replies (15) ========================================== #36873: http.server: Document explicitly that symbolic links are follo https://bugs.python.org/issue36873 #36870: test_asyncio: test_drain_raises() fails randomly on Windows https://bugs.python.org/issue36870 #36868: New behavior of OpenSSL hostname verification not exposed, inc https://bugs.python.org/issue36868 #36867: Make semaphore_tracker track other system resources https://bugs.python.org/issue36867 #36866: Certificate verification errors in urllib.request become URLEr https://bugs.python.org/issue36866 #36865: FileInput does not allow 'rt' mode, but all its existing deleg https://bugs.python.org/issue36865 #36862: Add Visual Studio 2019 support to PCBuild/env.bat https://bugs.python.org/issue36862 #36855: update tests relying on a copy of a TLS certificate (pem) to s https://bugs.python.org/issue36855 #36854: GC operates out of global runtime state. https://bugs.python.org/issue36854 #36850: shutil.copy2 fails with even with source network filesystem no https://bugs.python.org/issue36850 #36849: Native libcurses on HP-UX not properly detected https://bugs.python.org/issue36849 #36842: Implement PEP 578 https://bugs.python.org/issue36842 #36839: Support the buffer protocol in code objects https://bugs.python.org/issue36839 #36833: Add tests for Datetime C API Macros https://bugs.python.org/issue36833 #36826: ast_unparser.c doesn't handle := expressions https://bugs.python.org/issue36826 Most recent 15 issues waiting for review (15) ============================================= #36867: Make semaphore_tracker track other system resources https://bugs.python.org/issue36867 #36865: FileInput does not allow 'rt' mode, but all its existing deleg https://bugs.python.org/issue36865 #36862: Add Visual Studio 2019 support to PCBuild/env.bat https://bugs.python.org/issue36862 #36859: sqlite3 dml statement detection does not account for CTEs https://bugs.python.org/issue36859 #36856: faulthandler._stack_overflow doesn't work on x86-linux with KP https://bugs.python.org/issue36856 #36854: GC operates out of global runtime state. https://bugs.python.org/issue36854 #36852: Python3.7.2 fails to cross-compile (yocto / openembedded) when https://bugs.python.org/issue36852 #36850: shutil.copy2 fails with even with source network filesystem no https://bugs.python.org/issue36850 #36849: Native libcurses on HP-UX not properly detected https://bugs.python.org/issue36849 #36842: Implement PEP 578 https://bugs.python.org/issue36842 #36841: Supporting customization of float encoding in JSON https://bugs.python.org/issue36841 #36839: Support the buffer protocol in code objects https://bugs.python.org/issue36839 #36829: CLI option to make PyErr_WriteUnraisable abort the current pro https://bugs.python.org/issue36829 #36825: Make TestCase aware of the command line arguments given to Tes https://bugs.python.org/issue36825 #36824: Refactor str tests to reflect that str and unicode are merged https://bugs.python.org/issue36824 Top 10 most discussed issues (10) ================================= #36792: [Windows] time: crash on formatting time with de_DE locale https://bugs.python.org/issue36792 31 msgs #27639: UserList.__getitem__ doesn't account for slices https://bugs.python.org/issue27639 12 msgs #36829: CLI option to make PyErr_WriteUnraisable abort the current pro https://bugs.python.org/issue36829 11 msgs #36843: AIX build fails with failure to get random numbers https://bugs.python.org/issue36843 11 msgs #36807: IDLE doesn't call os.fsync() https://bugs.python.org/issue36807 9 msgs #36816: self-signed.pythontest.net TLS certificate key is too weak https://bugs.python.org/issue36816 8 msgs #35925: test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster https://bugs.python.org/issue35925 7 msgs #36395: Add deferred single-threaded/fake executor to concurrent.futur https://bugs.python.org/issue36395 7 msgs #36848: autospec fails with AttributeError when mocked class has __sig https://bugs.python.org/issue36848 7 msgs #25541: Wrong usage of sockaddr_un struct for abstract namespace unix https://bugs.python.org/issue25541 6 msgs Issues closed (85) ================== #1054: scriptsinstall target fails in alternate build dir https://bugs.python.org/issue1054 closed by SilentGhost #6911: Document changes in asynchat https://bugs.python.org/issue6911 closed by CuriousLearner #11001: Various obvious errors in cookies documentation https://bugs.python.org/issue11001 closed by gregory.p.smith #14440: Close background process if IDLE closes abnormally. https://bugs.python.org/issue14440 closed by asvetlov #16024: Doc cleanup regarding path=fd, dir_fd, follow_symlinks, etc https://bugs.python.org/issue16024 closed by larry #20185: Derby #17: Convert 49 sites to Argument Clinic across 13 files https://bugs.python.org/issue20185 closed by mdk #20709: os.utime(path_to_directory): wrong documentation for Windows. https://bugs.python.org/issue20709 closed by mdk #24339: iso6937 encoding missing https://bugs.python.org/issue24339 closed by mdk #24712: Docs page's sidebar vibrates on mouse wheel scroll on Chrome. https://bugs.python.org/issue24712 closed by matrixise #25251: Unknown MS Compiler version 1900 https://bugs.python.org/issue25251 closed by anthonypjshaw #25436: argparse.ArgumentError missing error message in __repr__ https://bugs.python.org/issue25436 closed by dfortunov #26374: concurrent_futures Executor.map semantics better specified in https://bugs.python.org/issue26374 closed by bquinlan #26903: ProcessPoolExecutor(max_workers=64) crashes on Windows https://bugs.python.org/issue26903 closed by bquinlan #26978: Implement pathlib.Path.link (Using os.link) https://bugs.python.org/issue26978 closed by pitrou #27497: csv module: Add return value to DictWriter.writeheader https://bugs.python.org/issue27497 closed by vstinner #28238: In xml.etree.ElementTree findall() can't search all elements i https://bugs.python.org/issue28238 closed by scoder #28729: Exception from sqlite3 adapter masked by sqlite3.InterfaceErro https://bugs.python.org/issue28729 closed by mdk #28795: Misleading stating, that SIGINT handler is installed by defaul https://bugs.python.org/issue28795 closed by mdk #29075: Remove Windows Vista support https://bugs.python.org/issue29075 closed by vstinner #29164: make test always fail at 218/405 ( AssertionError: ',' not fou https://bugs.python.org/issue29164 closed by cheryl.sabella #30262: Don't expose sqlite3 Cache and Statement https://bugs.python.org/issue30262 closed by berker.peksag #30271: Make sqlite3 statement cache optional https://bugs.python.org/issue30271 closed by palaviv #30668: DOC: missing word in license.rst https://bugs.python.org/issue30668 closed by matrixise #30793: Parsing error on f-string-expressions containing strings with https://bugs.python.org/issue30793 closed by eric.smith #31200: address sanitizer build fails https://bugs.python.org/issue31200 closed by mdk #31589: Links for French documentation PDF is broken: LaTeX issue with https://bugs.python.org/issue31589 closed by mdk #31813: python -m ensurepip hangs https://bugs.python.org/issue31813 closed by anthonypjshaw #31855: mock_open is not compatible with read(n) (and pickle.load) https://bugs.python.org/issue31855 closed by xtreak #31873: Inconsistent capitalization of proper noun - Unicode. https://bugs.python.org/issue31873 closed by mdk #32393: nav menu jitter in old documentation https://bugs.python.org/issue32393 closed by mdk #32971: Docs on unittest.TestCase.assertRaises() should clarify contex https://bugs.python.org/issue32971 closed by anthonypjshaw #33039: int() and math.trunc don't accept objects that only define __i https://bugs.python.org/issue33039 closed by serhiy.storchaka #33530: Implement Happy Eyeball in asyncio https://bugs.python.org/issue33530 closed by asvetlov #33948: doc truncated lines in PDF https://bugs.python.org/issue33948 closed by mdk #34214: Pylint recusion stack overflow abort https://bugs.python.org/issue34214 closed by cheryl.sabella #34600: python3 regression ElementTree.iterparse() unable to capture c https://bugs.python.org/issue34600 closed by scoder #34820: binascii.c:1578:1: error: the control flow of function ???bina https://bugs.python.org/issue34820 closed by anthonypjshaw #34848: range.index only takes one argument when it's documented as ta https://bugs.python.org/issue34848 closed by vstinner #35125: asyncio shield: remove inner callback on outer cancellation https://bugs.python.org/issue35125 closed by asvetlov #35354: Generator functions stack overflow https://bugs.python.org/issue35354 closed by Mark.Shannon #35393: Typo in documentation https://bugs.python.org/issue35393 closed by cheryl.sabella #35900: Add pickler hook for the user to customize the serialization o https://bugs.python.org/issue35900 closed by pitrou #36166: DOC: Fix markup on function parameter on datamodel.rst https://bugs.python.org/issue36166 closed by cheryl.sabella #36189: DOC: Correct word in tutorial introduction https://bugs.python.org/issue36189 closed by cheryl.sabella #36239: gettext: GNUTranslations doesn't parse properly comments in de https://bugs.python.org/issue36239 closed by mdk #36275: DOC: venv.create doesn't include prompt parameter https://bugs.python.org/issue36275 closed by cheryl.sabella #36533: logging regression with threading + fork are mixed in 3.7.1rc2 https://bugs.python.org/issue36533 closed by gregory.p.smith #36545: Python 3.5 OOM during test_socket on make https://bugs.python.org/issue36545 closed by dekken #36551: Optimize list comprehensions with preallocate size and protect https://bugs.python.org/issue36551 closed by anthonypjshaw #36552: Replace OverflowError with ValueError when calculating length https://bugs.python.org/issue36552 closed by serhiy.storchaka #36601: signals can be caught by any thread https://bugs.python.org/issue36601 closed by gregory.p.smith #36737: Warnings operate out of global runtime state. https://bugs.python.org/issue36737 closed by eric.snow #36766: Typos in docs and code comments https://bugs.python.org/issue36766 closed by terry.reedy #36774: f-strings: Add a !d conversion for ease of debugging https://bugs.python.org/issue36774 closed by eric.smith #36778: test_site.StartupImportTests.test_startup_imports fails if def https://bugs.python.org/issue36778 closed by vstinner #36783: No documentation for _FromXandFold C API functions https://bugs.python.org/issue36783 closed by cheryl.sabella #36784: __import__ with empty folder after importlib.invalidate_caches https://bugs.python.org/issue36784 closed by mdk #36787: Python3 regresison: String formatting of None object https://bugs.python.org/issue36787 closed by eric.smith #36798: f-strings do not support top-level := https://bugs.python.org/issue36798 closed by gvanrossum #36800: Invalid coding error hidden on Windows https://bugs.python.org/issue36800 closed by eryksun #36801: Wait for connection_lost in StreamWriter.drain https://bugs.python.org/issue36801 closed by asvetlov #36802: Revert back StreamWriter awrite/aclose but provide await write https://bugs.python.org/issue36802 closed by asvetlov #36806: Forbid creating of stream objects outside of asyncio https://bugs.python.org/issue36806 closed by asvetlov #36808: Understanding "cannot import name" exception https://bugs.python.org/issue36808 closed by eryksun #36811: Warning when compiling _elementree.c https://bugs.python.org/issue36811 closed by scoder #36812: posix_spawnp returns error when used with file_actions https://bugs.python.org/issue36812 closed by vstinner #36814: posix_spawn explicit file_actions=None throws error https://bugs.python.org/issue36814 closed by vstinner #36815: Test Failure MacOS : test_socket test_host_resolution_bad_addr https://bugs.python.org/issue36815 closed by SilentGhost #36817: Add = to f-strings for easier debugging. https://bugs.python.org/issue36817 closed by eric.smith #36828: Cannot install et-xmlfile https://bugs.python.org/issue36828 closed by xtreak #36830: Typo in collections.deque https://bugs.python.org/issue36830 closed by matrixise #36831: ElementTree.find attribute matching with empty namespace https://bugs.python.org/issue36831 closed by scoder #36832: Port zipp to zipfile https://bugs.python.org/issue36832 closed by barry #36835: Move the warnings runtime state into per-interpreter state. https://bugs.python.org/issue36835 closed by eric.snow #36836: Test Suite not working on 3.4, 3.5 on MacOS Mojave https://bugs.python.org/issue36836 closed by CuriousLearner #36838: running 'make html' from the Doc tree emits an unwelcoming err https://bugs.python.org/issue36838 closed by gregory.p.smith #36844: abiflag `m` is no longer showing when compiled with --enable-s https://bugs.python.org/issue36844 closed by Anthony Sottile #36846: range_iterator does not use __index__ https://bugs.python.org/issue36846 closed by serhiy.storchaka #36847: Segmentation fault (core dumped) Found when we import "schedul https://bugs.python.org/issue36847 closed by jay.net.in #36851: Frame stack is not cleaned after execution is finished with re https://bugs.python.org/issue36851 closed by pablogsal #36857: bisect should support descending order https://bugs.python.org/issue36857 closed by rhettinger #36861: Update to Unicode 12.1.0 https://bugs.python.org/issue36861 closed by benjamin.peterson #36864: Double Spaces in the documentation https://bugs.python.org/issue36864 closed by mdk #36872: passing negative values through modules https://bugs.python.org/issue36872 closed by SilentGhost #1160328: urllib2 post error when using httpproxy https://bugs.python.org/issue1160328 closed by martin.panter From p.f.moore at gmail.com Fri May 10 15:32:12 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 10 May 2019 20:32:12 +0100 Subject: [Python-Dev] Ignore spurious 3.4 checkin messages! In-Reply-To: References: Message-ID: On Fri, 10 May 2019 at 18:54, Ned Deily wrote: > > Eek! I was just doing a bit of branch cleanup in the cpython repo and managed to trigger a bunch (30-ish) of duplicate checkin messages to bugs.python.org for old commits. I will remove them from b.p.o. Please ignore any 3.4 spam email. Sorry! It's not dead! See, it just moved! From vstinner at redhat.com Fri May 10 21:32:26 2019 From: vstinner at redhat.com (Victor Stinner) Date: Sat, 11 May 2019 03:32:26 +0200 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: References: Message-ID: Hi, First of all, I just found an old issue that we will solved by my PEP 587 :-) Add Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal errors https://bugs.python.org/issue30560 I studied code of applications embedding Python. Most of them has to decode bytes strings to get wchar_t* to set home, argv, program name, etc. I'm not sure that they use the "correct" encoding, especially since Python 3.7 got UTF-8 Mode (PEP 540) and C locale coercion (PEP 538). I tried to convert the source code of each project into pseudo-code which looks like C code used in CPython. I removed all error handling code: look at each reference, the original code is usually way more complex. Some project has to wrap each function of the Python C API manually, which adds even more boilerplate code. Some project set/unset environment varaibles. Others prefer global configuration variables like Py_NoSiteFlag. It seems like Py_FrozenFlag is commonly used. Maybe I should make the flag public and try to find it a better name: /* If greater than 0, suppress _PyPathConfig_Calculate() warnings. If set to -1 (default), inherit Py_FrozenFlag value. */ int _frozen; About pyinstaller which changes C standard stream buffering: Py_Initialize() now also does that when buffered_stdio=0. See config_init_stdio() in Python/coreconfig.c. Moreover, this function now *always* set standard streams to O_BINARY mode on Windows. I'm not sure if it's correct or not. Blender ------- Pseudo-code of BPY_python_start:: BLI_strncpy_wchar_from_utf8(program_path_wchar, BKE_appdir_program_path()); Py_SetProgramName(program_path_wchar); PyImport_ExtendInittab(bpy_internal_modules); Py_SetPythonHome(py_path_bundle_wchar); Py_SetStandardStreamEncoding("utf-8", "surrogateescape"); Py_NoSiteFlag = 1; Py_FrozenFlag = 1; Py_Initialize(); Ref: https://git.blender.org/gitweb/gitweb.cgi/blender.git/blob/HEAD:/source/blender/python/intern/bpy_interface.c fontforge --------- Pseudo-code of fontforge when Python is used to run a script:: Py_Initialize() for init_file in init_files: PyRun_SimpleFileEx(init_file) exitcode = Py_Main(arg, argv) Py_Finalize() exit(exitcode) Ref: https://bugs.python.org/issue36204#msg337256 py2app ------ Pseudo-code:: unsetenv("PYTHONOPTIMIZE"); unsetenv("PYTHONDEBUG"); unsetenv("PYTHONDONTWRITEBYTECODE"); unsetenv("PYTHONIOENCODING"); unsetenv("PYTHONDUMPREFS"); unsetenv("PYTHONMALLOCSTATS"); setenv("PYTHONDONTWRITEBYTECODE", "1", 1); setenv("PYTHONUNBUFFERED", "1", 1); setenv("PYTHONPATH", build_python_path(), 1); setlocale(LC_ALL, "en_US.UTF-8"); mbstowcs(w_program, c_program, PATH_MAX+1); Py_SetProgramName(w_program); Py_Initialize() argv_new[0] = _Py_DecodeUTF8_surrogateescape(script, strlen(script)); ... PySys_SetArgv(argc, argv_new); PyRun_SimpleFile(fp, script); Py_Finalize(); Ref: https://bitbucket.org/ronaldoussoren/py2app/src/default/py2app/apptemplate/src/main.c See also: https://bitbucket.org/ronaldoussoren/py2app/src/default/py2app/bundletemplate/src/main.m OpenOffice ---------- Pseudo-code of ``PythonInit``:: mbstowcs(wide, home, PATH_MAX + 1); Py_SetPythonHome(wide); setenv("PYTHONPATH", getenv("PYTHONPATH") + ":" + path_bootstrap); PyImport_AppendInittab("pyuno", PyInit_pyuno); Py_DontWriteBytecodeFlag = 1; Py_Initialize(); Ref: pyuno/source/loader/pyuno_loader.cxx, see: https://docs.libreoffice.org/pyuno/html/pyuno__loader_8cxx_source.html vim --- Pseudo-code:: mbstowcs(py_home_buf, p_py3home); Py_SetPythonHome(py_home_buf); PyImport_AppendInittab("vim", Py3Init_vim); Py_Initialize(); Ref: https://github.com/vim/vim/blob/master/src/if_python3.c pyinstaller ----------- Pseudo-code:: pyi_locale_char2wchar(progname_w, status->archivename) SetProgramName(progname_w); pyi_locale_char2wchar(pyhome_w, status->mainpath) SetPythonHome(pyhome_w); pypath_w = build_path(); Py_SetPath(pypath_w); Py_NoSiteFlag = 1; Py_FrozenFlag = 1; Py_DontWriteBytecodeFlag = 1; Py_NoUserSiteDirectory = 1; Py_IgnoreEnvironmentFlag = 1; Py_VerboseFlag = 0; Py_OptimizeFlag = 1; if (unbuffered) { #ifdef _WIN32 _setmode(fileno(stdin), _O_BINARY); _setmode(fileno(stdout), _O_BINARY); #endif setbuf(stdin, (char *)NULL); setbuf(stdout, (char *)NULL); setbuf(stderr, (char *)NULL); } Py_Initialize(); PySys_SetPath(pypath_w); PySys_SetArgvEx(argc, wargv, 0); Ref: https://github.com/pyinstaller/pyinstaller/blob/1844d69f5aa1d64d3feca912ed1698664a3faf3e/bootloader/src/pyi_pythonlib.c Victor From tjreedy at udel.edu Sat May 11 18:24:46 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 11 May 2019 18:24:46 -0400 Subject: [Python-Dev] Online Devguide mostly not working Message-ID: https://devguide.python.org gives the intro page with TOC on sidebar and at end. Clicking anything, such as Getting Started, which tries to display https://devguide.python.org/setup/, returns a Read the Docs page "Sorry This page does not exist yet." 'Down for everyone' site also cannot access. -- Terry Jan Reedy From jcgoble3 at gmail.com Sat May 11 18:37:06 2019 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Sat, 11 May 2019 18:37:06 -0400 Subject: [Python-Dev] Online Devguide mostly not working In-Reply-To: References: Message-ID: Confirming that I also cannot access the Getting Started page. I'm in Ohio, if it matters. On Sat, May 11, 2019 at 6:26 PM Terry Reedy wrote: > > https://devguide.python.org gives the intro page with TOC on sidebar and > at end. Clicking anything, such as Getting Started, which tries to > display https://devguide.python.org/setup/, returns a Read the Docs page > "Sorry This page does not exist yet." 'Down for everyone' site also > cannot access. > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jcgoble3%40gmail.com From wes.turner at gmail.com Sun May 12 03:10:57 2019 From: wes.turner at gmail.com (Wes Turner) Date: Sun, 12 May 2019 03:10:57 -0400 Subject: [Python-Dev] Online Devguide mostly not working In-Reply-To: References: Message-ID: https://cpython-devguide.readthedocs.io/ seems to work but https://devguide.python.org/* does not https://readthedocs.org/projects/cpython-devguide/ lists maintainers, who I've cc'd AFAIU, there's no reason that the HTTP STS custom domain CNAME support would've broken this: https://github.com/rtfd/readthedocs.org/issues/4135 On Saturday, May 11, 2019, Jonathan Goble wrote: > Confirming that I also cannot access the Getting Started page. I'm in > Ohio, if it matters. > > On Sat, May 11, 2019 at 6:26 PM Terry Reedy wrote: > > > > https://devguide.python.org gives the intro page with TOC on sidebar and > > at end. Clicking anything, such as Getting Started, which tries to > > display https://devguide.python.org/setup/, returns a Read the Docs page > > "Sorry This page does not exist yet." 'Down for everyone' site also > > cannot access. > > > > -- > > Terry Jan Reedy > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > jcgoble3%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > wes.turner%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ja.py at farowl.co.uk Sun May 12 09:17:59 2019 From: ja.py at farowl.co.uk (Jeff Allen) Date: Sun, 12 May 2019 14:17:59 +0100 Subject: [Python-Dev] Online Devguide mostly not working In-Reply-To: References: Message-ID: <0dd06bd3-3a69-0f49-8f6b-93aa81045767@farowl.co.uk> Is it suspicious that in the detailed log we see: 'canonical_url': 'http://devguide.python.org/', ? I guess this comes from project admin configuration at RTD, additional to your conf.py. https://docs.readthedocs.io/en/stable/canonical.html (Just guessing really.) Jeff Allen On 12/05/2019 08:10, Wes Turner wrote: > https://cpython-devguide.readthedocs.io/ seems to work but > https://devguide.python.org /* does not > > https://readthedocs.org/projects/cpython-devguide/ lists maintainers, > who I've cc'd > > AFAIU, there's no reason that the HTTP STS custom domain CNAME support > would've broken this: > https://github.com/rtfd/readthedocs.org/issues/4135 > > On Saturday, May 11, 2019, Jonathan Goble > wrote: > > Confirming that I also cannot access the Getting Started page. I'm in > Ohio, if it matters. > > On Sat, May 11, 2019 at 6:26 PM Terry Reedy > wrote: > > > > https://devguide.python.org gives the intro page with TOC on > sidebar and > > at end.? Clicking anything, such as Getting Started, which tries to > > display https://devguide.python.org/setup/ > , returns a Read the Docs page > > "Sorry This page does not exist yet."? 'Down for everyone' site also > > cannot access. > > > > -- > > Terry Jan Reedy > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jcgoble3%40gmail.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ja.py%40farowl.co.uk -------------- next part -------------- An HTML attachment was scrubbed... URL: From willingc at willingconsulting.com Sun May 12 12:51:16 2019 From: willingc at willingconsulting.com (Carol Willing) Date: Sun, 12 May 2019 09:51:16 -0700 Subject: [Python-Dev] Online Devguide Status: Working Again Message-ID: Sorry for the inconvenience yesterday. From steve.dower at python.org Mon May 13 12:28:36 2019 From: steve.dower at python.org (Steve Dower) Date: Mon, 13 May 2019 09:28:36 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: References: Message-ID: <6246039b-2ae6-5637-2e95-13b166d8d693@python.org> On 10May2019 1832, Victor Stinner wrote: > Hi, > > First of all, I just found an old issue that we will solved by my PEP 587 :-) > > Add Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal errors > https://bugs.python.org/issue30560 Yes, this should be a feature of any redesigned embedding API. > I studied code of applications embedding Python. Most of them has to > decode bytes strings to get wchar_t* to set home, argv, program name, > etc. I'm not sure that they use the "correct" encoding, especially > since Python 3.7 got UTF-8 Mode (PEP 540) and C locale coercion (PEP > 538). Unless you studied Windows-only applications embedding Python, _all_ of them will have had to decode strings into Unicode, since that's what our API expects. All of the Windows-only applications I know of that embed Python are closed source, and none are owned by Red Hat. I'm going to assume you missed that entire segment of the ecosystem :) But it also seems like perhaps we just need to expose a single API that does "decode this like CPython would" so that they can call it? We don't need a whole PEP or a widely publicised and discussed redesign of embedding to add this, and since it would solve a very real problem then we should just do it. > I tried to convert the source code of each project into pseudo-code > which looks like C code used in CPython. Thanks, this is helpful! My take: * all the examples are trying to be isolated from the system Python install (except Vim?) * all the examples want to import some of their own modules before running user code * nobody understands how to configure embedded Python :) Also from my own work with/on other projects: * embedders need to integrate native thread management with Python threads * embedders want to use their own files/libraries * embedders want to totally override getpath, not augment/configure it Cheers, Steve From vstinner at redhat.com Mon May 13 17:52:04 2019 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 13 May 2019 23:52:04 +0200 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: <6246039b-2ae6-5637-2e95-13b166d8d693@python.org> References: <6246039b-2ae6-5637-2e95-13b166d8d693@python.org> Message-ID: )Le lun. 13 mai 2019 ? 18:28, Steve Dower a ?crit : > My take: > * all the examples are trying to be isolated from the system Python > install (except Vim?) "Isolation" means different things: * ignore configuration files * ignore environment variables * custom path configuration (sys.path, sys.executable, etc.) It seems like the most common need is to have a custom path configuration. Py_IsolatedFlag isn't used. Only py2app manually ignores a few environment variables. > * all the examples want to import some of their own modules before > running user code Well, running code between Py_Initialize() and running the final Python code is not new, and my PEP doesn't change anything here: it's still possible, as it was previously. You can use PyRun_SimpleFile() after Py_Initialize() for example. Maybe I misunderstood your point. > * nobody understands how to configure embedded Python :) Well, that's the problem I'm trying to solve by designing an homogeneous API, rather than scattered global configuration variables, environment variables, function calls, etc. > Also from my own work with/on other projects: > * embedders need to integrate native thread management with Python threads Sorry, I see the relationship with the initialization. > * embedders want to use their own files/libraries That's the path configuration, no? > * embedders want to totally override getpath, not augment/configure it On Python 3.7, Py_SetPath() is the closest thing to configure path configuration. But I don't see how to override sys.executable (Py_GetProgramFullPath), sys.prefix, sys.exec_prefix, nor (internal) dll_path. In the examples that I found, SetProgramName(), SetPythonHome() and Py_SetPath() are called. My PEP 587 allows to completely ignore getpath.c/getpath.c easily by setting explicitly: * use_module_search_path, module_search_paths * executable * prefix * exec_prefix * dll_path (Windows only) If you set these fields, you fully control where Python looks for modules. Extract of the C code: /* Do we need to calculate the path? */ if (!config->use_module_search_paths || (config->executable == NULL) || (config->prefix == NULL) #ifdef MS_WINDOWS || (config->dll_path == NULL) #endif || (config->exec_prefix == NULL)) { _PyInitError err = _PyCoreConfig_CalculatePathConfig(config); if (_Py_INIT_FAILED(err)) { return err; } } OpenOffice doesn't bother with complex code, it just appends a path to PYTHONPATH: setenv("PYTHONPATH", getenv("PYTHONPATH") + ":" + path_bootstrap); It can use PyWideStringList_Append(&config.module_search_paths, path_bootstrap), as shown in one example of my PEP. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From steve.dower at python.org Mon May 13 18:52:10 2019 From: steve.dower at python.org (Steve Dower) Date: Mon, 13 May 2019 15:52:10 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: References: <6246039b-2ae6-5637-2e95-13b166d8d693@python.org> Message-ID: In response to all of your responses: No need to take offense, I was merely summarising the research you posted in a way that looks more like scenarios or requirements. It's a typical software engineering task. Being able to collect snippets and let people draw their own conclusions is one thing, but those of us (including yourself) who are actively working in this area are totally allowed to present our analysis as well. Given the raw material, the summary, and the recommendations, anyone else can do the same analysis and join the discussion, and that's what we're doing. But you can't simply present raw material and assume that people will naturally end up at the same conclusion - that's how you end up with overly simplistic plans where everyone "agrees" because they projected their own opinions into it, then are surprised when it turns out that other people had different opinions. Cheers, Steve On 13May2019 1452, Victor Stinner wrote: > )Le lun. 13 mai 2019 ? 18:28, Steve Dower a ?crit : >> My take: >> * all the examples are trying to be isolated from the system Python >> install (except Vim?) > > "Isolation" means different things: > > * ignore configuration files > * ignore environment variables > * custom path configuration (sys.path, sys.executable, etc.) > > It seems like the most common need is to have a custom path configuration. > > Py_IsolatedFlag isn't used. Only py2app manually ignores a few > environment variables. > > >> * all the examples want to import some of their own modules before >> running user code > > Well, running code between Py_Initialize() and running the final > Python code is not new, and my PEP doesn't change anything here: it's > still possible, as it was previously. You can use PyRun_SimpleFile() > after Py_Initialize() for example. > > Maybe I misunderstood your point. > > >> * nobody understands how to configure embedded Python :) > > Well, that's the problem I'm trying to solve by designing an > homogeneous API, rather than scattered global configuration variables, > environment variables, function calls, etc. > > >> Also from my own work with/on other projects: >> * embedders need to integrate native thread management with Python threads > > Sorry, I see the relationship with the initialization. > > >> * embedders want to use their own files/libraries > > That's the path configuration, no? > > >> * embedders want to totally override getpath, not augment/configure it > > On Python 3.7, Py_SetPath() is the closest thing to configure path > configuration. But I don't see how to override sys.executable > (Py_GetProgramFullPath), sys.prefix, sys.exec_prefix, nor (internal) > dll_path. > > In the examples that I found, SetProgramName(), SetPythonHome() and > Py_SetPath() are called. > > My PEP 587 allows to completely ignore getpath.c/getpath.c easily by > setting explicitly: > > * use_module_search_path, module_search_paths > * executable > * prefix > * exec_prefix > * dll_path (Windows only) > > If you set these fields, you fully control where Python looks for > modules. Extract of the C code: > > /* Do we need to calculate the path? */ > if (!config->use_module_search_paths > || (config->executable == NULL) > || (config->prefix == NULL) > #ifdef MS_WINDOWS > || (config->dll_path == NULL) > #endif > || (config->exec_prefix == NULL)) > { > _PyInitError err = _PyCoreConfig_CalculatePathConfig(config); > if (_Py_INIT_FAILED(err)) { > return err; > } > } > > OpenOffice doesn't bother with complex code, it just appends a path to > PYTHONPATH: > > setenv("PYTHONPATH", getenv("PYTHONPATH") + ":" + path_bootstrap); > > It can use PyWideStringList_Append(&config.module_search_paths, > path_bootstrap), as shown in one example of my PEP. > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > From vstinner at redhat.com Mon May 13 19:52:06 2019 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 14 May 2019 01:52:06 +0200 Subject: [Python-Dev] Adding a tzidx cache to datetime In-Reply-To: <619825e1-be2e-b8e7-6eef-07921baebc7f@egenix.com> References: <619825e1-be2e-b8e7-6eef-07921baebc7f@egenix.com> Message-ID: Le ven. 10 mai 2019 ? 09:22, M.-A. Lemburg a ?crit : > Given that many datetime objects in practice don't use timezones > (e.g. in large data stores you typically use UTC and naive datetime > objects), I think that making the object itself larger to accommodate > for a cache, which will only be used a smaller percentage of the use > cases, isn't warranted. Going from 64 bytes to 72 bytes also sounds > like this could have negative effects on cache lines. > > If you need a per object cache, you can either use weakref > objects or maintain a separate dictionary in dateutil or other > timezone helpers which indexes objects by id(obj). > > That said, if you only add a byte field which doesn't make the object > larger in practice (you merely use space that alignments would > use anyway), this shouldn't be a problem. The use of that field > should be documented, though, so that other implementations can > use/provide it as well. >From Marc-Andre Lemburg, I understand that Paul's PR is a good compromise and that other datetime implementations which cannot use tzidx() cache (because it's limited to an integer in [0; 254]) can subclass datetime or use a cache outside datetime. Note: right now, creating a weakref to a datetime fails. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From paul at ganssle.io Mon May 13 20:01:39 2019 From: paul at ganssle.io (Paul Ganssle) Date: Mon, 13 May 2019 20:01:39 -0400 Subject: [Python-Dev] Adding a tzidx cache to datetime In-Reply-To: References: <619825e1-be2e-b8e7-6eef-07921baebc7f@egenix.com> Message-ID: <89f632ac-de3e-a985-ed92-c314fd16a88f@ganssle.io> > From Marc-Andre Lemburg, I understand that Paul's PR is a good > compromise and that other datetime implementations which cannot use > tzidx() cache (because it's limited to an integer in [0; 254]) can > subclass datetime or use a cache outside datetime. One idea that we can put out there (though I'm hesitant to suggest it, because generally Python avoids this sort of language lawyering anyway), is that I think it's actually fine to allow the situations under which `tzidx()` will cache a value could be implementation-dependent, and to document that in CPython it's only integers? in [0; 254]. The reason to mention this is that I suspect that PyPy, which has a pure-python implementation of datetime, will likely either choose to forgo the cache entirely and always fall through to the underlying function call or cache /any/ Python object returned, since with a pure Python implementation, they do not have the advantage of storing the tzidx cache in an unused padding byte. Other than the speed concerns, because of the fallback nature of datetime.tzidx, whether or not the cache is hit will not be visible to the end user, so I think it's fair to allow interpreter implementations to choose when a value is or is not cached according to what works best for their users. On 5/13/19 7:52 PM, Victor Stinner wrote: > Le ven. 10 mai 2019 ? 09:22, M.-A. Lemburg a ?crit : >> Given that many datetime objects in practice don't use timezones >> (e.g. in large data stores you typically use UTC and naive datetime >> objects), I think that making the object itself larger to accommodate >> for a cache, which will only be used a smaller percentage of the use >> cases, isn't warranted. Going from 64 bytes to 72 bytes also sounds >> like this could have negative effects on cache lines. >> >> If you need a per object cache, you can either use weakref >> objects or maintain a separate dictionary in dateutil or other >> timezone helpers which indexes objects by id(obj). >> >> That said, if you only add a byte field which doesn't make the object >> larger in practice (you merely use space that alignments would >> use anyway), this shouldn't be a problem. The use of that field >> should be documented, though, so that other implementations can >> use/provide it as well. > From Marc-Andre Lemburg, I understand that Paul's PR is a good > compromise and that other datetime implementations which cannot use > tzidx() cache (because it's limited to an integer in [0; 254]) can > subclass datetime or use a cache outside datetime. > > Note: right now, creating a weakref to a datetime fails. > > Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From brett at python.org Mon May 13 23:49:15 2019 From: brett at python.org (Brett Cannon) Date: Mon, 13 May 2019 20:49:15 -0700 Subject: [Python-Dev] Online Devguide mostly not working In-Reply-To: <0dd06bd3-3a69-0f49-8f6b-93aa81045767@farowl.co.uk> References: <0dd06bd3-3a69-0f49-8f6b-93aa81045767@farowl.co.uk> Message-ID: It's working for me, so it was probably just a hiccup. On Sun, May 12, 2019 at 6:19 AM Jeff Allen wrote: > Is it suspicious that in the detailed log we see: > > 'canonical_url': 'http://devguide.python.org/', > > ? I guess this comes from project admin configuration at RTD, additional > to your conf.py. > > https://docs.readthedocs.io/en/stable/canonical.html > > (Just guessing really.) > > Jeff Allen > > On 12/05/2019 08:10, Wes Turner wrote: > > https://cpython-devguide.readthedocs.io/ seems to work but > https://devguide.python.org/* does not > > https://readthedocs.org/projects/cpython-devguide/ lists maintainers, who > I've cc'd > > AFAIU, there's no reason that the HTTP STS custom domain CNAME support > would've broken this: > https://github.com/rtfd/readthedocs.org/issues/4135 > > On Saturday, May 11, 2019, Jonathan Goble wrote: > >> Confirming that I also cannot access the Getting Started page. I'm in >> Ohio, if it matters. >> >> On Sat, May 11, 2019 at 6:26 PM Terry Reedy wrote: >> > >> > https://devguide.python.org gives the intro page with TOC on sidebar >> and >> > at end. Clicking anything, such as Getting Started, which tries to >> > display https://devguide.python.org/setup/, returns a Read the Docs >> page >> > "Sorry This page does not exist yet." 'Down for everyone' site also >> > cannot access. >> > >> > -- >> > Terry Jan Reedy >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/jcgoble3%40gmail.com >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com >> > > _______________________________________________ > Python-Dev mailing listPython-Dev at python.orghttps://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ja.py%40farowl.co.uk > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ja.py at farowl.co.uk Tue May 14 02:52:36 2019 From: ja.py at farowl.co.uk (Jeff Allen) Date: Tue, 14 May 2019 07:52:36 +0100 Subject: [Python-Dev] Online Devguide mostly not working In-Reply-To: References: <0dd06bd3-3a69-0f49-8f6b-93aa81045767@farowl.co.uk> Message-ID: Hiccups cured thanks to Mariatta and Carol https://github.com/python/devguide/pull/484. (And my guess was wrong.) Jeff Allen On 14/05/2019 04:49, Brett Cannon wrote: > It's working for me, so it was probably just a hiccup. > > On Sun, May 12, 2019 at 6:19 AM Jeff Allen > wrote: > > Is it suspicious that in the detailed log we see: > > 'canonical_url': 'http://devguide.python.org/', > > ? I guess this comes from project admin configuration at RTD, > additional to your conf.py. > > https://docs.readthedocs.io/en/stable/canonical.html > > (Just guessing really.) > > Jeff Allen > > On 12/05/2019 08:10, Wes Turner wrote: >> https://cpython-devguide.readthedocs.io/ seems to work but >> https://devguide.python.org /* does >> not >> >> https://readthedocs.org/projects/cpython-devguide/ lists >> maintainers, who I've cc'd >> >> AFAIU, there's no reason that the HTTP STS custom domain CNAME >> support would've broken this: >> https://github.com/rtfd/readthedocs.org/issues/4135 >> >> On Saturday, May 11, 2019, Jonathan Goble > > wrote: >> >> Confirming that I also cannot access the Getting Started >> page. I'm in >> Ohio, if it matters. >> >> On Sat, May 11, 2019 at 6:26 PM Terry Reedy > > wrote: >> > >> > https://devguide.python.org gives the intro page with TOC >> on sidebar and >> > at end.? Clicking anything, such as Getting Started, which >> tries to >> > display https://devguide.python.org/setup/, returns a Read >> the Docs page >> > "Sorry This page does not exist yet."? 'Down for everyone' >> site also >> > cannot access. >> > >> > -- >> > Terry Jan Reedy >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/jcgoble3%40gmail.com >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/wes.turner%40gmail.com >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe:https://mail.python.org/mailman/options/python-dev/ja.py%40farowl.co.uk > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Tue May 14 13:52:15 2019 From: steve.dower at python.org (Steve Dower) Date: Tue, 14 May 2019 10:52:15 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: References: Message-ID: <46a53765-e3dc-ac8f-bc39-2cc9077b9205@python.org> On 10May2019 1832, Victor Stinner wrote: > I studied code of applications embedding Python. Most of them has to > decode bytes strings to get wchar_t* to set home, argv, program name, > etc. I'm not sure that they use the "correct" encoding, especially > since Python 3.7 got UTF-8 Mode (PEP 540) and C locale coercion (PEP > 538). It looks like Py_DecodeLocale() is available very early on - why wouldn't we recommend using this function? It seems to be nearly a drop-in replacement for mbtowcs in the samples, and if memory allocation is a big deal perhaps we could just add a version that writes to a buffer? That would provide a supported workaround for the encoding issues and unblock people hitting trouble right now, yes? Cheers, Steve From barry at python.org Tue May 14 21:11:14 2019 From: barry at python.org (Barry Warsaw) Date: Tue, 14 May 2019 18:11:14 -0700 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted Message-ID: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> As the BDFL-Delegate for PEP 581, and with the unanimous backing of the rest of the Steering Council, I hereby Accept this PEP. We would like to thank Mariatta for championing PEP 581, and to all the contributors to the discussion, both pro and con. We appreciate your candor and respect for your fellow developers. The SC believes that this migration is in the best interest of the Python community, and we look forward to the elaboration of the detailed migration plan (PEP 588). We also extend our heartfelt thanks Berker, Ezio, and everyone who has helped develop and maintain Roundup and bugs.python.org over the years. bpo has been a critical component for Python development for a very long time, and we all greatly appreciate the time, effort, and devotion you have put into this resource. https://www.python.org/dev/peps/pep-0581/ Onward we go! The migration will be a large effort, with much planning, development, and testing, and we welcome volunteers who wish to help make it a reality. I look forward to your contributions on PEP 588 and the actual work of migrating issues to GitHub. Cheers, -Barry (BDFL-Delegate, and on behalf of the Python Steering Council) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From vstinner at redhat.com Tue May 14 21:42:17 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 15 May 2019 03:42:17 +0200 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> Message-ID: Congrats Mariatta :-) Victor Le mer. 15 mai 2019 ? 03:14, Barry Warsaw a ?crit : > > As the BDFL-Delegate for PEP 581, and with the unanimous backing of the rest of the Steering Council, I hereby Accept this PEP. > > We would like to thank Mariatta for championing PEP 581, and to all the contributors to the discussion, both pro and con. We appreciate your candor and respect for your fellow developers. The SC believes that this migration is in the best interest of the Python community, and we look forward to the elaboration of the detailed migration plan (PEP 588). > > We also extend our heartfelt thanks Berker, Ezio, and everyone who has helped develop and maintain Roundup and bugs.python.org over the years. bpo has been a critical component for Python development for a very long time, and we all greatly appreciate the time, effort, and devotion you have put into this resource. > > https://www.python.org/dev/peps/pep-0581/ > > Onward we go! The migration will be a large effort, with much planning, development, and testing, and we welcome volunteers who wish to help make it a reality. I look forward to your contributions on PEP 588 and the actual work of migrating issues to GitHub. > > Cheers, > -Barry (BDFL-Delegate, and on behalf of the Python Steering Council) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com -- Night gathers, and now my watch begins. It shall not end until my death. From steve at holdenweb.com Wed May 15 03:40:58 2019 From: steve at holdenweb.com (Steve Holden) Date: Wed, 15 May 2019 08:40:58 +0100 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> Message-ID: As a mere user I'd like to thank the developer team for biting this bullet. Remembering the transition to Git I am sure that the further democratisation (?) of the development process will similarly increase the diversity and scope of the development effort. It will indeed be a significant effort to effect this migration. While it's inevitable this will involve significant dev time, perhaps there roles can be identified specifically as to be filled by _other_ than core devs. On Wed, May 15, 2019 at 2:43 AM Victor Stinner wrote: > Congrats Mariatta :-) > > Victor > > Le mer. 15 mai 2019 ? 03:14, Barry Warsaw a ?crit : > > > > As the BDFL-Delegate for PEP 581, and with the unanimous backing of the > rest of the Steering Council, I hereby Accept this PEP. > > > > We would like to thank Mariatta for championing PEP 581, and to all the > contributors to the discussion, both pro and con. We appreciate your > candor and respect for your fellow developers. The SC believes that this > migration is in the best interest of the Python community, and we look > forward to the elaboration of the detailed migration plan (PEP 588). > > > > We also extend our heartfelt thanks Berker, Ezio, and everyone who has > helped develop and maintain Roundup and bugs.python.org over the years. > bpo has been a critical component for Python development for a very long > time, and we all greatly appreciate the time, effort, and devotion you have > put into this resource. > > > > https://www.python.org/dev/peps/pep-0581/ > > > > Onward we go! The migration will be a large effort, with much planning, > development, and testing, and we welcome volunteers who wish to help make > it a reality. I look forward to your contributions on PEP 588 and the > actual work of migrating issues to GitHub. > > > > Cheers, > > -Barry (BDFL-Delegate, and on behalf of the Python Steering Council) > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > > > > -- > Night gathers, and now my watch begins. It shall not end until my death. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed May 15 04:42:55 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 May 2019 10:42:55 +0200 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> Message-ID: <20190515104255.27236e1a@fsol> On Wed, 15 May 2019 08:40:58 +0100 Steve Holden wrote: > As a mere user I'd like to thank the developer team for biting this bullet. > Remembering the transition to Git I am sure that the further > democratisation (?) of the development process will similarly increase the > diversity and scope of the development effort. "Similarly"? We still lack some metrics on that point, IIRC. Regards Antoine. From solipsis at pitrou.net Wed May 15 04:48:30 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 May 2019 10:48:30 +0200 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> Message-ID: <20190515104830.5b307230@fsol> On Tue, 14 May 2019 18:11:14 -0700 Barry Warsaw wrote: > As the BDFL-Delegate for PEP 581, and with the unanimous backing of the rest of the Steering Council, I hereby Accept this PEP. For future reference, is it possible to post the Steering Council's reflection and rationale on the PEP? Thank you Regards Antoine. From vano at mail.mipt.ru Wed May 15 04:55:23 2019 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 15 May 2019 11:55:23 +0300 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: <20190515104830.5b307230@fsol> References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: On 15.05.2019 11:48, Antoine Pitrou wrote: > On Tue, 14 May 2019 18:11:14 -0700 > Barry Warsaw wrote: > >> As the BDFL-Delegate for PEP 581, and with the unanimous backing of the rest of the Steering Council, I hereby Accept this PEP. > For future reference, is it possible to post the Steering Council's > reflection and rationale on the PEP? +1. Specifically, I'd like to know if the risks and the potential for GitHub missing any needed features were estimated. The PEP says nothing about this. > Thank you > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From venthur at debian.org Wed May 15 04:44:35 2019 From: venthur at debian.org (Bastian Venthur) Date: Wed, 15 May 2019 10:44:35 +0200 Subject: [Python-Dev] Adding a toml module to the standard lib? Message-ID: Hi, I'd like to discuss the idea to add a module to parse TOML [toml-lang] to Python's standard library. PEP-0518 -- Specifying Minimum Build System Requirements for Python Projects [pep] suggests to store build system dependencies in `pyproject.toml`, yet Python itself does not support this format. Various packaging related projects like pip and pipenv already support PEP-0518 and vendored one of the existing TOML libraries in order to read `pyproject.toml` files. Besides that, TOML seems a better alternative to .cfg/.ini, .json -- both of which are already supported by Python's standard lib and parsing/dumping TOML properly is tricky enough to solve it properly [requirements]. There are a couple of TOML implementations out there [toml, pytoml, tomlkit] and one would have to find out which one to prefer and migrate into the stdlib. If the result of this discussion is leaning towards adding TOML, I'd volunteer to do it. This includes: coordinating with the maintainer of the chosen library, writing the PEP (hopefully with some help) and maintain the module for at least two years. Cheers, Bastian [toml-lang]: https://github.com/toml-lang/toml [pep]: https://www.python.org/dev/peps/pep-0518 [pip]: https://github.com/pypa/pip [pipenv]: https://github.com/pypa/pipenv [toml]: https://github.com/uiri/toml [pytoml]: https://github.com/avakar/pytoml [tomlkit]: https://github.com/sdispater/tomlkit [requirements]: https://devguide.python.org/stdlibchanges/#requirements -- Dr. Bastian Venthur http://venthur.de Debian Developer venthur at debian org From christian at python.org Wed May 15 05:28:21 2019 From: christian at python.org (Christian Heimes) Date: Wed, 15 May 2019 11:28:21 +0200 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: <4838c6f5-27cc-3823-4ba5-c20a6d0867d9@python.org> On 15/05/2019 10.55, Ivan Pozdeev via Python-Dev wrote: > On 15.05.2019 11:48, Antoine Pitrou wrote: >> On Tue, 14 May 2019 18:11:14 -0700 >> Barry Warsaw wrote: >> >>> As the BDFL-Delegate for PEP 581, and with the unanimous backing of >>> the rest of the Steering Council, I hereby Accept this PEP. >> For future reference, is it possible to post the Steering Council's >> reflection and rationale on the PEP? > +1. Specifically, I'd like to know if the risks and the potential for > GitHub missing any needed features were estimated. The PEP says nothing > about this. I'm not happy with the fact, that the PEP does not mention any arguments or concerns that were raised against the current feature set of Github's issue tracker. In the past, PEPs also listed risks and arguments against a change, then weighted the arguments. What are the next step? Will there be another PEP that explores how we are going to deal with migration, workflow changes, and how we plan to map current BPO features to Github? Christian From solipsis at pitrou.net Wed May 15 05:33:41 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 15 May 2019 11:33:41 +0200 Subject: [Python-Dev] Adding a toml module to the standard lib? References: Message-ID: <20190515113341.19187011@fsol> On Wed, 15 May 2019 10:44:35 +0200 Bastian Venthur wrote: > Hi, > > I'd like to discuss the idea to add a module to parse TOML [toml-lang] > to Python's standard library. How stable is the TOML format? Is it bound to change significantly in the coming years? If the format is stable enough, then I think it's a good idea. Regards Antoine. From p.f.moore at gmail.com Wed May 15 05:40:07 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 May 2019 10:40:07 +0100 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: <20190515104830.5b307230@fsol> References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: On Wed, 15 May 2019 at 09:51, Antoine Pitrou wrote: > > On Tue, 14 May 2019 18:11:14 -0700 > Barry Warsaw wrote: > > > As the BDFL-Delegate for PEP 581, and with the unanimous backing of the rest of the Steering Council, I hereby Accept this PEP. > > For future reference, is it possible to post the Steering Council's > reflection and rationale on the PEP? Also, is there an archive of the discussions anywhere? The PEP says discussions happened on Zulip, but I don't follow that and I don't know where I can find an archived copy of the discussions. It's not as if I'm going to object to the PEP (I'd have participated in the discussions if I had a strong opinion) but I am curious. Paul From venthur at debian.org Wed May 15 06:47:08 2019 From: venthur at debian.org (Bastian Venthur) Date: Wed, 15 May 2019 12:47:08 +0200 Subject: [Python-Dev] Adding a toml module to the standard lib? In-Reply-To: <20190515113341.19187011@fsol> References: <20190515113341.19187011@fsol> Message-ID: On 15.05.19 11:33, Antoine Pitrou wrote: > How stable is the TOML format? Is it bound to change significantly in > the coming years? > > If the format is stable enough, then I think it's a good idea. The last update to the spec [1] was 10 months ago and added a few features. The version before that was stable for more than 3 years. It is also worth noting that he writes about the current version [2]: "As of version 0.5.0, TOML should be considered extremely stable. The goal is for version 1.0.0 to be backwards compatible (as much as humanly possible) with version 0.5.0. All implementations are strongly encouraged to become 0.5.0 compatible so that the transition to 1.0.0 will be simple when that happens." That is of course no guarantee, but maybe the best we can hope for. Cheers, Bastian [1]: https://github.com/toml-lang/toml/releases [2]: https://github.com/toml-lang/toml -- Dr. Bastian Venthur http://venthur.de Debian Developer venthur at debian org From p.f.moore at gmail.com Wed May 15 07:00:09 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 May 2019 12:00:09 +0100 Subject: [Python-Dev] Adding a toml module to the standard lib? In-Reply-To: References: <20190515113341.19187011@fsol> Message-ID: On Wed, 15 May 2019 at 11:49, Bastian Venthur wrote: > > On 15.05.19 11:33, Antoine Pitrou wrote: > > How stable is the TOML format? Is it bound to change significantly in > > the coming years? > > > > If the format is stable enough, then I think it's a good idea. > > > The last update to the spec [1] was 10 months ago and added a few > features. The version before that was stable for more than 3 years. It > is also worth noting that he writes about the current version [2]: I think it's a good idea too - the analysis in PEP 518 concluded that TOML was, while maybe not perfect, certainly a good general purpose config file format, and I think that analysis is applicable in other contexts as well. While "easy to install/vendor" was a point in the analysis, I think having TOML support in the stdlib would be useful for simple scripts/applications looking for a good "out of the box" config solution. Paul From steve at pearwood.info Wed May 15 07:35:16 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 May 2019 21:35:16 +1000 Subject: [Python-Dev] Adding a toml module to the standard lib? In-Reply-To: References: Message-ID: <20190515113516.GP5010@ando.pearwood.info> On Wed, May 15, 2019 at 10:44:35AM +0200, Bastian Venthur wrote: > Hi, > > I'd like to discuss the idea to add a module to parse TOML [toml-lang] > to Python's standard library. You ought to read this thread here: https://mail.python.org/pipermail/python-ideas/2018-October/054101.html It isn't very long (only about a dozen or so quite short posts) but you probably ought to be at least familiar with it, and if possibly respond to any opposing comments. Thanks, -- Steven From steve at pearwood.info Wed May 15 08:14:48 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 15 May 2019 22:14:48 +1000 Subject: [Python-Dev] Adding a toml module to the standard lib? In-Reply-To: <20190515113516.GP5010@ando.pearwood.info> References: <20190515113516.GP5010@ando.pearwood.info> Message-ID: <20190515121448.GR5010@ando.pearwood.info> On Wed, May 15, 2019 at 09:35:16PM +1000, Steven D'Aprano wrote: > It isn't very long (only about a dozen or so quite short posts) but you > probably ought to be at least familiar with it, and if possibly respond > to any opposing comments. Sorry, to clarify, I don't mean respond on the Python-Ideas thread. I think it is sufficient to keep the current discussion here. -- Steven From vstinner at redhat.com Wed May 15 08:37:57 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 15 May 2019 14:37:57 +0200 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: <4838c6f5-27cc-3823-4ba5-c20a6d0867d9@python.org> References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> <4838c6f5-27cc-3823-4ba5-c20a6d0867d9@python.org> Message-ID: Le mer. 15 mai 2019 ? 11:31, Christian Heimes a ?crit : > What are the next step? Will there be another PEP that explores how we > are going to deal with migration, workflow changes, and how we plan to > map current BPO features to Github? Yes, it's the: PEP 588 -- GitHub Issues Migration Plan https://www.python.org/dev/peps/pep-0588/ Victor -- Night gathers, and now my watch begins. It shall not end until my death. From vstinner at redhat.com Wed May 15 08:39:49 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 15 May 2019 14:39:49 +0200 Subject: [Python-Dev] Adding a toml module to the standard lib? In-Reply-To: References: <20190515113341.19187011@fsol> Message-ID: H Bastian, IMHO we should wait until the format reach version 1.0, since the stdlib has a slow release cycle (one release every 18 months). Too slow for a "fast moving" standard. In the meanwhile, I'm sure setuptools and pip will manage to install a toml parser/generator for their needs, as they already do :-) Victor Le mer. 15 mai 2019 ? 12:50, Bastian Venthur a ?crit : > > On 15.05.19 11:33, Antoine Pitrou wrote: > > How stable is the TOML format? Is it bound to change significantly in > > the coming years? > > > > If the format is stable enough, then I think it's a good idea. > > > The last update to the spec [1] was 10 months ago and added a few > features. The version before that was stable for more than 3 years. It > is also worth noting that he writes about the current version [2]: > > "As of version 0.5.0, TOML should be considered extremely stable. The > goal is for version 1.0.0 to be backwards compatible (as much as humanly > possible) with version 0.5.0. All implementations are strongly > encouraged to become 0.5.0 compatible so that the transition to 1.0.0 > will be simple when that happens." > > That is of course no guarantee, but maybe the best we can hope for. > > > Cheers, > > Bastian > > > [1]: https://github.com/toml-lang/toml/releases > [2]: https://github.com/toml-lang/toml > > > -- > Dr. Bastian Venthur http://venthur.de > Debian Developer venthur at debian org > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com -- Night gathers, and now my watch begins. It shall not end until my death. From paul at ganssle.io Wed May 15 08:57:57 2019 From: paul at ganssle.io (Paul Ganssle) Date: Wed, 15 May 2019 08:57:57 -0400 Subject: [Python-Dev] Adding a toml module to the standard lib? In-Reply-To: References: <20190515113341.19187011@fsol> Message-ID: As someone involved in the packaging side of this, I think while we'd eventually /appreciate/ a TOML parser in the standard library, I agree with Victor that there's no rush, for two reasons: 1. setuptools and pip have a decent number of dependencies that we vendor /anyway/, so vendoring one more is not a big deal. 2. We will be supporting older versions of Python for some time to come, so we'd need to vendor a TOML-parser backport for several years before we could actually use the one in the standard library. I think /if/ a 1.0 version of the spec is going to be forthcoming (not clear from the issue requesting it: https://github.com/toml-lang/toml/issues/515 ), then it's worth waiting at least a bit longer for it. I also think it's probably worth asking the current maintainers of TOML-parsing libraries if /they/ think it's time to adopt/adapt one of their libraries for use in the standard library. They probably have a better perspective on the stability and maturity of their codebases. Best, Paul On 5/15/19 8:39 AM, Victor Stinner wrote: > H Bastian, > > IMHO we should wait until the format reach version 1.0, since the > stdlib has a slow release cycle (one release every 18 months). Too > slow for a "fast moving" standard. > > In the meanwhile, I'm sure setuptools and pip will manage to install a > toml parser/generator for their needs, as they already do :-) > > Victor > > Le mer. 15 mai 2019 ? 12:50, Bastian Venthur a ?crit : >> On 15.05.19 11:33, Antoine Pitrou wrote: >>> How stable is the TOML format? Is it bound to change significantly in >>> the coming years? >>> >>> If the format is stable enough, then I think it's a good idea. >> >> The last update to the spec [1] was 10 months ago and added a few >> features. The version before that was stable for more than 3 years. It >> is also worth noting that he writes about the current version [2]: >> >> "As of version 0.5.0, TOML should be considered extremely stable. The >> goal is for version 1.0.0 to be backwards compatible (as much as humanly >> possible) with version 0.5.0. All implementations are strongly >> encouraged to become 0.5.0 compatible so that the transition to 1.0.0 >> will be simple when that happens." >> >> That is of course no guarantee, but maybe the best we can hope for. >> >> >> Cheers, >> >> Bastian >> >> >> [1]: https://github.com/toml-lang/toml/releases >> [2]: https://github.com/toml-lang/toml >> >> >> -- >> Dr. Bastian Venthur http://venthur.de >> Debian Developer venthur at debian org >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From vstinner at redhat.com Wed May 15 10:56:05 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 15 May 2019 16:56:05 +0200 Subject: [Python-Dev] [python-committers] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: Hi Paul, Le mer. 15 mai 2019 ? 11:40, Paul Moore a ?crit : > Also, is there an archive of the discussions anywhere? The PEP says > discussions happened on Zulip, but I don't follow that and I don't > know where I can find an archived copy of the discussions. Well, the PEP has been discussed a lot at many places since May 2018. The PEP 581 has been (first?) discussed at the Language Summit which was part of Pycon US 2018 (May 2018). Discussion on the PR: https://github.com/python/peps/pull/681/ Multiple threads on Discourse: https://discuss.python.org/t/move-pep-581-discussion/873 https://discuss.python.org/t/pep-581-using-github-issues/535 https://discuss.python.org/t/what-are-next-steps-for-pep-581/864 https://discuss.python.org/t/pep-process-after-pep-8016/558/4 Thread on python-dev: https://mail.python.org/pipermail/python-dev/2019-March/156626.html Threads on python-committers: https://mail.python.org/pipermail/python-committers/2018-May/005428.html https://mail.python.org/pipermail/python-committers/2018-June/005506.html https://mail.python.org/pipermail/python-committers/2018-July/005657.html Discussion on Zulip Chat: https://python.zulipchat.com/#narrow/stream/130206-pep581 The Steering Council discussed it internally as well: https://github.com/python/steering-council/blob/master/updates/2019-04-26_steering-council-update.md The PEP 581 and 588 have been discussed at the Language Summit which was part of Pycon US 2019 (2 weeks ago). Victor -- Night gathers, and now my watch begins. It shall not end until my death. From sebastian.koslowski at gmail.com Wed May 15 02:37:43 2019 From: sebastian.koslowski at gmail.com (Sebastian Koslowski) Date: Wed, 15 May 2019 08:37:43 +0200 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 2nd version In-Reply-To: <46a53765-e3dc-ac8f-bc39-2cc9077b9205@python.org> References: <46a53765-e3dc-ac8f-bc39-2cc9077b9205@python.org> Message-ID: On Tue, May 14, 2019, 19:52 Steve Dower wrote: > It looks like Py_DecodeLocale() is available very early on - why > wouldn't we recommend using this function? It seems to be nearly a > drop-in replacement for mbtowcs in the samples, and if memory allocation > is a big deal perhaps we could just add a version that writes to a buffer? > > Actually, it is recommended in the docs https://docs.python.org/3/c-api/init.html#c.Py_SetPythonHome Sebastian -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed May 15 11:18:15 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 May 2019 16:18:15 +0100 Subject: [Python-Dev] [python-committers] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: On Wed, 15 May 2019 at 15:56, Victor Stinner wrote: > > Hi Paul, > Le mer. 15 mai 2019 ? 11:40, Paul Moore a ?crit : > > Also, is there an archive of the discussions anywhere? The PEP says > > discussions happened on Zulip, but I don't follow that and I don't > > know where I can find an archived copy of the discussions. > > Well, the PEP has been discussed a lot at many places since May 2018. Thanks for all of these. I appreciate the time you took locating them for me. But I do have to say that I still can't really follow any useful "thread of discussion" - it all seems very fragmented, and it's difficult to see the progress towards consensus. Maybe that's just because I'm too used to mailing lists :-) > The PEP 581 has been (first?) discussed at the Language Summit which > was part of Pycon US 2018 (May 2018). Was that written up, or is it all just from people's memories by now? > https://github.com/python/peps/pull/681/ Ah - I don't really follow this sort of PR discussion, as the github emails don't tend to have sufficient context on what's being said, so I (mostly) gave up a long time ago. Also, I tend to assume that discussions on PRs are mostly about details of wording, and substantive changes will be dealt with in a wider forum. I wonder if I should be following PRs on the PEPs repository more closely...? > Multiple threads on Discourse: > > https://discuss.python.org/t/move-pep-581-discussion/873 > https://discuss.python.org/t/pep-581-using-github-issues/535 > https://discuss.python.org/t/what-are-next-steps-for-pep-581/864 > https://discuss.python.org/t/pep-process-after-pep-8016/558/4 > > Thread on python-dev: > > https://mail.python.org/pipermail/python-dev/2019-March/156626.html > > Threads on python-committers: > > https://mail.python.org/pipermail/python-committers/2018-May/005428.html > https://mail.python.org/pipermail/python-committers/2018-June/005506.html > https://mail.python.org/pipermail/python-committers/2018-July/005657.html I saw these, but didn't get much of a sense of progress towards agreement. Again, maybe just because they were lots of fragmented threads and locations. > Discussion on Zulip Chat: > > https://python.zulipchat.com/#narrow/stream/130206-pep581 I can't see this without logging in :-( > The Steering Council discussed it internally as well: > > https://github.com/python/steering-council/blob/master/updates/2019-04-26_steering-council-update.md I did see that, I was more wondering what led to that decision (whether it was a general consensus from the core devs that it was a good move, or mainly the SC's own view that prevailed). > The PEP 581 and 588 have been discussed at the Language Summit which > was part of Pycon US 2019 (2 weeks ago). Again, has there been any write up of that (yet)? As I say, I don't object to the decision, I'm more just trying to better understand the process of being involved under the new regime of the SC, combined with multiple fragmented forums for discussion. It feels a lot harder these days to keep track of all the discussions/decisions going on. But maybe that's a good thing - only people with a genuine interest get involved, and I can spend less of my time reading mailing lists! :-) Paul From brett at python.org Wed May 15 11:57:31 2019 From: brett at python.org (Brett Cannon) Date: Wed, 15 May 2019 08:57:31 -0700 Subject: [Python-Dev] [python-committers] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: On Wed, May 15, 2019 at 8:18 AM Paul Moore wrote: > On Wed, 15 May 2019 at 15:56, Victor Stinner wrote: > > > > Hi Paul, > > Le mer. 15 mai 2019 ? 11:40, Paul Moore a ?crit : > > > Also, is there an archive of the discussions anywhere? The PEP says > > > discussions happened on Zulip, but I don't follow that and I don't > > > know where I can find an archived copy of the discussions. > > > > Well, the PEP has been discussed a lot at many places since May 2018. > > Thanks for all of these. I appreciate the time you took locating them > for me. But I do have to say that I still can't really follow any > useful "thread of discussion" - it all seems very fragmented, and it's > difficult to see the progress towards consensus. Maybe that's just > because I'm too used to mailing lists :-) > > > The PEP 581 has been (first?) discussed at the Language Summit which > > was part of Pycon US 2018 (May 2018). > > Was that written up, or is it all just from people's memories by now? > There's at least https://lwn.net/Articles/754779/. Don't remember if other people wrote up their own summary. [SNIP] > The PEP 581 and 588 have been discussed at the Language Summit which > > was part of Pycon US 2019 (2 weeks ago). > > Again, has there been any write up of that (yet)? > Not yet, but A. Jesse Jiryu Davis attended so the PSF could blog about it. > > As I say, I don't object to the decision, I'm more just trying to > better understand the process of being involved under the new regime > of the SC, I think everyone is. :) In the case of this PEP the various members of the SC have been keeping up with the PEP and its discussions over the year that the PEP has been around, we discussed the pros and cons that people brought up, thought through what will be required for us to do the migration if the PEP was accepted, and in the end decided it was worth the effort. > combined with multiple fragmented forums for discussion. A PEP is being worked on to try and propose a way to straighten this all out, but travel, life, etc. has been keeping that one from being finished. I've added discussing the status of it to our next meeting's agenda. > It > feels a lot harder these days to keep track of all the > discussions/decisions going on. But maybe that's a good thing - only > people with a genuine interest get involved, and I can spend less of > my time reading mailing lists! :-) > Obviously YMMV, but I actually find it easier. :) -Brett > > Paul > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 15 12:17:41 2019 From: brett at python.org (Brett Cannon) Date: Wed, 15 May 2019 09:17:41 -0700 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> <4838c6f5-27cc-3823-4ba5-c20a6d0867d9@python.org> Message-ID: On Wed, May 15, 2019 at 5:38 AM Victor Stinner wrote: > Le mer. 15 mai 2019 ? 11:31, Christian Heimes a > ?crit : > > What are the next step? Will there be another PEP that explores how we > > are going to deal with migration, workflow changes, and how we plan to > > map current BPO features to Github? > > Yes, it's the: > > PEP 588 -- GitHub Issues Migration Plan > https://www.python.org/dev/peps/pep-0588/ > And to be very clear here, PEP 588 is *not* accepted yet, so it's open for changes. I personally would consider the accepting of PEP 581 as a signal that the SC thinks it's worth putting the effort into migrating to GitHub for issues and so now we can focus our efforts as a team in trying to make this result in the workflow that we want. And I suspect everyone knows this, but just in case, I want to personally state that I hope everyone understands that there is no way everyone will be happy with the outcome of this transition and that's okay. :) Workflow is one of those things where people are very often opinionated (just look at all of us and our preference in programming language ;) . Obviously we can all do what we can to be accommodating and come up with a solution that works for as many people as possible (including external folks like triagers and issue reporters), but I hope everyone goes into this being as understanding as possible so we can try to get the best outcome we can. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed May 15 12:51:47 2019 From: steve.dower at python.org (Steve Dower) Date: Wed, 15 May 2019 09:51:47 -0700 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: On 15May2019 0240, Paul Moore wrote: > On Wed, 15 May 2019 at 09:51, Antoine Pitrou wrote: >> >> On Tue, 14 May 2019 18:11:14 -0700 >> Barry Warsaw wrote: >> >>> As the BDFL-Delegate for PEP 581, and with the unanimous backing of the rest of the Steering Council, I hereby Accept this PEP. >> >> For future reference, is it possible to post the Steering Council's >> reflection and rationale on the PEP? > > Also, is there an archive of the discussions anywhere? The PEP says > discussions happened on Zulip, but I don't follow that and I don't > know where I can find an archived copy of the discussions. I think it would be ideal if the PEP itself summarised the major points of these discussions, or at least the links to the discussions (assuming we trust Zulip and Discourse to never lose them). Right now I have a lot of fundamental problems with this PEP as written. But I'm assured that the Steering Council gave it thorough consideration and that the discussions covered all relevant aspects, so I'm not going to create too much of a fuss about it. If they want to move us completely to GitHub, so be it! Cheers, Steve From ezio.melotti at gmail.com Wed May 15 13:44:03 2019 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Wed, 15 May 2019 19:44:03 +0200 Subject: [Python-Dev] [python-committers] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: Hello, On Wed, May 15, 2019 at 5:18 PM Paul Moore wrote: > > On Wed, 15 May 2019 at 15:56, Victor Stinner wrote: > > > > Hi Paul, > > Le mer. 15 mai 2019 ? 11:40, Paul Moore a ?crit : > > > Also, is there an archive of the discussions anywhere? The PEP says > > > discussions happened on Zulip, but I don't follow that and I don't > > > know where I can find an archived copy of the discussions. > > > > Well, the PEP has been discussed a lot at many places since May 2018. > > Thanks for all of these. I appreciate the time you took locating them > for me. But I do have to say that I still can't really follow any > useful "thread of discussion" - it all seems very fragmented, and it's > difficult to see the progress towards consensus. Maybe that's just > because I'm too used to mailing lists :-) > I share the same concerns: 1) the discussion was fragmented between zulip/discuss/github/python-dev/python-committers/sprints/pycons and very difficult to follow, even for interested people (Victor already posted several links but missed a few others); 2) the progress toward consensus was not clear and the approval came somewhat unexpectedly (it was mentioned a couple of weeks ago on https://mail.python.org/pipermail/python-committers/2019-April/006705.html and AFAICT no further discussion took place in public forums since then); In addition: 1) the PEP contains several factual errors. I pointed this out during the core-sprints last year and more recently Berker pointed out some on GitHub: https://github.com/python/peps/pull/1013 ; 2) the "discussions-to" header of the PEP points to the zulip stream. The stream has not been active for 6 months (it got a few new messages today, the previous activity was in Dec 2018); 3) most of the discussions linked by Victor happened last year. Unless I missed some, the only discussions happened this year are the two on Discuss from February (with 3 messages each from a total of 5 authors), and the python-dev thread from March (with 12 messages from 7 authors). One of the two Discuss threads was a inquiry about the process (https://discuss.python.org/t/move-pep-581-discussion/873); 4) Berker is/was writing a competing PEP, in order to address the problems of PEP 581 more effectively since his comments on GitHub haven't been addressed; 5) next week a student is supposed to start working for the PSF on b.p.o and Roundup as part of Google Summer of Code (http://python-gsoc.org/psf_ideas.html); 6) PEP 8016 says "The council should look for ways to use these powers as little as possible. Instead of voting, it's better to seek consensus. Instead of ruling on individual PEPs, it's better to define a standard process for PEP decision making."; To summarize, I feel the approval of this PEP is premature and that consensus was reached in a way that wasn't very transparent, without considering some of the concerns. (This might also be a symptom of a wider problem caused by the fragmentation of the discussions between the old MLs, discuss, zulip, IRC, GitHub PRs and issues, and IRL meetings, but this is a separate topic.) Best Regards, Ezio Melotti > > The PEP 581 has been (first?) discussed at the Language Summit which > > was part of Pycon US 2018 (May 2018). > > Was that written up, or is it all just from people's memories by now? > > > https://github.com/python/peps/pull/681/ > > Ah - I don't really follow this sort of PR discussion, as the github > emails don't tend to have sufficient context on what's being said, so > I (mostly) gave up a long time ago. Also, I tend to assume that > discussions on PRs are mostly about details of wording, and > substantive changes will be dealt with in a wider forum. I wonder if I > should be following PRs on the PEPs repository more closely...? > > > Multiple threads on Discourse: > > > > https://discuss.python.org/t/move-pep-581-discussion/873 > > https://discuss.python.org/t/pep-581-using-github-issues/535 > > https://discuss.python.org/t/what-are-next-steps-for-pep-581/864 > > https://discuss.python.org/t/pep-process-after-pep-8016/558/4 > > > > Thread on python-dev: > > > > https://mail.python.org/pipermail/python-dev/2019-March/156626.html > > > > Threads on python-committers: > > > > https://mail.python.org/pipermail/python-committers/2018-May/005428.html > > https://mail.python.org/pipermail/python-committers/2018-June/005506.html > > https://mail.python.org/pipermail/python-committers/2018-July/005657.html > > I saw these, but didn't get much of a sense of progress towards > agreement. Again, maybe just because they were lots of fragmented > threads and locations. > > > Discussion on Zulip Chat: > > > > https://python.zulipchat.com/#narrow/stream/130206-pep581 > > I can't see this without logging in :-( > > > The Steering Council discussed it internally as well: > > > > https://github.com/python/steering-council/blob/master/updates/2019-04-26_steering-council-update.md > > I did see that, I was more wondering what led to that decision > (whether it was a general consensus from the core devs that it was a > good move, or mainly the SC's own view that prevailed). > > > The PEP 581 and 588 have been discussed at the Language Summit which > > was part of Pycon US 2019 (2 weeks ago). > > Again, has there been any write up of that (yet)? > > As I say, I don't object to the decision, I'm more just trying to > better understand the process of being involved under the new regime > of the SC, combined with multiple fragmented forums for discussion. It > feels a lot harder these days to keep track of all the > discussions/decisions going on. But maybe that's a good thing - only > people with a genuine interest get involved, and I can spend less of > my time reading mailing lists! :-) > > Paul > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > Code of Conduct: https://www.python.org/psf/codeofconduct/ From brett at python.org Wed May 15 13:45:25 2019 From: brett at python.org (Brett Cannon) Date: Wed, 15 May 2019 10:45:25 -0700 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: On Wed, May 15, 2019 at 9:53 AM Steve Dower wrote: > On 15May2019 0240, Paul Moore wrote: > > On Wed, 15 May 2019 at 09:51, Antoine Pitrou > wrote: > >> > >> On Tue, 14 May 2019 18:11:14 -0700 > >> Barry Warsaw wrote: > >> > >>> As the BDFL-Delegate for PEP 581, and with the unanimous backing of > the rest of the Steering Council, I hereby Accept this PEP. > >> > >> For future reference, is it possible to post the Steering Council's > >> reflection and rationale on the PEP? > > > > Also, is there an archive of the discussions anywhere? The PEP says > > discussions happened on Zulip, but I don't follow that and I don't > > know where I can find an archived copy of the discussions. > > I think it would be ideal if the PEP itself summarised the major points > of these discussions, or at least the links to the discussions (assuming > we trust Zulip and Discourse to never lose them). > > Right now I have a lot of fundamental problems with this PEP as written. > But I'm assured that the Steering Council gave it thorough consideration > Yes, sorry if that wasn't communicated better. This was in no way a knee-jerk decision. Beyond all of us following this topic personally since it was first brought up last year, we have also been discussing it nearly every meeting in some form or another since we began discussing PEPs (which would be probably since February). > and that the discussions covered all relevant aspects, so I'm not going > to create too much of a fuss about it. If they want to move us > completely to GitHub, so be it! > I've added to our next meeting's agenda discussing potentially fleshing out PEP 581 a bit more. -Brett > > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed May 15 14:43:00 2019 From: guido at python.org (Guido van Rossum) Date: Wed, 15 May 2019 11:43:00 -0700 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: IMO the text of PEP 581 could use some improvements to capture more of the debate. For example: - If people want to submit PRs to the peps repo that correct *factual* mistakes in PEP 581, they're welcome to (and I will personally see that they will be merged). For example, IIRC you *can* reply to a bpo issue via email, so that bullet should probably be struck. - If someone wants to volunteer a PR to the peps repo that adds a list of objections against the move, e.g. losing certain bpo capabilities, or specific undesirable behaviors or properties of GitHub, that will also be entertained (but be reasonable). But trust me that the SC didn't take this decision lightly. It was unanimous, and we have all thought about this a great deal (and listened to long arguments pro and con). It's also impossible to satisfy everyone -- some people find GitHub unacceptable (I've heard about 1-2 people who refused to create GitHub accounts), but others find bpo unusable (the need to have a bpo account in order to accept the CLA has been brought up repeatedly). A successful migration will take a lot of work, and PEP 588 is where we keep track of tasks to be accomplished. (There are more that haven't made it to that PEP, e.g. the mechanics of the actual move.) We're also hoping to get professional help managing the migration so that not everything relies on volunteers. (Speaking of relying on volunteers, if you haven't seen Russel Keith-Magee's keynote at the most recent PyCon US, please watch it. It is an amazing thing. https://www.youtube.com/watch?v=ftP5BQh1-YM -- start at 21:00 to skip the conference opening remarks.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Wed May 15 15:37:55 2019 From: ethan at stoneleaf.us (Ethan Furman) Date: Wed, 15 May 2019 12:37:55 -0700 Subject: [Python-Dev] deprecation of abstractstaticmethod and abstractclassmethod Message-ID: <51c99452-91d3-a479-2666-1af554c433e2@stoneleaf.us> In issue 11610* abstractclassmethod and abstractstaticmethod were deprecated, apparently because they were redundant with the new technique of calling `classmethod` or `staticmethod` followed by a call to `abstractmethod`. To put it in code: # deprecated class Foo(ABC): @abstractclassmethod def foo_happens(cls): # do some fooey stuff # the new(er) way class Foo(ABC): @classmethod @abstractmethod def foo_happens(cls): # do some fooey stuff I would like to remove the deprecated status of `abstractclassmethod` and `abstractstaticmethod` mainly because: - using the combined decorator is easy to get right (@abstractmethod followed by @classmethod doesn't work) - getting the order wrong can be hard to spot and fix Obviously, decorator order matters for many, if not most, decorators out there -- so why should these two be any different? Because 'abstract', 'class', and 'static' are adjectives -- they're describing the method, rather than changing it**; to use an example, what is the difference between "hot, dry sand" and "dry, hot sand"? The sand is just as dry and just as hot either way. In a debugging session looking at: @abstractmethod @classmethod def some_func(self, this, that, the_other): # many # many ... ... ... # many # lines # of # code Not noticing that the two decorators are in reverse order would be very easy to do. Because order matters here, but cognitively should not, a helper function to make sure it is always done right is a prime candidate to be added to a module -- and, luckily for us, those helper functions already exist! Unfortunately, they are also deprecated, discouraging their use, when we should be encouraging their use. What are the reasons to /not/ remove the deprecation? -- ~Ethan~ * https://bugs.python.org/issue11610 ** I realize that abstractmethod does actually change the function, but that's an implementation detail. From njs at pobox.com Wed May 15 15:48:15 2019 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 15 May 2019 12:48:15 -0700 Subject: [Python-Dev] deprecation of abstractstaticmethod and abstractclassmethod In-Reply-To: <51c99452-91d3-a479-2666-1af554c433e2@stoneleaf.us> References: <51c99452-91d3-a479-2666-1af554c433e2@stoneleaf.us> Message-ID: I don't care about the deprecation either way. But can we fix the individual decorators so both orders work? Even if it requires a special case in the code, it seems worthwhile to remove a subtle user-visible footgun. On Wed, May 15, 2019, 12:39 Ethan Furman wrote: > In issue 11610* abstractclassmethod and abstractstaticmethod were > deprecated, apparently because they were redundant with the new technique > of calling `classmethod` or `staticmethod` followed by a call to > `abstractmethod`. To put it in code: > > # deprecated > > class Foo(ABC): > > @abstractclassmethod > def foo_happens(cls): > # do some fooey stuff > > # the new(er) way > > class Foo(ABC): > > @classmethod > @abstractmethod > def foo_happens(cls): > # do some fooey stuff > > > I would like to remove the deprecated status of `abstractclassmethod` and > `abstractstaticmethod` mainly because: > > - using the combined decorator is easy to get right > (@abstractmethod followed by @classmethod doesn't work) > > - getting the order wrong can be hard to spot and fix > > Obviously, decorator order matters for many, if not most, decorators out > there -- so why should these two be any different? Because 'abstract', > 'class', and 'static' are adjectives -- they're describing the method, > rather than changing it**; to use an example, what is the difference > between "hot, dry sand" and "dry, hot sand"? The sand is just as dry and > just as hot either way. In a debugging session looking at: > > @abstractmethod > @classmethod > def some_func(self, this, that, the_other): > # many > # many > ... > ... > ... > # many > # lines > # of > # code > > Not noticing that the two decorators are in reverse order would be very > easy to do. > > Because order matters here, but cognitively should not, a helper function > to make sure it is always done right is a prime candidate to be added to a > module -- and, luckily for us, those helper functions already exist! > Unfortunately, they are also deprecated, discouraging their use, when we > should be encouraging their use. > > What are the reasons to /not/ remove the deprecation? > > -- > ~Ethan~ > > > > * https://bugs.python.org/issue11610 > > ** I realize that abstractmethod does actually change the function, but > that's an implementation detail. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/njs%40pobox.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed May 15 16:00:16 2019 From: steve.dower at python.org (Steve Dower) Date: Wed, 15 May 2019 13:00:16 -0700 Subject: [Python-Dev] deprecation of abstractstaticmethod and abstractclassmethod In-Reply-To: References: <51c99452-91d3-a479-2666-1af554c433e2@stoneleaf.us> Message-ID: <571659b9-d63e-4bf9-ba06-56e7555b9aa8@python.org> On 15May2019 1248, Nathaniel Smith wrote: > I don't care about the deprecation either way. But can we fix the > individual decorators so both orders work? Even if it requires a special > case in the code, it seems worthwhile to remove a subtle user-visible > footgun. The classmethod and staticmethod objects already have a getter for __isabstractmethod__ that forwards to the underlying object, so they should just need a setter as well, right? I also have no strong opinion about deprecation. But fixing this seems pretty straightforward and may be a good task for someone's first C contribution? Cheers, Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Wed May 15 16:09:30 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 16 May 2019 05:09:30 +0900 Subject: [Python-Dev] Consolidating channel of record [was: 581 (Using GitHub issues for CPython) is accepted] In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: <23772.29178.404867.929820@turnbull.sk.tsukuba.ac.jp> Stripping the list of addressees, since I'm pretty sure all the people who will *make* the decision read Python-Dev regularly, except perhaps Carol. Paul Moore writes: > Thanks for all of these. I appreciate the time you took locating them > for me. But I do have to say that I still can't really follow any > useful "thread of discussion" - it all seems very fragmented, and it's > difficult to see the progress towards consensus. Maybe that's just > because I'm too used to mailing lists :-) Please, let's not start by privileging any particular type of channel in this discussion. I know what I like, but it's far more important to have a single place to refer to past discussion IMO. It's bad enough with python-ideas and python-dev. Maybe this fragmentation is OK in the long run, but at least while the Steering Council is shaking down (say, until release of 3.9?), the Council should consider anointing two archived "channels of record", one for private deliberations of the Council itself (for the sake of future members), and one for PEP discussions. Of course if the SC chooses Discourse for the PEP channel, people *will* discuss PEPs on Python-Dev and IRL. The point is "no fair pointing people to *other* channels for reference". If you want to make a public argument, make it in the proper place. Everything else is effectively private. If you want to refer to that in the public discussion, read it into the public record. The stricture for the Council deliberation channel is different, since I expect the archives would be private to Council members: if you came into this discussion in the middle, what conversations would you want to be able to review? While I'm here, is there a place where general Pythonistas can bring matters to the attention of the Council? Steve From chris.barker at noaa.gov Wed May 15 16:32:05 2019 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Wed, 15 May 2019 13:32:05 -0700 Subject: [Python-Dev] Consolidating channel of record [was: 581 (Using GitHub issues for CPython) is accepted] In-Reply-To: <23772.29178.404867.929820@turnbull.sk.tsukuba.ac.jp> References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> <23772.29178.404867.929820@turnbull.sk.tsukuba.ac.jp> Message-ID: Frankly, multiple long meandering threads in s single mailing list are not s very good archive either. Ideally, the PEP is updated with a summary of the issues discussed as the discussion unfolds. (And links to the main discussion threads?) It founds like that didn?t happen in this case, but it?s not necessarily too late. As the community seems to be moving to a wider variety of fora, this will become all the more critical. - CHB > On May 15, 2019, at 1:09 PM, Stephen J. Turnbull wrote: > > Stripping the list of addressees, since I'm pretty sure all the > people who will *make* the decision read Python-Dev regularly, except > perhaps Carol. > > Paul Moore writes: > >> Thanks for all of these. I appreciate the time you took locating them >> for me. But I do have to say that I still can't really follow any >> useful "thread of discussion" - it all seems very fragmented, and it's >> difficult to see the progress towards consensus. Maybe that's just >> because I'm too used to mailing lists :-) > > Please, let's not start by privileging any particular type of channel > in this discussion. I know what I like, but it's far more important > to have a single place to refer to past discussion IMO. It's bad > enough with python-ideas and python-dev. > > Maybe this fragmentation is OK in the long run, but at least while the > Steering Council is shaking down (say, until release of 3.9?), the > Council should consider anointing two archived "channels of record", > one for private deliberations of the Council itself (for the sake of > future members), and one for PEP discussions. > > Of course if the SC chooses Discourse for the PEP channel, people > *will* discuss PEPs on Python-Dev and IRL. The point is "no fair > pointing people to *other* channels for reference". If you want to > make a public argument, make it in the proper place. Everything else > is effectively private. If you want to refer to that in the public > discussion, read it into the public record. > > The stricture for the Council deliberation channel is different, since > I expect the archives would be private to Council members: if you came > into this discussion in the middle, what conversations would you want > to be able to review? > > While I'm here, is there a place where general Pythonistas can bring > matters to the attention of the Council? > > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov From vstinner at redhat.com Wed May 15 16:41:32 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 15 May 2019 22:41:32 +0200 Subject: [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: Le mer. 15 mai 2019 ? 20:46, Guido van Rossum a ?crit : > But trust me that the SC didn't take this decision lightly. It was unanimous, and we have all thought about this a great deal (and listened to long arguments pro and con). It's also impossible to satisfy everyone -- some people find GitHub unacceptable (I've heard about 1-2 people who refused to create GitHub accounts), but others find bpo unusable (the need to have a bpo account in order to accept the CLA has been brought up repeatedly). Around me, many people told me that they were awaiting for the PEP 581 to be accepted,. They are now *very kind* that Python issues are moving to GitHub. Honestly, I don't understand why people are excited by bug tracker, for me it's not the most exciting part of development :-) But it seems like many people were unhappy with Roundup. At least, I can say that I'm not a big fan of it's UI (maybe too complex, too many fields) compared to GitHub. I can also understand that it's annoying for contributor to have to create 2 separated accounts (bugs.python.org and GitHub) and link them *carefully* (mistakes are common, you can see that when a contributor waits for the "CLA signed" label). Sometimes, they are some glitches with bugs.python.org <=> GitHub integration: some notifications take longer 30 sec or simply never happen. I understand that GitHub is better to *report* bugs (and maybe to comment an issue), but maybe worse for bug triage, and that we want to make sure that it's easier to contribute to Python for "regular contributors". I also know that GitHub isn't perfect and that it will take time to build a new efficient workflow on top of GitHub. > A successful migration will take a lot of work, and PEP 588 is where we keep track of tasks to be accomplished. (There are more that haven't made it to that PEP, e.g. the mechanics of the actual move.) We're also hoping to get professional help managing the migration so that not everything relies on volunteers. That would be great! The migration of the code took something like 2 years and it was a little bit painful for everybody at the beginning. Having someone paid for that would make the migration shorter and maybe also smoother. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From brett at python.org Wed May 15 17:13:38 2019 From: brett at python.org (Brett Cannon) Date: Wed, 15 May 2019 14:13:38 -0700 Subject: [Python-Dev] Consolidating channel of record [was: 581 (Using GitHub issues for CPython) is accepted] In-Reply-To: <23772.29178.404867.929820@turnbull.sk.tsukuba.ac.jp> References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> <23772.29178.404867.929820@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, May 15, 2019 at 1:10 PM Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: [SNIP] > > While I'm here, is there a place where general Pythonistas can bring > matters to the attention of the Council? > https://github.com/python/steering-council is where we're asking people to ask stuff so it's recorded in a central location that's publicly visible. -Brett > > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mariatta at python.org Wed May 15 18:06:54 2019 From: mariatta at python.org (Mariatta) Date: Wed, 15 May 2019 15:06:54 -0700 Subject: [Python-Dev] Python Language Summit 2019 blog posts Message-ID: If you have FOMO (fear of missing out) of Python Language Summit 2019, worry no more. We invited A. Jesse Jiryu Davis to cover for the language summit, and the blog posts are starting to appear in The PSF's official blog. Starts here: http://pyfound.blogspot.com/2019/05/the-2019-python-language-summit.html More links will be added as they become available. Currently available posts: - Russell Keith-Magee: Python On Other Platforms https://pyfound.blogspot.com/2019/05/russell-keith-magee-python-on-other.html - Paul Ganssle: Time Zones In The Standard Library https://pyfound.blogspot.com/2019/05/paul-ganssle-time-zones-in-standard.html Thank you Jesse for writing up these posts, and thanks to all language summit presenters and attendees. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed May 15 18:07:19 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 15 May 2019 23:07:19 +0100 Subject: [Python-Dev] Consolidating channel of record [was: 581 (Using GitHub issues for CPython) is accepted] In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> <23772.29178.404867.929820@turnbull.sk.tsukuba.ac.jp> Message-ID: On Wed, 15 May 2019 at 21:34, Chris Barker - NOAA Federal via Python-Dev wrote: > > Frankly, multiple long meandering threads in s single mailing list are > not s very good archive either. They aren't, I agree. But in my view, they constitute the *source material* that the summaries in the PEP should be based on. No-one should need to read the archives if they just want to know the points being made. But for someone who wonders how a particular conclusion was reached, having a single archive of record (to use Stephen's phrase) makes sense - certainly people *can* search multiple sources, but we should be trying to be transparent, and not making it too hard to research the background of a decision is part of that. > Ideally, the PEP is updated with a summary of the issues discussed as > the discussion unfolds. I'm not sure that's just "ideally" - one of the key purposes of a PEP should be to summarise the discussions. > (And links to the main discussion threads?) That's nice to have, yes. > It founds like that didn?t happen in this case, but it?s not > necessarily too late. Definitely. In fact, I doubt there's much of any real interest that needs to be referenced (workflow debates are not the most interesting reading, in my experience :-)) > As the community seems to be moving to a wider variety of fora, this > will become all the more critical. I completely agree here. > On May 15, 2019, at 1:09 PM, Stephen J. Turnbull wrote: > > Please, let's not start by privileging any particular type of channel > > in this discussion. I know what I like, but it's far more important > > to have a single place to refer to past discussion IMO. It's bad > > enough with python-ideas and python-dev. 100% agree. This shouldn't be about whether any particular channel is "better" or "worse" than any other. It's just about not having the original content of the discussions that feed into an accepted PEP be scattered too widely. > > The stricture for the Council deliberation channel is different, since > > I expect the archives would be private to Council members: if you came > > into this discussion in the middle, what conversations would you want > > to be able to review? In theory, PEP 13 says that the council should look for consensus rather than making decisions on their own. Ideally, the council's private deliberations should be of little interest to anyone else, because either (a) the consensus should be clearly visible from the public record of the debate, or (b) the public record should show that the discussion was getting nowhere, and the council had to make the final decision (at which point, the community has effectively given up the right to argue about why the council chose what they did). The reason the decision on PEP 581 started me thinking about this was precisely because I hadn't seen any signs of that consensus or of a stalled discussion. Maybe I wasn't looking in the right places, but I *thought* I was following the majority of PEP-related discussions (at least skimming them). So I was somewhat startled to see a formal decision come with what felt like no real warning. Paul From tjreedy at udel.edu Wed May 15 18:32:39 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 15 May 2019 18:32:39 -0400 Subject: [Python-Dev] Python Language Summit 2019 blog posts In-Reply-To: References: Message-ID: On 5/15/2019 6:06 PM, Mariatta wrote: > If you have FOMO (fear of missing out) of Python Language Summit 2019, > worry no more. > > We invited A. Jesse Jiryu Davis to cover for the language summit, and > the blog posts are starting to appear in The PSF's official blog. > > Starts here: > http://pyfound.blogspot.com/2019/05/the-2019-python-language-summit.html > > More links will be added as they become available. > > Currently available posts: > > -?Russell Keith-Magee: Python On Other Platforms > https://pyfound.blogspot.com/2019/05/russell-keith-magee-python-on-other.html > > -?Paul Ganssle: Time Zones In The Standard Library > https://pyfound.blogspot.com/2019/05/paul-ganssle-time-zones-in-standard.html > > Thank you Jesse for writing up these posts, and thanks to all language > summit presenters and attendees. These appear on the Python Planet list, mirrored at news.gmane.org as gwene.org.python.planet -- Terry Jan Reedy From vstinner at redhat.com Wed May 15 19:10:46 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 01:10:46 +0200 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version Message-ID: Hi, Thanks to the constructive discussions, I enhanced my PEP 587. I don't plan any further change, the PEP is now ready to review (and maybe even for pronouncement, hi Thomas! :-)). The Rationale now better explains all challenges and the complexity of the Python Initialization Configuration. The "Isolate Python" section is a short guide explaining how configure Python to embed it into an application. The "Path Configuration" section elaborates the most interesting part of the configuration: configure where Python looks for modules (sys.path). I added PyWideStringList_Insert() to allow to prepend a path in module_search_paths. The "Python Issues" section give a long list of issues solved directly or indirectly by this PEP. I'm open for bikeshedding on PyConfig fields names and added functions names ;-) I hesitate for "use_module_search_paths": maybe "module_search_paths_set" is a better name, as in "is module_search_paths set?". The purpose of this field is to allow to have an empty sys.path (ask PyConfig_Read() to not override it). IMHO an empty sys.path makes sense for some specific use cases, like executing Pyhon code without any external module. My PEP 587 proposes better names: Py_FrozenFlag becomes PyConfig.pathconfig_warnings and Py_DebugFlag becomes PyConfig.parser_debug. I also avoided double negation. For example, Py_DontWriteBytecodeFlag becomes write_bytecode. Changes between version 3 and version 2: * PyConfig: Add configure_c_stdio and parse_argv; rename _frozen to pathconfig_warnings. * Rename functions using bytes strings and wide strings. For example, Py_PreInitializeFromWideArgs() becomes Py_PreInitializeFromArgs(), and PyConfig_SetArgv() becomes PyConfig_SetBytesArgv(). * Add PyWideStringList_Insert() function. * New "Path configuration", "Isolate Python", "Python Issues" and "Version History" sections. * PyConfig_SetString() and PyConfig_SetBytesString() now requires the configuration as the first argument. * Rename Py_UnixMain() to Py_BytesMain() HTML version: https://www.python.org/dev/peps/pep-0587/ Full PEP text below. I know that the PEP is long, but well, it's a complex topic, and I chose to add many examples to make the API easier to understand. Victor --- PEP: 587 Title: Python Initialization Configuration Author: Victor Stinner , Nick Coghlan BDFL-Delegate: Thomas Wouters Discussions-To: python-dev at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Mar-2019 Python-Version: 3.8 Abstract ======== Add a new C API to configure the Python Initialization providing finer control on the whole configuration and better error reporting. It becomes possible to read the configuration and modify it before it is applied. It also becomes possible to completely override how Python computes the module search paths (``sys.path``). Building a customized Python which behaves as regular Python becomes easier using the new ``Py_RunMain()`` function. Moreover, command line arguments passed to ``PyConfig.argv`` are now parsed as the regular Python parses command line options, and ``PyConfig.xoptions`` are handled as ``-X opt`` command line options. This extracts a subset of the API design from the PEP 432 development and refactoring work that is now considered sufficiently stable to make public (allowing 3rd party embedding applications access to the same configuration APIs that the native CPython CLI is now using). Rationale ========= Python is highly configurable but its configuration evolved organically. The initialization configuration is scattered all around the code using different ways to set them: global configuration variables (ex: ``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``), command line arguments (ex: ``-b``), configuration files (ex: ``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A straightforward and reliable way to configure Python is needed. Some configuration parameters are not accessible from the C API, or not easily. For example, there is no API to override the default values of ``sys.executable``. Some options like ``PYTHONPATH`` can only be set using an environment variable which has a side effect on Python child processes. Some options also depends on other options: see `Priority and Rules`_. Python 3.7 API does not provide a consistent view of the overall configuration. The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as input whereas the Python filesystem encoding is set during the initialization which can lead to mojibake. Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory allocation failure which is not convenient when Python is embedded. Moreover, ``Py_Main()`` could exit directly the process rather than returning an exit code. Proposed new API reports the error or exit code to the caller which can decide how to handle it. Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly was almost impossible in Python 3.6. The code base has been deeply reworked in Python 3.7 and then in Python 3.8 to read the configuration into a structure with no side effect. It becomes possible to clear the configuration (release memory) and read again the configuration if the encoding changed . It is required to implement properly the UTF-8 which changes the encoding using ``-X utf8`` command line option. Internally, bytes ``argv`` strings are decoded from the filesystem encoding. The ``-X dev`` changes the memory allocator (behaves as ``PYTHONMALLOC=debug``), whereas it was not possible to change the memory allocation *while* parsing the command line arguments. The new design of the internal implementation not only allowed to implement properly ``-X utf8`` and ``-X dev``, it also allows to change the Python behavior way more easily, especially for corner cases like that, and ensure that the configuration remains consistent: see `Priority and Rules`_. This PEP is a partial implementation of PEP 432 which is the overall design. New fields can be added later to ``PyConfig`` structure to finish the implementation of the PEP 432 (e.g. by adding a new partial initialization API which allows to configure Python using Python objects to finish the full initialization). However, those features are omitted from this PEP as even the native CPython CLI doesn't work that way - the public API proposal in this PEP is limited to features which have already been implemented and adopted as private APIs for us in the native CPython CLI. Python Initialization C API =========================== This PEP proposes to add the following new structures, functions and macros. New structures (4): * ``PyConfig`` * ``PyInitError`` * ``PyPreConfig`` * ``PyWideStringList`` New functions (17): * ``Py_PreInitialize(config)`` * ``Py_PreInitializeFromBytesArgs(config, argc, argv)`` * ``Py_PreInitializeFromArgs(config, argc, argv)`` * ``PyWideStringList_Append(list, item)`` * ``PyWideStringList_Insert(list, index, item)`` * ``PyConfig_SetString(config,config_str, str)`` * ``PyConfig_SetBytesString(config, config_str, str)`` * ``PyConfig_SetBytesArgv(config, argc, argv)`` * ``PyConfig_SetArgv(config, argc, argv)`` * ``PyConfig_Read(config)`` * ``PyConfig_Clear(config)`` * ``Py_InitializeFromConfig(config)`` * ``Py_InitializeFromBytesArgs(config, argc, argv)`` * ``Py_InitializeFromArgs(config, argc, argv)`` * ``Py_BytesMain(argc, argv)`` * ``Py_RunMain()`` * ``Py_ExitInitError(err)`` New macros (9): * ``PyPreConfig_INIT`` * ``PyConfig_INIT`` * ``Py_INIT_OK()`` * ``Py_INIT_ERR(MSG)`` * ``Py_INIT_NO_MEMORY()`` * ``Py_INIT_EXIT(EXITCODE)`` * ``Py_INIT_IS_ERROR(err)`` * ``Py_INIT_IS_EXIT(err)`` * ``Py_INIT_FAILED(err)`` This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type) and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these internal structures. ``PyInterpreterState.config`` becomes the new reference configuration, replacing global configuration variables and other private variables. PyWideStringList ---------------- ``PyWideStringList`` is a list of ``wchar_t*`` strings. Example to initialize a string from C static array:: static wchar_t* argv[2] = { L"-c", L"pass", }; PyWideStringList config_argv = PyWideStringList_INIT; config_argv.length = Py_ARRAY_LENGTH(argv); config_argv.items = argv; ``PyWideStringList`` structure fields: * ``length`` (``Py_ssize_t``) * ``items`` (``wchar_t**``) Methods: * ``PyInitError PyWideStringList_Append(PyWideStringList *list, const wchar_t *item)``: Append *item* to *list*. * ``PyInitError PyWideStringList_Insert(PyWideStringList *list, Py_ssize_t index, const wchar_t *item)``: Insert *item* into *list* at *index*. If *index* is greater than *list* length, just append *item* to *list*. If *length* is non-zero, *items* must be non-NULL and all strings must be non-NULL. PyInitError ----------- ``PyInitError`` is a structure to store an error message or an exit code for the Python Initialization. For an error, it stores the C function name which created the error. Example:: PyInitError alloc(void **ptr, size_t size) { *ptr = PyMem_RawMalloc(size); if (*ptr == NULL) { return Py_INIT_NO_MEMORY(); } return Py_INIT_OK(); } int main(int argc, char **argv) { void *ptr; PyInitError err = alloc(&ptr, 16); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } PyMem_Free(ptr); return 0; } ``PyInitError`` fields: * ``exitcode`` (``int``): argument passed to ``exit()``, only set by ``Py_INIT_EXIT()``. * ``err_msg`` (``const char*``): error message * private ``_func`` field: used by ``Py_INIT_ERR()`` to store the C function name which created the error. * private ``_type`` field: for internal usage only. Macro to create an error: * ``Py_INIT_OK()``: Success. * ``Py_INIT_ERR(err_msg)``: Initialization error with a message. * ``Py_INIT_NO_MEMORY()``: Memory allocation failure (out of memory). * ``Py_INIT_EXIT(exitcode)``: Exit Python with the specified exit code. Other macros and functions: * ``Py_INIT_IS_ERROR(err)``: Is the result an error? * ``Py_INIT_IS_EXIT(err)``: Is the result an exit? * ``Py_INIT_FAILED(err)``: Is the result an error or an exit? Similar to ``Py_INIT_IS_ERROR(err) || Py_INIT_IS_EXIT(err)``. * ``Py_ExitInitError(err)``: Call ``exit(exitcode)`` on Unix or ``ExitProcess(exitcode)`` if the result is an exit, call ``Py_FatalError(err_msg)`` if the result is an error. Must not be called if the result is a success. Pre-Initialization with PyPreConfig ----------------------------------- ``PyPreConfig`` structure is used to pre-initialize Python: * Set the memory allocator * Configure the LC_CTYPE locale * Set the UTF-8 mode Example using the pre-initialization to enable the UTF-8 Mode:: PyPreConfig preconfig = PyPreConfig_INIT; preconfig.utf8_mode = 1; PyInitError err = Py_PreInitialize(&preconfig); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* at this point, Python will speak UTF-8 */ Py_Initialize(); /* ... use Python API here ... */ Py_Finalize(); Functions to pre-initialize Python: * ``PyInitError Py_PreInitialize(const PyPreConfig *config)`` * ``PyInitError Py_PreInitializeFromBytesArgs(const PyPreConfig *config, int argc, char **argv)`` * ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *config, int argc, wchar_t **argv)`` These functions can be called with *config* set to ``NULL``. If Python is initialized with command line arguments, the command line arguments must also be passed to pre-initialize Python, since they have an effect on the pre-configuration like encodings. For example, the ``-X utf8`` command line option enables the UTF-8 Mode. These functions can be called with *config* set to ``NULL``. The caller is responsible to handle error or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. ``PyPreConfig`` fields: * ``allocator`` (``char*``, default: ``NULL``): Name of the memory allocator (ex: ``"malloc"``). * ``coerce_c_locale`` (``int``, default: 0): If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE locale to decide if it should be coerced. * ``coerce_c_locale_warn`` (``int``, default: 0): If non-zero, emit a warning if the C locale is coerced. * ``dev_mode`` (``int``, default: 0): See ``PyConfig.dev_mode``. * ``isolated`` (``int``, default: 0): See ``PyConfig.isolated``. * ``legacy_windows_fs_encoding`` (``int``, Windows only, default: 0): If non-zero, disable UTF-8 Mode, set the Python filesystem encoding to ``mbcs``, set the filesystem error handler to ``replace``. * ``use_environment`` (``int``, default: 1): See ``PyConfig.use_environment``. * ``utf8_mode`` (``int``, default: 0): If non-zero, enable the UTF-8 mode. ``PyPreConfig`` private field, for internal use only: * ``_config_version`` (``int``, default: config version): Configuration version, used for ABI compatibility. The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``, ``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python enable them depending on the user configuration. In this case, it's safer to explicitly pre-initialize Python to ensure that encodings are configured before the Python initialization starts. Example to get the same encoding than regular Python:: PyPreConfig preconfig = PyPreConfig_INIT; preconfig.coerce_c_locale = -1; preconfig.coerce_c_locale_warn = -1; preconfig.utf8_mode = -1; PyInitError err = Py_PreInitialize(&preconfig); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } Initialization with PyConfig ---------------------------- The ``PyConfig`` structure contains all parameters to configure Python. Example setting the program name:: PyInitError err; PyConfig config = PyConfig_INIT; err = PyConfig_SetString(&config.program_name, L"my_program"); if (_Py_INIT_FAILED(err)) { Py_ExitInitError(err); } err = Py_InitializeFromConfig(&config); PyConfig_Clear(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } ``PyConfig`` methods: * ``PyInitError PyConfig_SetString(PyConfig *config, wchar_t **config_str, const wchar_t *str)``: Copy the wide character string *str* into ``*config_str``. * ``PyInitError PyConfig_SetBytesString(PyConfig *config, wchar_t **config_str, const char *str)``: Decode *str* using ``Py_DecodeLocale()`` and set the result into ``*config_str``. Pre-initialize Python if needed to ensure that encodings are properly configured. * ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, wchar_t **argv)``: Set command line arguments from wide character strings. * ``PyInitError PyConfig_SetBytesArgv(PyConfig *config, int argc, char **argv)``: Set command line arguments: decode bytes using ``Py_DecodeLocale()``. Pre-initialize Python if needed to ensure that encodings are properly configured. * ``PyInitError PyConfig_Read(PyConfig *config)``: Read all Python configuration. Fields which are already set are left unchanged. * ``void PyConfig_Clear(PyConfig *config)``: Release configuration memory. Functions to initialize Python: * ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``: Initialize Python from *config* configuration. *config* can be ``NULL``. The caller of these methods and functions is responsible to handle failure or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. ``PyConfig`` fields: * ``argv`` (``PyWideStringList``, default: empty): Command line arguments, ``sys.argv``. It is parsed and updated by default, set ``parse_argv`` to 0 to avoid that. * ``base_exec_prefix`` (``wchar_t*``, default: ``NULL``): ``sys.base_exec_prefix``. * ``base_prefix`` (``wchar_t*``, default: ``NULL``): ``sys.base_prefix``. * ``buffered_stdio`` (``int``, default: 1): If equals to 0, enable unbuffered mode, make stdout and stderr streams to be unbuffered. * ``bytes_warning`` (``int``, default: 0): If equals to 1, issue a warning when comparing ``bytes`` or ``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If equal or greater to 2, raise a ``BytesWarning`` exception. * ``check_hash_pycs_mode`` (``wchar_t*``, default: ``"default"``): ``--check-hash-based-pycs`` command line option value (see PEP 552). * ``configure_c_stdio`` (``int``, default: 1): If non-zero, configure C standard streams (``stdio``, ``stdout``, ``stdout``). For example, set their mode to ``O_BINARY`` on Windows. * ``dev_mode`` (``int``, default: 0): Development mode * ``dll_path`` (``wchar_t*``, Windows only, default: ``NULL``): Windows DLL path. * ``dump_refs`` (``int``, default: 0): If non-zero, dump all objects which are still alive at exit * ``exec_prefix`` (``wchar_t*``, default: ``NULL``): ``sys.exec_prefix``. * ``executable`` (``wchar_t*``, default: ``NULL``): ``sys.executable``. * ``faulthandler`` (``int``, default: 0): If non-zero, call ``faulthandler.enable()``. * ``filesystem_encoding`` (``wchar_t*``, default: ``NULL``): Filesystem encoding, ``sys.getfilesystemencoding()``. * ``filesystem_errors`` (``wchar_t*``, default: ``NULL``): Filesystem encoding errors, ``sys.getfilesystemencodeerrors()``. * ``use_hash_seed`` (``int``, default: 0), ``hash_seed`` (``unsigned long``, default: 0): Randomized hash function seed. * ``home`` (``wchar_t*``, default: ``NULL``): Python home directory. * ``import_time`` (``int``, default: 0): If non-zero, profile import time. * ``inspect`` (``int``, default: 0): Enter interactive mode after executing a script or a command. * ``install_signal_handlers`` (``int``, default: 1): Install signal handlers? * ``interactive`` (``int``, default: 0): Interactive mode. * ``legacy_windows_stdio`` (``int``, Windows only, default: 0): If non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. * ``malloc_stats`` (``int``, default: 0): If non-zero, dump memory allocation statistics at exit. * ``module_search_path_env`` (``wchar_t*``, default: ``NULL``): ``PYTHONPATH`` environment variale value. * ``use_module_search_paths`` (``int``, default: 0), ``module_search_paths`` (``PyWideStringList``, default: empty): ``sys.path``. * ``optimization_level`` (``int``, default: 0): Compilation optimization level. * ``parse_argv`` (``int``, default: 1): If non-zero, parse ``argv`` command line arguments and update ``argv``. * ``parser_debug`` (``int``, default: 0): If non-zero, turn on parser debugging output (for expert only, depending on compilation options). * ``pathconfig_warnings`` (``int``, default: 1): If equal to 0, suppress warnings when computing the path configuration. * ``prefix`` (``wchar_t*``, default: ``NULL``): ``sys.prefix``. * ``program_name`` (``wchar_t*``, default: ``NULL``): Program name. * ``program`` (``wchar_t*``, default: ``NULL``): ``argv[0]`` or an empty string. * ``pycache_prefix`` (``wchar_t*``, default: ``NULL``): ``.pyc`` cache prefix. * ``quiet`` (``int``, default: 0): Quiet mode. For example, don't display the copyright and version messages even in interactive mode. * ``run_command`` (``wchar_t*``, default: ``NULL``): ``-c COMMAND`` argument. * ``run_filename`` (``wchar_t*``), default: ``NULL``: ``python3 SCRIPT`` argument. * ``run_module`` (``wchar_t*``, default: ``NULL``): ``python3 -m MODULE`` argument. * ``show_alloc_count`` (``int``, default: 0): Show allocation counts at exit? * ``show_ref_count`` (``int``, default: 0): Show total reference count at exit? * ``site_import`` (``int``, default: 1): Import the ``site`` module at startup? * ``skip_source_first_line`` (``int``, default: 0): Skip the first line of the source? * ``stdio_encoding`` (``wchar_t*``, default: ``NULL``), ``stdio_errors`` (``wchar_t*``, default: ``NULL``): Encoding and encoding errors of ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. * ``tracemalloc`` (``int``, default: 0): If non-zero, call ``tracemalloc.start(value)``. * ``user_site_directory`` (``int``, default: 1): If non-zero, add user site directory to ``sys.path``. * ``verbose`` (``int``, default: 0): If non-zero, enable verbose mode. * ``warnoptions`` (``PyWideStringList``, default: empty): Options of the ``warnings`` module to build warnings filters. * ``write_bytecode`` (``int``, default: 1): If non-zero, write ``.pyc`` files. * ``xoptions`` (``PyWideStringList``, default: empty): ``sys._xoptions``. ``PyConfig`` private fields, for internal use only: * ``_config_version`` (``int``, default: config version): Configuration version, used for ABI compatibility. * ``_install_importlib`` (``int``, default: 1): Install importlib? * ``_init_main`` (``int``, default: 1): If equal to 0, stop Python initialization before the "main" phase (see PEP 432). By default, the ``argv`` arguments are parsed as regular Python command line arguments and ``argv`` is updated to strip parsed Python arguments: see `Command Line Arguments`_. Set ``parse_argv`` to 0 to avoid parsing and updating ``argv``. If ``argv`` is empty, an empty string is added to ensure that ``sys.argv`` always exists and is never empty. The ``xoptions`` options are parsed to set other options: see `-X Options`_. More complete example modifying the configuration before calling ``PyConfig_Read()``, and then modify the read configuration:: PyInitError init_python(const char *program_name) { PyInitError err; PyConfig config = PyConfig_INIT; /* Set the program name before reading the configuraton (decode byte string from the locale encoding) */ err = PyConfig_SetBytesString(&config.program_name, program_name); if (_Py_INIT_FAILED(err)) { goto fail; } /* Read all configuration at once */ err = PyConfig_Read(&config); if (_Py_INIT_FAILED(err)) { goto fail; } /* Append our custom search path to sys.path */ err = PyWideStringList_Append(&config.module_search_paths, L"/path/to/more/modules"); if (_Py_INIT_FAILED(err)) { goto fail; } /* Override executable computed by PyConfig_Read() */ err = PyConfig_SetString(&config, &config.executable, L"my_executable"); if (_Py_INIT_FAILED(err)) { goto fail; } err = Py_InitializeFromConfig(&config); /* Py_InitializeFromConfig() copied config which must now be cleared to release memory */ PyConfig_Clear(&config); return err; fail: PyConfig_Clear(&config); Py_ExitInitError(err); } .. note:: ``PyConfig`` does not have any field for extra inittab functions: ``PyImport_AppendInittab()`` and ``PyImport_ExtendInittab()`` functions are still relevant (and can be called before Python initialization). Initialization with constant PyConfig ------------------------------------- When no ``PyConfig`` method is used but only ``Py_InitializeFromConfig()``, the caller is responsible for managing ``PyConfig`` memory. In that case, constant strings and constant string lists can be used to avoid dynamically allocated memory. It can be used for most simple configurations. Example of Python initialization enabling the isolated mode:: PyConfig config = PyConfig_INIT; config.isolated = 1; PyInitError err = Py_InitializeFromConfig(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* ... use Python API here ... */ Py_Finalize(); ``PyConfig_Clear()`` is not needed in this example since ``config`` does not contain any dynamically allocated string: ``Py_InitializeFromConfig`` is responsible to fill other fields and manage the memory. For convenience, two other functions are provided for constant ``PyConfig``: * ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int argc, wchar_t **argv)`` * ``PyInitError Py_InitializeFromBytesArgs(const PyConfig *config, int argc, char **argv)`` They be called with *config* set to ``NULL``. The caller of these functions is responsible to handle failure or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. Path Configuration ------------------ ``PyConfig`` contains multiple fields for the path configuration: * Path configuration input fields: * ``home`` * ``module_search_path_env`` * ``pathconfig_warnings`` * Path configuration output fields: * ``dll_path`` (Windows only) * ``exec_prefix`` * ``executable`` * ``prefix`` * ``use_module_search_paths``, ``module_search_paths`` Set ``pathconfig_warnings`` to 0 to suppress warnings when computing the path configuration. It is possible to completely ignore the function computing the default path configuration by setting explicitly all path configuration output fields listed above. A string is considered as set even if it's an empty string. ``module_search_paths`` is considered as set if ``use_module_search_paths`` is set to 1. In this case, path configuration input fields are ignored as well. If ``base_prefix`` or ``base_exec_prefix`` fields are not set, they inherit their value from ``prefix`` and ``exec_prefix`` respectively. If ``site_import`` is non-zero, ``sys.path`` can be modified by the ``site`` module. For example, if ``user_site_directory`` is non-zero, the user site directory is added to ``sys.path`` (if it exists). Isolate Python -------------- The default configuration is designed to behave as a regular Python. To embed Python into an application, it's possible to tune the configuration to better isolated the embedded Python from the system: * Set ``isolated`` to 1 to ignore environment variables and not prepend the current directory to ``sys.path``. * Set the `Path Configuration`_ ("output fields") to ignore the function computing the default path configuration. Py_BytesMain() -------------- Python 3.7 provides a high-level ``Py_Main()`` function which requires to pass command line arguments as ``wchar_t*`` strings. It is non-trivial to use the correct encoding to decode bytes. Python has its own set of issues with C locale coercion and UTF-8 Mode. This PEP adds a new ``Py_BytesMain()`` function which takes command line arguments as bytes:: int Py_BytesMain(int argc, char **argv) Py_RunMain() ------------ The new ``Py_RunMain()`` function executes the command (``PyConfig.run_command``), the script (``PyConfig.run_filename``) or the module (``PyConfig.run_module``) specified on the command line or in the configuration, and then finalizes Python. It returns an exit status that can be passed to the ``exit()`` function. Example of customized Python in isolated mode:: #include int main(int argc, char *argv[]) { PyConfig config = PyConfig_INIT; config.isolated = 1; PyInitError err = Py_InitializeFromBytesArgs(&config, argc, argv); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* put more configuration code here if needed */ return Py_RunMain(); } The example is a basic implementation of the "System Python Executable" discussed in PEP 432. Memory allocations and Py_DecodeLocale() ---------------------------------------- Python memory allocation functions like ``PyMem_RawMalloc()`` must not be used before Python pre-initialization, whereas calling directly ``malloc()`` and ``free()`` is always safe. For ``PyPreConfig`` and constant ``PyConfig``, the caller is responsible to manage dynamically allocated memory; constant strings and constant string lists can be used to avoid memory allocations. Dynamic ``PyConfig`` requires to call ``PyConfig_Clear()`` to release memory. ``Py_DecodeLocale()`` must not be called before the pre-initialization. Backwards Compatibility ======================= This PEP only adds a new API: it leaves the existing API unchanged and has no impact on the backwards compatibility. The implementation ensures that the existing API is compatible with the new API. For example, ``PyConfig`` uses the value of global configuration variables as default values. Annex: Python Configuration =========================== Priority and Rules ------------------ Priority of configuration parameters, highest to lowest: * ``PyConfig`` * ``PyPreConfig`` * Configuration files * Command line options * Environment variables * Global configuration variables Priority of warning options, highest to lowest: * ``PyConfig.warnoptions`` * ``PyConfig.dev_mode`` (add ``"default"``) * ``PYTHONWARNINGS`` environment variables * ``-W WARNOPTION`` command line argument * ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater than 1, or add ``"default::BytesWarning``) Rules on ``PyConfig`` parameters: * If ``isolated`` is non-zero, ``use_environment`` and ``user_site_directory`` are set to 0. * If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to 0. * If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``, ``faulthandler`` is set to 1, and ``"default"`` filter is added to ``warnoptions``. But the ``PYTHONMALLOC`` environment variable has the priority over ``dev_mode`` to set the memory allocator. * If ``base_prefix`` is not set, it inherits ``prefix`` value. * If ``base_exec_prefix`` is not set, it inherits ``exec_prefix`` value. * If the ``python._pth`` configuration file is present, ``isolated`` is set to 1 and ``site_import`` is set to 0; but ``site_import`` is set to 1 if ``python._pth`` contains ``import site``. Rules on ``PyConfig`` and ``PyPreConfig`` parameters: * If ``PyPreConfig.legacy_windows_fs_encoding`` is non-zero, set ``PyConfig.utf8_mode`` to 0, set ``PyConfig.filesystem_encoding`` to ``mbcs``, and set ``PyConfig.filesystem_errors`` to ``replace``. Configuration Files ------------------- Python configuration files: * ``pyvenv.cfg`` * ``python._pth`` (Windows only) * ``pybuilddir.txt`` (Unix only) Global Configuration Variables ------------------------------ Global configuration variables mapped to ``PyPreConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` ``Py_UTF8Mode`` ``utf8_mode`` ======================================== ================================ (NOT) means that the ``PyPreConfig`` value is the oposite of the global configuration variable value. Global configuration variables mapped to ``PyConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_BytesWarningFlag`` ``bytes_warning`` ``Py_DebugFlag`` ``parser_debug`` ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` (NOT) ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_FrozenFlag`` ``pathconfig_warnings`` (NOT) ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) ``Py_InspectFlag`` ``inspect`` ``Py_InteractiveFlag`` ``interactive`` ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` ``Py_NoSiteFlag`` ``site_import`` (NOT) ``Py_NoUserSiteDirectory`` ``user_site_directory`` (NOT) ``Py_OptimizeFlag`` ``optimization_level`` ``Py_QuietFlag`` ``quiet`` ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` (NOT) ``Py_VerboseFlag`` ``verbose`` ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` ======================================== ================================ (NOT) means that the ``PyConfig`` value is the oposite of the global configuration variable value. ``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are only available on Windows. Command Line Arguments ---------------------- Usage:: python3 [options] python3 [options] -c COMMAND python3 [options] -m MODULE python3 [options] SCRIPT Command line options mapped to pseudo-action on ``PyPreConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-E`` ``use_environment = 0`` ``-I`` ``isolated = 1`` ``-X dev`` ``dev_mode = 1`` ``-X utf8`` ``utf8_mode = 1`` ``-X utf8=VALUE`` ``utf8_mode = VALUE`` ================================ ================================ Command line options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-b`` ``bytes_warning++`` ``-B`` ``write_bytecode = 0`` ``-c COMMAND`` ``run_command = COMMAND`` ``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE`` ``-d`` ``parser_debug++`` ``-E`` ``use_environment = 0`` ``-i`` ``inspect++`` and ``interactive++`` ``-I`` ``isolated = 1`` ``-m MODULE`` ``run_module = MODULE`` ``-O`` ``optimization_level++`` ``-q`` ``quiet++`` ``-R`` ``use_hash_seed = 0`` ``-s`` ``user_site_directory = 0`` ``-S`` ``site_import`` ``-t`` ignored (kept for backwards compatibility) ``-u`` ``buffered_stdio = 0`` ``-v`` ``verbose++`` ``-W WARNING`` add ``WARNING`` to ``warnoptions`` ``-x`` ``skip_source_first_line = 1`` ``-X OPTION`` add ``OPTION`` to ``xoptions`` ================================ ================================ ``-h``, ``-?`` and ``-V`` options are handled without ``PyConfig``. -X Options ---------- -X options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-X dev`` ``dev_mode = 1`` ``-X faulthandler`` ``faulthandler = 1`` ``-X importtime`` ``import_time = 1`` ``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX`` ``-X showalloccount`` ``show_alloc_count = 1`` ``-X showrefcount`` ``show_ref_count = 1`` ``-X tracemalloc=N`` ``tracemalloc = N`` ================================ ================================ Environment Variables --------------------- Environment variables mapped to ``PyPreConfig`` fields: ================================= ============================================= Variable ``PyPreConfig`` field ================================= ============================================= ``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding`` ``PYTHONMALLOC`` ``allocator`` ``PYTHONUTF8`` ``utf8_mode`` ================================= ============================================= Environment variables mapped to ``PyConfig`` fields: ================================= ==================================== Variable ``PyConfig`` field ================================= ==================================== ``PYTHONDEBUG`` ``parser_debug`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONDONTWRITEBYTECODE`` ``write_bytecode`` ``PYTHONDUMPREFS`` ``dump_refs`` ``PYTHONEXECUTABLE`` ``program_name`` ``PYTHONFAULTHANDLER`` ``faulthandler`` ``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed`` ``PYTHONHOME`` ``home`` ``PYTHONINSPECT`` ``inspect`` ``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors`` ``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio`` ``PYTHONMALLOCSTATS`` ``malloc_stats`` ``PYTHONNOUSERSITE`` ``user_site_directory`` ``PYTHONOPTIMIZE`` ``optimization_level`` ``PYTHONPATH`` ``module_search_path_env`` ``PYTHONPROFILEIMPORTTIME`` ``import_time`` ``PYTHONPYCACHEPREFIX,`` ``pycache_prefix`` ``PYTHONTRACEMALLOC`` ``tracemalloc`` ``PYTHONUNBUFFERED`` ``buffered_stdio`` ``PYTHONVERBOSE`` ``verbose`` ``PYTHONWARNINGS`` ``warnoptions`` ================================= ==================================== ``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are specific to Windows. Annex: Python 3.7 API ===================== Python 3.7 has 4 functions in its C API to initialize and finalize Python: * ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python * ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python Python 3.7 can be configured using `Global Configuration Variables`_, `Environment Variables`_, and the following functions: * ``PyImport_AppendInittab()`` * ``PyImport_ExtendInittab()`` * ``PyMem_SetAllocator()`` * ``PyMem_SetupDebugHooks()`` * ``PyObject_SetArenaAllocator()`` * ``Py_SetPath()`` * ``Py_SetProgramName()`` * ``Py_SetPythonHome()`` * ``Py_SetStandardStreamEncoding()`` * ``PySys_AddWarnOption()`` * ``PySys_AddXOption()`` * ``PySys_ResetWarnOptions()`` There is also a high-level ``Py_Main()`` function. Python Issues ============= Issues that will be fixed by this PEP, directly or indirectly: * `bpo-1195571 `_: "simple callback system for Py_FatalError" * `bpo-11320 `_: "Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony)" * `bpo-13533 `_: "Would like Py_Initialize to play friendly with host app" * `bpo-14956 `_: "custom PYTHONPATH may break apps embedding Python" * `bpo-19983 `_: "When interrupted during startup, Python should not call abort() but exit()" * `bpo-22213 `_: "Make pyvenv style virtual environments easier to configure when embedding Python". This PEP more or * `bpo-22257 `_: "PEP 432: Redesign the interpreter startup sequence" * `bpo-29778 `_: "_Py_CheckPython3 uses uninitialized dllpath when embedder sets module path with Py_SetPath" * `bpo-30560 `_: "Add Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal errors". * `bpo-31745 `_: "Overloading "Py_GetPath" does not work" * `bpo-32573 `_: "All sys attributes (.argv, ...) should exist in embedded environments". * `bpo-34725 `_: "Py_GetProgramFullPath() odd behaviour in Windows" * `bpo-36204 `_: "Deprecate calling Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?" * `bpo-33135 `_: "Define field prefixes for the various config structs". The PEP now defines well how warnings options are handled. Issues of the PEP implementation: * `bpo-16961 `_: "No regression tests for -E and individual environment vars" * `bpo-20361 `_: "-W command line options and PYTHONWARNINGS environmental variable should not override -b / -bb command line options" * `bpo-26122 `_: "Isolated mode doesn't ignore PYTHONHASHSEED" * `bpo-29818 `_: "Py_SetStandardStreamEncoding leads to a memory error in debug mode" * `bpo-31845 `_: "PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect" * `bpo-32030 `_: "PEP 432: Rewrite Py_Main()" * `bpo-32124 `_: "Document functions safe to be called before Py_Initialize()" * `bpo-33042 `_: "New 3.7 startup sequence crashes PyInstaller" * `bpo-33932 `_: "Calling Py_Initialize() twice now triggers a fatal error (Python 3.7)" * `bpo-34008 `_: "Do we support calling Py_Main() after Py_Initialize()?" * `bpo-34170 `_: "Py_Initialize(): computing path configuration must not have side effect (PEP 432)" * `bpo-34589 `_: "Py_Initialize() and Py_Main() should not enable C locale coercion" * `bpo-34639 `_: "PYTHONCOERCECLOCALE is ignored when using -E or -I option" * `bpo-36142 `_: "Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings" * `bpo-36202 `_: "Calling Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake" * `bpo-36301 `_: "Add _Py_PreInitialize() function" * `bpo-36443 `_: "Disable coerce_c_locale and utf8_mode by default in _PyPreConfig?" * `bpo-36444 `_: "Python initialization: remove _PyMainInterpreterConfig" * `bpo-36471 `_: "PEP 432, PEP 587: Add _Py_RunMain()" * `bpo-36763 `_: "PEP 587: Rework initialization API to prepare second version of the PEP" * `bpo-36775 `_: "Rework filesystem codec implementation" * `bpo-36900 `_: "Use _PyCoreConfig rather than global configuration variables" Issues related to this PEP: * `bpo-12598 `_: "Move sys variable initialization from import.c to sysmodule.c" * `bpo-15577 `_: "Real argc and argv in embedded interpreter" * `bpo-16202 `_: "sys.path[0] security issues" * `bpo-18309 `_: "Make python slightly more relocatable" * `bpo-25631 `_: "Segmentation fault with invalid Unicode command-line arguments in embedded Python" * `bpo-26007 `_: "Support embedding the standard library in an executable" * `bpo-31210 `_: "Can not import modules if sys.prefix contains DELIM". * `bpo-31349 `_: "Embedded initialization ignores Py_SetProgramName()" * `bpo-33919 `_: "Expose _PyCoreConfig structure to Python" * `bpo-35173 `_: "Re-use already existing functionality to allow Python 2.7.x (both embedded and standalone) to locate the module path according to the shared library" Version History =============== * Version 3: * ``PyConfig``: Add ``configure_c_stdio`` and ``parse_argv``, rename ``_frozen`` to ``pathconfig_warnings``. * Rename functions using bytes strings and wide character strings. For example, ``Py_PreInitializeFromWideArgs`` becomes ``Py_PreInitializeFromArgs``, and ``PyConfig_SetArgv`` becomes ``PyConfig_SetBytesArgv``. * Add ``PyWideStringList_Insert()`` function. * New "Path configuration", "Isolate Python", "Python Issues" and "Version History" sections. * ``PyConfig_SetString()`` and ``PyConfig_SetBytesString()`` now requires the configuration as the first argument. * Rename ``Py_UnixMain()`` to ``Py_BytesMain()`` * Version 2: Add ``PyConfig`` methods (ex: ``PyConfig_Read()``), add ``PyWideStringList_Append()``, rename ``PyWideCharList`` to ``PyWideStringList``. * Version 1: Initial version. Copyright ========= This document has been placed in the public domain. From vstinner at redhat.com Wed May 15 21:23:55 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 03:23:55 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() Message-ID: Hi, I recently modified Python 3.8 to start logging I/O error in io destructors when using the development mode (-X dev) or when Python is built in debug mode. This change allows to debug bpo-18748 very strange crash in PyThread_exit_thread(), when a thread closes an arbitrary file descriptor by mistake. The problem is that exceptions raised in a destructor cannot be passed to the current function call: the exception is logged into sys.stderr with a message like "Exception ignored in: ...". It's easy to miss such message in stderr. Thomas Grainger opened 36829 to propose to add -X abortunraisable command line option: abort immediately the current process (by calling Py_FatalError()) at the first PyErr_WriteUnraisable() call. Thomas had like a way to easily detect these in CI. I'm not satisfied by his proposal, since it only gives little control to the user on how "unraisable exceptions" are handled: the process dies, that's all. https://bugs.python.org/issue36829 I proposed a different approach: add a new sys.unraisablehook hook which is called to handle an "unraisable exception". To handle them differently, replace the hook. For example, I wrote a custom hook to log these exceptions into a file (the output on the Python test suite is interesting!). It also becomes trivial to reimplement Thomas's idea (kill the process): import signal def hook(unraisable): signal.raise_signal(signal.SIGABRT) sys.unraisablehook = hook I plan to merge my implementation soon, are you fine with that? https://github.com/python/cpython/pull/13187 -- The first implementation of my API used sys.unraisablehook(exc_type, exc_value, exc_tb, obj). The problem is that Serhiy Storchaka asked me to add a new error message field which breaks the API: the API is not future-proof. I modified my API to create an object to pack arguments. The new API becomes sys.unraisablehook(unraisable) where unraisable has 4 fields: exc_type, exc_value, exc_tb, obj. This API is now future-proof: adding a new field will not break existing custom hooks! By the way, I like this idea of adding an optional error message, and I plan to implement it once sys.unraisablehook is merged ;-) I already implemented it previously, but I reverted my changes to make the PR simpler to review. Extract of the documentation: """ Add new :func:`sys.unraisablehook` function which can be overriden to control how "unraisable exceptions" are handled. It is called when an exception has occurred but there is no way for Python to handle it. For example, when a destructor raises an exception or during garbage collection (:func:`gc.collect`). """ My implementation has a limitation: if PyErr_WriteUnraisable() is called after the Python finalization cleared sys attributes (almost the last function call of Py_Finalize), the default hook is called instead of the custom hook. In this case, sys.stderr is None and so the default hook does nothing. These late calls to PyErr_WriteUnraisable() cannot be catched with my approached, whereas Thomas Grainger's command line option "-X abortunraisable" allows that. My concern with Thomas's approach is that if the problem is killed with SIGABRT by such late PyErr_WriteUnraisable(), a low-level debugger like gdb is needed to investigate the crash, since Python is already finalized and so cannot be used to investigate. I prefer to allow arbitrary hook with the limitation, rather than always kill the process with SIGABRT at the first PyErr_WriteUnraisable() and require to use a low-level debugger. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From njs at pobox.com Wed May 15 22:44:38 2019 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 15 May 2019 19:44:38 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: On Wed, May 15, 2019 at 6:25 PM Victor Stinner wrote: > I proposed a different approach: add a new sys.unraisablehook hook > which is called to handle an "unraisable exception". To handle them > differently, replace the hook. For example, I wrote a custom hook to > log these exceptions into a file (the output on the Python test suite > is interesting!). It also becomes trivial to reimplement Thomas's idea > (kill the process): What happens if the hook raises an exception? -n -- Nathaniel J. Smith -- https://vorpus.org From gregory.szorc at gmail.com Thu May 16 00:34:18 2019 From: gregory.szorc at gmail.com (Gregory Szorc) Date: Wed, 15 May 2019 21:34:18 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: References: Message-ID: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> On 5/15/2019 4:10 PM, Victor Stinner wrote: > Hi, > > Thanks to the constructive discussions, I enhanced my PEP 587. I don't > plan any further change, the PEP is now ready to review (and maybe > even for pronouncement, hi Thomas! :-)). > > The Rationale now better explains all challenges and the complexity of > the Python Initialization Configuration. > > The "Isolate Python" section is a short guide explaining how configure > Python to embed it into an application. > > The "Path Configuration" section elaborates the most interesting part > of the configuration: configure where Python looks for modules > (sys.path). I added PyWideStringList_Insert() to allow to prepend a > path in module_search_paths. > > The "Python Issues" section give a long list of issues solved directly > or indirectly by this PEP. > > I'm open for bikeshedding on PyConfig fields names and added functions > names ;-) I hesitate for "use_module_search_paths": maybe > "module_search_paths_set" is a better name, as in "is > module_search_paths set?". The purpose of this field is to allow to > have an empty sys.path (ask PyConfig_Read() to not override it). IMHO > an empty sys.path makes sense for some specific use cases, like > executing Pyhon code without any external module. > > My PEP 587 proposes better names: Py_FrozenFlag becomes > PyConfig.pathconfig_warnings and Py_DebugFlag becomes > PyConfig.parser_debug. I also avoided double negation. For example, > Py_DontWriteBytecodeFlag becomes write_bytecode. > > Changes between version 3 and version 2: > > * PyConfig: Add configure_c_stdio and parse_argv; rename _frozen to > pathconfig_warnings. > * Rename functions using bytes strings and wide strings. For example, > Py_PreInitializeFromWideArgs() becomes Py_PreInitializeFromArgs(), and > PyConfig_SetArgv() becomes PyConfig_SetBytesArgv(). > * Add PyWideStringList_Insert() function. > * New "Path configuration", "Isolate Python", "Python Issues" and > "Version History" sections. > * PyConfig_SetString() and PyConfig_SetBytesString() now requires the > configuration as the first argument. > * Rename Py_UnixMain() to Py_BytesMain() > > > HTML version: > https://www.python.org/dev/peps/pep-0587/ > > Full PEP text below. > > I know that the PEP is long, but well, it's a complex topic, and I > chose to add many examples to make the API easier to understand. I saw your request for feedback on Twitter a few days back and found this thread. This PEP is of interest to me because I'm the maintainer of PyOxidizer - a project for creating single file executables embedding Python. As part of hacking on PyOxidizer, I will admit to grumbling about the current state of the configuration and initialization mechanisms. The reliance on global variables and the haphazard way in which you must call certain functions before others was definitely a bit frustrating to deal with. I don't want to wade into too much bikeshedding in my review. I'll let the professionals deal with things like naming :) Also, I haven't read previous posts about this PEP. Apologies if my comments bring up old topics. Let's get on with the review... My most important piece of feedback is: thank you for tackling this! Your work to shore up the inner workings of interpreter state and management is a big deal on multiple dimensions. I send my sincere gratitude. Overall, I'm very happy with the state of the proposal. Better than what we currently have on nearly every dimension. When reading my feedback, please keep in mind that I'm in like 95% agreement with the proposal as is. The following paragraphs detail points of feedback. PyPreConfig_INIT and PyConfig_INIT as macros that return a struct feel weird to me. Specifically, the `PyPreConfig preconfig = PyPreConfig_INIT;` pattern doesn't feel right. I'm sort of OK with these being implemented as macros. But I think they should look like function calls so the door is open to converting them to function calls in the future. An argument to make them actual function calls today is to facilitate better FFI interop. As it stands, non-C/C++ bindings to the API will need to reimplement the macro's logic. That might be simple today. But who knows what complexity may be added in years ahead. An opaque function implementation future proofs the API. PyPreConfig.allocator being a char* seems a bit weird. Does this imply having to use strcmp() to determine which allocator to use? Perhaps the allocator setting should be an int mapping to a constant instead? Relatedly, how are custom allocators registered? e.g. from Rust, I want to use Rust's allocator. How would I do that in this API? Do I still need to call PyMem_SetAllocator()? I thought a point of this proposal was to consolidate per-interpreter config settings? I'm a little confused about the pre-initialization functions that take command arguments. Is this intended to only be used for parsing the arguments that `python` recognizes? Presumably a custom application embedding Python would never use these APIs unless it wants to emulate the behavior of `python`? (I suppose this can be clarified in the API docs once this is implemented.) What about PyImport_FrozenModules? This is a global variable related to Python initialization (it contains _frozen_importlib and _frozen_importlib_external) but it is not accounted for in the PEP. I rely on this struct in PyOxidizer to replace the importlib modules with custom versions so we can do 0-copy in-memory import of Python bytecode for the entirety of the standard library. Should the PyConfig have a reference to the _frozen[] to use? Should the _frozen struct be made part of the public API? The PEP mentions a private PyConfig._install_importlib member. I'm curious what this is because it may be relevant to PyOxidizer. FWIW I /might/ be interested in a mechanism to better control importlib initialization because PyOxidizer is currently doing dirty things at run-time to register the custom 0-copy meta path importer. I /think/ my desired API would be a mechanism to control the name(s) of the frozen module(s) to use to bootstrap importlib. Or there would be a way to register the names of additional frozen modules to import and run as part of initializing importlib (before any .py-based stdlib modules are imported). Then PyOxidizer wouldn't need to hack up the source code to importlib, compile custom bytecode, and inject it via PyImport_FrozenModules. I concede this may be out of scope for the PEP. But if the API is being reworked, I'd certainly welcome making it easier for tools like PyOxidizer to work their crazy module importing magic :) I really like the new Py_RunMain() API and associated PyConfig members. I also invented this wheel in PyOxidizer and the new API should result in me deleting some code that I wish I didn't have to write in the first place :) Since I mentioned PyOxidizer a lot, you may want to take a gander at https://github.com/indygreg/PyOxidizer/blob/64514f862b57846801f9ae4af5968e2e4a541ab7/pyoxidizer/src/pyembed/pyinterp.rs. That's the code I wrote for embedding a Python interpreter in Rust. I invented a data structure for representing a Python interpreter configuration. And the similarities to PyConfig are striking. I think that's a good sign :) It might be useful to read through that file - especially the init function (line with `pub fn init`) to see if anything I'm doing pushes the boundaries of the proposed API. Feel free to file GitHub issues if you see obvious bugs with PyOxidizer's Python initialization logic while you're at it :) Also, one thing that tripped me up a few times when writing PyOxidizer was managing the lifetimes of memory that various global variables point to. The short version is I was setting Python globals to point to memory allocated by Rust and I managed to crash Python by freeing memory before it should have been. Since the new API seems to preserve support for global variables, I'm curious if there are changes to how memory must be managed. It would be really nice to get to a state where you only need to ensure the PyConfig instance and all its referenced memory only needs to outlive the interpreter it configures. That would make the memory lifetimes story intuitive and easily compatible with Rust. One feature that I think is missing from the proposal (and this is related to the previous paragraph) is the ability to prevent config fallback to things that aren't PyConfig and PyPreConfig. There is `PyConfig.parse_argv` to disable command argument parsing and `PyConfig.use_environment` to disable environment variable fallback. But AFAICT there is no option to disable configuration file fallback nor global variable fallback. As someone who embeds Python and loves total control, I would absolutely love to opt in to a "Py[Pre]Config only" mode where those structs are the only things that control interpreter behavior. I'd opt in to that in a heartbeat if it supported all the customization that PyOxidizer requires! ... and I think that's all the points of feedback I have! Again, this proposal is terrific overall and so much better than what we have today. The wall of text I just wrote is disproportionate in size to the quality of the PEP. I almost feel bad writing so much feedback for such a terrific PEP ;) Excellent work, Victor. I can't wait to see these changes materialize! > --- > > PEP: 587 > Title: Python Initialization Configuration > Author: Victor Stinner , Nick Coghlan > BDFL-Delegate: Thomas Wouters > Discussions-To: python-dev at python.org > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 27-Mar-2019 > Python-Version: 3.8 > > Abstract > ======== > > Add a new C API to configure the Python Initialization providing finer > control on the whole configuration and better error reporting. > > It becomes possible to read the configuration and modify it before it is > applied. It also becomes possible to completely override how Python > computes the module search paths (``sys.path``). > > Building a customized Python which behaves as regular Python becomes > easier using the new ``Py_RunMain()`` function. Moreover, command line > arguments passed to ``PyConfig.argv`` are now parsed as the regular > Python parses command line options, and ``PyConfig.xoptions`` are > handled as ``-X opt`` command line options. > > This extracts a subset of the API design from the PEP 432 development and > refactoring work that is now considered sufficiently stable to make public > (allowing 3rd party embedding applications access to the same configuration > APIs that the native CPython CLI is now using). > > > Rationale > ========= > > Python is highly configurable but its configuration evolved organically. > The initialization configuration is scattered all around the code using > different ways to set them: global configuration variables (ex: > ``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``), > command line arguments (ex: ``-b``), configuration files (ex: > ``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A > straightforward and reliable way to configure Python is needed. > > Some configuration parameters are not accessible from the C API, or not > easily. For example, there is no API to override the default values of > ``sys.executable``. > > Some options like ``PYTHONPATH`` can only be set using an environment > variable which has a side effect on Python child processes. > > Some options also depends on other options: see `Priority and Rules`_. > Python 3.7 API does not provide a consistent view of the overall > configuration. > > The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as > input whereas the Python filesystem encoding is set during the > initialization which can lead to mojibake. > > Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory > allocation failure which is not convenient when Python is embedded. > Moreover, ``Py_Main()`` could exit directly the process rather than > returning an exit code. Proposed new API reports the error or exit code > to the caller which can decide how to handle it. > > Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly > was almost impossible in Python 3.6. The code base has been deeply > reworked in Python 3.7 and then in Python 3.8 to read the configuration > into a structure with no side effect. It becomes possible to clear the > configuration (release memory) and read again the configuration if the > encoding changed . It is required to implement properly the UTF-8 which > changes the encoding using ``-X utf8`` command line option. Internally, > bytes ``argv`` strings are decoded from the filesystem encoding. The > ``-X dev`` changes the memory allocator (behaves as > ``PYTHONMALLOC=debug``), whereas it was not possible to change the > memory allocation *while* parsing the command line arguments. The new > design of the internal implementation not only allowed to implement > properly ``-X utf8`` and ``-X dev``, it also allows to change the Python > behavior way more easily, especially for corner cases like that, and > ensure that the configuration remains consistent: see `Priority and > Rules`_. > > This PEP is a partial implementation of PEP 432 which is the overall > design. New fields can be added later to ``PyConfig`` structure to > finish the implementation of the PEP 432 (e.g. by adding a new partial > initialization API which allows to configure Python using Python objects to > finish the full initialization). However, those features are omitted from this > PEP as even the native CPython CLI doesn't work that way - the public API > proposal in this PEP is limited to features which have already been implemented > and adopted as private APIs for us in the native CPython CLI. > > > Python Initialization C API > =========================== > > This PEP proposes to add the following new structures, functions and > macros. > > New structures (4): > > * ``PyConfig`` > * ``PyInitError`` > * ``PyPreConfig`` > * ``PyWideStringList`` > > New functions (17): > > * ``Py_PreInitialize(config)`` > * ``Py_PreInitializeFromBytesArgs(config, argc, argv)`` > * ``Py_PreInitializeFromArgs(config, argc, argv)`` > * ``PyWideStringList_Append(list, item)`` > * ``PyWideStringList_Insert(list, index, item)`` > * ``PyConfig_SetString(config,config_str, str)`` > * ``PyConfig_SetBytesString(config, config_str, str)`` > * ``PyConfig_SetBytesArgv(config, argc, argv)`` > * ``PyConfig_SetArgv(config, argc, argv)`` > * ``PyConfig_Read(config)`` > * ``PyConfig_Clear(config)`` > * ``Py_InitializeFromConfig(config)`` > * ``Py_InitializeFromBytesArgs(config, argc, argv)`` > * ``Py_InitializeFromArgs(config, argc, argv)`` > * ``Py_BytesMain(argc, argv)`` > * ``Py_RunMain()`` > * ``Py_ExitInitError(err)`` > > New macros (9): > > * ``PyPreConfig_INIT`` > * ``PyConfig_INIT`` > * ``Py_INIT_OK()`` > * ``Py_INIT_ERR(MSG)`` > * ``Py_INIT_NO_MEMORY()`` > * ``Py_INIT_EXIT(EXITCODE)`` > * ``Py_INIT_IS_ERROR(err)`` > * ``Py_INIT_IS_EXIT(err)`` > * ``Py_INIT_FAILED(err)`` > > This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type) > and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these > internal structures. ``PyInterpreterState.config`` becomes the new > reference configuration, replacing global configuration variables and > other private variables. > > > PyWideStringList > ---------------- > > ``PyWideStringList`` is a list of ``wchar_t*`` strings. > > Example to initialize a string from C static array:: > > static wchar_t* argv[2] = { > L"-c", > L"pass", > }; > PyWideStringList config_argv = PyWideStringList_INIT; > config_argv.length = Py_ARRAY_LENGTH(argv); > config_argv.items = argv; > > ``PyWideStringList`` structure fields: > > * ``length`` (``Py_ssize_t``) > * ``items`` (``wchar_t**``) > > Methods: > > * ``PyInitError PyWideStringList_Append(PyWideStringList *list, const > wchar_t *item)``: > Append *item* to *list*. > * ``PyInitError PyWideStringList_Insert(PyWideStringList *list, > Py_ssize_t index, const wchar_t *item)``: > Insert *item* into *list* at *index*. If *index* is greater than > *list* length, just append *item* to *list*. > > If *length* is non-zero, *items* must be non-NULL and all strings must > be non-NULL. > > PyInitError > ----------- > > ``PyInitError`` is a structure to store an error message or an exit code > for the Python Initialization. For an error, it stores the C function > name which created the error. > > Example:: > > PyInitError alloc(void **ptr, size_t size) > { > *ptr = PyMem_RawMalloc(size); > if (*ptr == NULL) { > return Py_INIT_NO_MEMORY(); > } > return Py_INIT_OK(); > } > > int main(int argc, char **argv) > { > void *ptr; > PyInitError err = alloc(&ptr, 16); > if (Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > PyMem_Free(ptr); > return 0; > } > > ``PyInitError`` fields: > > * ``exitcode`` (``int``): > argument passed to ``exit()``, only set by ``Py_INIT_EXIT()``. > * ``err_msg`` (``const char*``): error message > * private ``_func`` field: used by ``Py_INIT_ERR()`` to store the C > function name which created the error. > * private ``_type`` field: for internal usage only. > > Macro to create an error: > > * ``Py_INIT_OK()``: Success. > * ``Py_INIT_ERR(err_msg)``: Initialization error with a message. > * ``Py_INIT_NO_MEMORY()``: Memory allocation failure (out of memory). > * ``Py_INIT_EXIT(exitcode)``: Exit Python with the specified exit code. > > Other macros and functions: > > * ``Py_INIT_IS_ERROR(err)``: Is the result an error? > * ``Py_INIT_IS_EXIT(err)``: Is the result an exit? > * ``Py_INIT_FAILED(err)``: Is the result an error or an exit? Similar > to ``Py_INIT_IS_ERROR(err) || Py_INIT_IS_EXIT(err)``. > * ``Py_ExitInitError(err)``: Call ``exit(exitcode)`` on Unix or > ``ExitProcess(exitcode)`` if the result is an exit, call > ``Py_FatalError(err_msg)`` if the result is an error. Must not be > called if the result is a success. > > Pre-Initialization with PyPreConfig > ----------------------------------- > > ``PyPreConfig`` structure is used to pre-initialize Python: > > * Set the memory allocator > * Configure the LC_CTYPE locale > * Set the UTF-8 mode > > Example using the pre-initialization to enable the UTF-8 Mode:: > > PyPreConfig preconfig = PyPreConfig_INIT; > preconfig.utf8_mode = 1; > > PyInitError err = Py_PreInitialize(&preconfig); > if (Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > > /* at this point, Python will speak UTF-8 */ > > Py_Initialize(); > /* ... use Python API here ... */ > Py_Finalize(); > > Functions to pre-initialize Python: > > * ``PyInitError Py_PreInitialize(const PyPreConfig *config)`` > * ``PyInitError Py_PreInitializeFromBytesArgs(const PyPreConfig > *config, int argc, char **argv)`` > * ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *config, > int argc, wchar_t **argv)`` > > These functions can be called with *config* set to ``NULL``. > > If Python is initialized with command line arguments, the command line > arguments must also be passed to pre-initialize Python, since they have > an effect on the pre-configuration like encodings. For example, the > ``-X utf8`` command line option enables the UTF-8 Mode. > > These functions can be called with *config* set to ``NULL``. The caller > is responsible to handle error or exit using ``Py_INIT_FAILED()`` and > ``Py_ExitInitError()``. > > ``PyPreConfig`` fields: > > * ``allocator`` (``char*``, default: ``NULL``): > Name of the memory allocator (ex: ``"malloc"``). > * ``coerce_c_locale`` (``int``, default: 0): > If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE > locale to decide if it should be coerced. > * ``coerce_c_locale_warn`` (``int``, default: 0): > If non-zero, emit a warning if the C locale is coerced. > * ``dev_mode`` (``int``, default: 0): > See ``PyConfig.dev_mode``. > * ``isolated`` (``int``, default: 0): > See ``PyConfig.isolated``. > * ``legacy_windows_fs_encoding`` (``int``, Windows only, default: 0): > If non-zero, disable UTF-8 Mode, set the Python filesystem encoding to > ``mbcs``, set the filesystem error handler to ``replace``. > * ``use_environment`` (``int``, default: 1): > See ``PyConfig.use_environment``. > * ``utf8_mode`` (``int``, default: 0): > If non-zero, enable the UTF-8 mode. > > ``PyPreConfig`` private field, for internal use only: > > * ``_config_version`` (``int``, default: config version): > Configuration version, used for ABI compatibility. > > The C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) are > disabled by default in ``PyPreConfig``. Set ``coerce_c_locale``, > ``coerce_c_locale_warn`` and ``utf8_mode`` to ``-1`` to let Python > enable them depending on the user configuration. In this case, it's > safer to explicitly pre-initialize Python to ensure that encodings are > configured before the Python initialization starts. Example to get the > same encoding than regular Python:: > > PyPreConfig preconfig = PyPreConfig_INIT; > preconfig.coerce_c_locale = -1; > preconfig.coerce_c_locale_warn = -1; > preconfig.utf8_mode = -1; > > PyInitError err = Py_PreInitialize(&preconfig); > if (Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > > > Initialization with PyConfig > ---------------------------- > > The ``PyConfig`` structure contains all parameters to configure Python. > > Example setting the program name:: > > PyInitError err; > PyConfig config = PyConfig_INIT; > > err = PyConfig_SetString(&config.program_name, L"my_program"); > if (_Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > > err = Py_InitializeFromConfig(&config); > PyConfig_Clear(&config); > > if (Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > > ``PyConfig`` methods: > > * ``PyInitError PyConfig_SetString(PyConfig *config, wchar_t > **config_str, const wchar_t *str)``: > Copy the wide character string *str* into ``*config_str``. > * ``PyInitError PyConfig_SetBytesString(PyConfig *config, wchar_t > **config_str, const char *str)``: > Decode *str* using ``Py_DecodeLocale()`` and set the result into > ``*config_str``. Pre-initialize Python if needed to ensure that > encodings are properly configured. > * ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, wchar_t **argv)``: > Set command line arguments from wide character strings. > * ``PyInitError PyConfig_SetBytesArgv(PyConfig *config, int argc, char > **argv)``: > Set command line arguments: decode bytes using ``Py_DecodeLocale()``. > Pre-initialize Python if needed to ensure that encodings are properly > configured. > * ``PyInitError PyConfig_Read(PyConfig *config)``: > Read all Python configuration. Fields which are already set are left > unchanged. > * ``void PyConfig_Clear(PyConfig *config)``: > Release configuration memory. > > Functions to initialize Python: > > * ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``: > Initialize Python from *config* configuration. *config* can be > ``NULL``. > > The caller of these methods and functions is responsible to handle > failure or exit using ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. > > ``PyConfig`` fields: > > * ``argv`` (``PyWideStringList``, default: empty): > Command line arguments, ``sys.argv``. > It is parsed and updated by default, set ``parse_argv`` to 0 to avoid > that. > * ``base_exec_prefix`` (``wchar_t*``, default: ``NULL``): > ``sys.base_exec_prefix``. > * ``base_prefix`` (``wchar_t*``, default: ``NULL``): > ``sys.base_prefix``. > * ``buffered_stdio`` (``int``, default: 1): > If equals to 0, enable unbuffered mode, make stdout and stderr streams > to be unbuffered. > * ``bytes_warning`` (``int``, default: 0): > If equals to 1, issue a warning when comparing ``bytes`` or > ``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If > equal or greater to 2, raise a ``BytesWarning`` exception. > * ``check_hash_pycs_mode`` (``wchar_t*``, default: ``"default"``): > ``--check-hash-based-pycs`` command line option value (see PEP 552). > * ``configure_c_stdio`` (``int``, default: 1): > If non-zero, configure C standard streams (``stdio``, ``stdout``, > ``stdout``). For example, set their mode to ``O_BINARY`` on Windows. > * ``dev_mode`` (``int``, default: 0): > Development mode > * ``dll_path`` (``wchar_t*``, Windows only, default: ``NULL``): > Windows DLL path. > * ``dump_refs`` (``int``, default: 0): > If non-zero, dump all objects which are still alive at exit > * ``exec_prefix`` (``wchar_t*``, default: ``NULL``): > ``sys.exec_prefix``. > * ``executable`` (``wchar_t*``, default: ``NULL``): > ``sys.executable``. > * ``faulthandler`` (``int``, default: 0): > If non-zero, call ``faulthandler.enable()``. > * ``filesystem_encoding`` (``wchar_t*``, default: ``NULL``): > Filesystem encoding, ``sys.getfilesystemencoding()``. > * ``filesystem_errors`` (``wchar_t*``, default: ``NULL``): > Filesystem encoding errors, ``sys.getfilesystemencodeerrors()``. > * ``use_hash_seed`` (``int``, default: 0), > ``hash_seed`` (``unsigned long``, default: 0): > Randomized hash function seed. > * ``home`` (``wchar_t*``, default: ``NULL``): > Python home directory. > * ``import_time`` (``int``, default: 0): > If non-zero, profile import time. > * ``inspect`` (``int``, default: 0): > Enter interactive mode after executing a script or a command. > * ``install_signal_handlers`` (``int``, default: 1): > Install signal handlers? > * ``interactive`` (``int``, default: 0): > Interactive mode. > * ``legacy_windows_stdio`` (``int``, Windows only, default: 0): > If non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for > ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. > * ``malloc_stats`` (``int``, default: 0): > If non-zero, dump memory allocation statistics at exit. > * ``module_search_path_env`` (``wchar_t*``, default: ``NULL``): > ``PYTHONPATH`` environment variale value. > * ``use_module_search_paths`` (``int``, default: 0), > ``module_search_paths`` (``PyWideStringList``, default: empty): > ``sys.path``. > * ``optimization_level`` (``int``, default: 0): > Compilation optimization level. > * ``parse_argv`` (``int``, default: 1): > If non-zero, parse ``argv`` command line arguments and update > ``argv``. > * ``parser_debug`` (``int``, default: 0): > If non-zero, turn on parser debugging output (for expert only, > depending on compilation options). > * ``pathconfig_warnings`` (``int``, default: 1): > If equal to 0, suppress warnings when computing the path > configuration. > * ``prefix`` (``wchar_t*``, default: ``NULL``): > ``sys.prefix``. > * ``program_name`` (``wchar_t*``, default: ``NULL``): > Program name. > * ``program`` (``wchar_t*``, default: ``NULL``): > ``argv[0]`` or an empty string. > * ``pycache_prefix`` (``wchar_t*``, default: ``NULL``): > ``.pyc`` cache prefix. > * ``quiet`` (``int``, default: 0): > Quiet mode. For example, don't display the copyright and version > messages even in interactive mode. > * ``run_command`` (``wchar_t*``, default: ``NULL``): > ``-c COMMAND`` argument. > * ``run_filename`` (``wchar_t*``), default: ``NULL``: > ``python3 SCRIPT`` argument. > * ``run_module`` (``wchar_t*``, default: ``NULL``): > ``python3 -m MODULE`` argument. > * ``show_alloc_count`` (``int``, default: 0): > Show allocation counts at exit? > * ``show_ref_count`` (``int``, default: 0): > Show total reference count at exit? > * ``site_import`` (``int``, default: 1): > Import the ``site`` module at startup? > * ``skip_source_first_line`` (``int``, default: 0): > Skip the first line of the source? > * ``stdio_encoding`` (``wchar_t*``, default: ``NULL``), > ``stdio_errors`` (``wchar_t*``, default: ``NULL``): > Encoding and encoding errors of ``sys.stdin``, ``sys.stdout`` > and ``sys.stderr``. > * ``tracemalloc`` (``int``, default: 0): > If non-zero, call ``tracemalloc.start(value)``. > * ``user_site_directory`` (``int``, default: 1): > If non-zero, add user site directory to ``sys.path``. > * ``verbose`` (``int``, default: 0): > If non-zero, enable verbose mode. > * ``warnoptions`` (``PyWideStringList``, default: empty): > Options of the ``warnings`` module to build warnings filters. > * ``write_bytecode`` (``int``, default: 1): > If non-zero, write ``.pyc`` files. > * ``xoptions`` (``PyWideStringList``, default: empty): > ``sys._xoptions``. > > ``PyConfig`` private fields, for internal use only: > > * ``_config_version`` (``int``, default: config version): > Configuration version, used for ABI compatibility. > * ``_install_importlib`` (``int``, default: 1): > Install importlib? > * ``_init_main`` (``int``, default: 1): > If equal to 0, stop Python initialization before the "main" phase > (see PEP 432). > > By default, the ``argv`` arguments are parsed as regular Python command > line arguments and ``argv`` is updated to strip parsed Python arguments: > see `Command Line Arguments`_. Set ``parse_argv`` to 0 to avoid parsing > and updating ``argv``. If ``argv`` is empty, an empty string is added to > ensure that ``sys.argv`` always exists and is never empty. > > The ``xoptions`` options are parsed to set other options: see `-X > Options`_. > > More complete example modifying the configuration before calling > ``PyConfig_Read()``, and then modify the read configuration:: > > PyInitError init_python(const char *program_name) > { > PyInitError err; > PyConfig config = PyConfig_INIT; > > /* Set the program name before reading the configuraton > (decode byte string from the locale encoding) */ > err = PyConfig_SetBytesString(&config.program_name, > program_name); > if (_Py_INIT_FAILED(err)) { > goto fail; > } > > /* Read all configuration at once */ > err = PyConfig_Read(&config); > if (_Py_INIT_FAILED(err)) { > goto fail; > } > > /* Append our custom search path to sys.path */ > err = PyWideStringList_Append(&config.module_search_paths, > L"/path/to/more/modules"); > if (_Py_INIT_FAILED(err)) { > goto fail; > } > > /* Override executable computed by PyConfig_Read() */ > err = PyConfig_SetString(&config, &config.executable, L"my_executable"); > if (_Py_INIT_FAILED(err)) { > goto fail; > } > > err = Py_InitializeFromConfig(&config); > > /* Py_InitializeFromConfig() copied config which must now be > cleared to release memory */ > PyConfig_Clear(&config); > > return err; > > fail: > PyConfig_Clear(&config); > Py_ExitInitError(err); > } > > .. note:: > ``PyConfig`` does not have any field for extra inittab functions: > ``PyImport_AppendInittab()`` and ``PyImport_ExtendInittab()`` > functions are still relevant (and can be called before Python > initialization). > > > Initialization with constant PyConfig > ------------------------------------- > > When no ``PyConfig`` method is used but only > ``Py_InitializeFromConfig()``, the caller is responsible for managing > ``PyConfig`` memory. In that case, constant strings and constant string > lists can be used to avoid dynamically allocated memory. It can be used > for most simple configurations. > > Example of Python initialization enabling the isolated mode:: > > PyConfig config = PyConfig_INIT; > config.isolated = 1; > > PyInitError err = Py_InitializeFromConfig(&config); > if (Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > /* ... use Python API here ... */ > Py_Finalize(); > > ``PyConfig_Clear()`` is not needed in this example since ``config`` does > not contain any dynamically allocated string: > ``Py_InitializeFromConfig`` is responsible to fill other fields and > manage the memory. > > For convenience, two other functions are provided for constant > ``PyConfig``: > > * ``PyInitError Py_InitializeFromArgs(const PyConfig *config, int > argc, wchar_t **argv)`` > * ``PyInitError Py_InitializeFromBytesArgs(const PyConfig *config, int > argc, char **argv)`` > > They be called with *config* set to ``NULL``. The caller of these > functions is responsible to handle failure or exit using > ``Py_INIT_FAILED()`` and ``Py_ExitInitError()``. > > > Path Configuration > ------------------ > > ``PyConfig`` contains multiple fields for the path configuration: > > * Path configuration input fields: > > * ``home`` > * ``module_search_path_env`` > * ``pathconfig_warnings`` > > * Path configuration output fields: > > * ``dll_path`` (Windows only) > * ``exec_prefix`` > * ``executable`` > * ``prefix`` > * ``use_module_search_paths``, ``module_search_paths`` > > Set ``pathconfig_warnings`` to 0 to suppress warnings when computing the > path configuration. > > It is possible to completely ignore the function computing the default > path configuration by setting explicitly all path configuration output > fields listed above. A string is considered as set even if it's an empty > string. ``module_search_paths`` is considered as set if > ``use_module_search_paths`` is set to 1. In this case, path > configuration input fields are ignored as well. > > If ``base_prefix`` or ``base_exec_prefix`` fields are not set, they > inherit their value from ``prefix`` and ``exec_prefix`` respectively. > > If ``site_import`` is non-zero, ``sys.path`` can be modified by the > ``site`` module. For example, if ``user_site_directory`` is non-zero, > the user site directory is added to ``sys.path`` (if it exists). > > > Isolate Python > -------------- > > The default configuration is designed to behave as a regular Python. > To embed Python into an application, it's possible to tune the > configuration to better isolated the embedded Python from the system: > > * Set ``isolated`` to 1 to ignore environment variables and not prepend > the current directory to ``sys.path``. > * Set the `Path Configuration`_ ("output fields") to ignore the function > computing the default path configuration. > > > Py_BytesMain() > -------------- > > Python 3.7 provides a high-level ``Py_Main()`` function which requires > to pass command line arguments as ``wchar_t*`` strings. It is > non-trivial to use the correct encoding to decode bytes. Python has its > own set of issues with C locale coercion and UTF-8 Mode. > > This PEP adds a new ``Py_BytesMain()`` function which takes command line > arguments as bytes:: > > int Py_BytesMain(int argc, char **argv) > > Py_RunMain() > ------------ > > The new ``Py_RunMain()`` function executes the command > (``PyConfig.run_command``), the script (``PyConfig.run_filename``) or > the module (``PyConfig.run_module``) specified on the command line or in > the configuration, and then finalizes Python. It returns an exit status > that can be passed to the ``exit()`` function. > > Example of customized Python in isolated mode:: > > #include > > int main(int argc, char *argv[]) > { > PyConfig config = PyConfig_INIT; > config.isolated = 1; > > PyInitError err = Py_InitializeFromBytesArgs(&config, argc, argv); > if (Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > > /* put more configuration code here if needed */ > > return Py_RunMain(); > } > > The example is a basic implementation of the "System Python Executable" > discussed in PEP 432. > > > Memory allocations and Py_DecodeLocale() > ---------------------------------------- > > Python memory allocation functions like ``PyMem_RawMalloc()`` must not > be used before Python pre-initialization, whereas calling directly > ``malloc()`` and ``free()`` is always safe. > > For ``PyPreConfig`` and constant ``PyConfig``, the caller is responsible > to manage dynamically allocated memory; constant strings and constant > string lists can be used to avoid memory allocations. > > Dynamic ``PyConfig`` requires to call ``PyConfig_Clear()`` to release > memory. > > ``Py_DecodeLocale()`` must not be called before the pre-initialization. > > > Backwards Compatibility > ======================= > > This PEP only adds a new API: it leaves the existing API unchanged and > has no impact on the backwards compatibility. > > The implementation ensures that the existing API is compatible with the > new API. For example, ``PyConfig`` uses the value of global > configuration variables as default values. > > > Annex: Python Configuration > =========================== > > Priority and Rules > ------------------ > > Priority of configuration parameters, highest to lowest: > > * ``PyConfig`` > * ``PyPreConfig`` > * Configuration files > * Command line options > * Environment variables > * Global configuration variables > > Priority of warning options, highest to lowest: > > * ``PyConfig.warnoptions`` > * ``PyConfig.dev_mode`` (add ``"default"``) > * ``PYTHONWARNINGS`` environment variables > * ``-W WARNOPTION`` command line argument > * ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater > than 1, or add ``"default::BytesWarning``) > > Rules on ``PyConfig`` parameters: > > * If ``isolated`` is non-zero, ``use_environment`` and > ``user_site_directory`` are set to 0. > * If ``legacy_windows_fs_encoding`` is non-zero, ``utf8_mode`` is set to > 0. > * If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``, > ``faulthandler`` is set to 1, and ``"default"`` filter is added to > ``warnoptions``. But the ``PYTHONMALLOC`` environment variable has the > priority over ``dev_mode`` to set the memory allocator. > * If ``base_prefix`` is not set, it inherits ``prefix`` value. > * If ``base_exec_prefix`` is not set, it inherits ``exec_prefix`` value. > * If the ``python._pth`` configuration file is present, ``isolated`` is > set to 1 and ``site_import`` is set to 0; but ``site_import`` is set > to 1 if ``python._pth`` contains ``import site``. > > Rules on ``PyConfig`` and ``PyPreConfig`` parameters: > > * If ``PyPreConfig.legacy_windows_fs_encoding`` is non-zero, > set ``PyConfig.utf8_mode`` to 0, set ``PyConfig.filesystem_encoding`` > to ``mbcs``, and set ``PyConfig.filesystem_errors`` to ``replace``. > > Configuration Files > ------------------- > > Python configuration files: > > * ``pyvenv.cfg`` > * ``python._pth`` (Windows only) > * ``pybuilddir.txt`` (Unix only) > > Global Configuration Variables > ------------------------------ > > Global configuration variables mapped to ``PyPreConfig`` fields: > > ======================================== ================================ > Variable Field > ======================================== ================================ > ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) > ``Py_IsolatedFlag`` ``isolated`` > ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` > ``Py_UTF8Mode`` ``utf8_mode`` > ======================================== ================================ > > (NOT) means that the ``PyPreConfig`` value is the oposite of the global > configuration variable value. > > Global configuration variables mapped to ``PyConfig`` fields: > > ======================================== ================================ > Variable Field > ======================================== ================================ > ``Py_BytesWarningFlag`` ``bytes_warning`` > ``Py_DebugFlag`` ``parser_debug`` > ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` (NOT) > ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` > ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` > ``Py_FrozenFlag`` ``pathconfig_warnings`` (NOT) > ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` > ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` > ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) > ``Py_InspectFlag`` ``inspect`` > ``Py_InteractiveFlag`` ``interactive`` > ``Py_IsolatedFlag`` ``isolated`` > ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` > ``Py_NoSiteFlag`` ``site_import`` (NOT) > ``Py_NoUserSiteDirectory`` ``user_site_directory`` (NOT) > ``Py_OptimizeFlag`` ``optimization_level`` > ``Py_QuietFlag`` ``quiet`` > ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` (NOT) > ``Py_VerboseFlag`` ``verbose`` > ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` > ======================================== ================================ > > (NOT) means that the ``PyConfig`` value is the oposite of the global > configuration variable value. > > ``Py_LegacyWindowsFSEncodingFlag`` and ``Py_LegacyWindowsStdioFlag`` are > only available on Windows. > > Command Line Arguments > ---------------------- > > Usage:: > > python3 [options] > python3 [options] -c COMMAND > python3 [options] -m MODULE > python3 [options] SCRIPT > > > Command line options mapped to pseudo-action on ``PyPreConfig`` fields: > > ================================ ================================ > Option ``PyConfig`` field > ================================ ================================ > ``-E`` ``use_environment = 0`` > ``-I`` ``isolated = 1`` > ``-X dev`` ``dev_mode = 1`` > ``-X utf8`` ``utf8_mode = 1`` > ``-X utf8=VALUE`` ``utf8_mode = VALUE`` > ================================ ================================ > > Command line options mapped to pseudo-action on ``PyConfig`` fields: > > ================================ ================================ > Option ``PyConfig`` field > ================================ ================================ > ``-b`` ``bytes_warning++`` > ``-B`` ``write_bytecode = 0`` > ``-c COMMAND`` ``run_command = COMMAND`` > ``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE`` > ``-d`` ``parser_debug++`` > ``-E`` ``use_environment = 0`` > ``-i`` ``inspect++`` and ``interactive++`` > ``-I`` ``isolated = 1`` > ``-m MODULE`` ``run_module = MODULE`` > ``-O`` ``optimization_level++`` > ``-q`` ``quiet++`` > ``-R`` ``use_hash_seed = 0`` > ``-s`` ``user_site_directory = 0`` > ``-S`` ``site_import`` > ``-t`` ignored (kept for backwards compatibility) > ``-u`` ``buffered_stdio = 0`` > ``-v`` ``verbose++`` > ``-W WARNING`` add ``WARNING`` to ``warnoptions`` > ``-x`` ``skip_source_first_line = 1`` > ``-X OPTION`` add ``OPTION`` to ``xoptions`` > ================================ ================================ > > ``-h``, ``-?`` and ``-V`` options are handled without ``PyConfig``. > > -X Options > ---------- > > -X options mapped to pseudo-action on ``PyConfig`` fields: > > ================================ ================================ > Option ``PyConfig`` field > ================================ ================================ > ``-X dev`` ``dev_mode = 1`` > ``-X faulthandler`` ``faulthandler = 1`` > ``-X importtime`` ``import_time = 1`` > ``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX`` > ``-X showalloccount`` ``show_alloc_count = 1`` > ``-X showrefcount`` ``show_ref_count = 1`` > ``-X tracemalloc=N`` ``tracemalloc = N`` > ================================ ================================ > > Environment Variables > --------------------- > > Environment variables mapped to ``PyPreConfig`` fields: > > ================================= ============================================= > Variable ``PyPreConfig`` field > ================================= ============================================= > ``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn`` > ``PYTHONDEVMODE`` ``dev_mode`` > ``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding`` > ``PYTHONMALLOC`` ``allocator`` > ``PYTHONUTF8`` ``utf8_mode`` > ================================= ============================================= > > Environment variables mapped to ``PyConfig`` fields: > > ================================= ==================================== > Variable ``PyConfig`` field > ================================= ==================================== > ``PYTHONDEBUG`` ``parser_debug`` > ``PYTHONDEVMODE`` ``dev_mode`` > ``PYTHONDONTWRITEBYTECODE`` ``write_bytecode`` > ``PYTHONDUMPREFS`` ``dump_refs`` > ``PYTHONEXECUTABLE`` ``program_name`` > ``PYTHONFAULTHANDLER`` ``faulthandler`` > ``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed`` > ``PYTHONHOME`` ``home`` > ``PYTHONINSPECT`` ``inspect`` > ``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors`` > ``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio`` > ``PYTHONMALLOCSTATS`` ``malloc_stats`` > ``PYTHONNOUSERSITE`` ``user_site_directory`` > ``PYTHONOPTIMIZE`` ``optimization_level`` > ``PYTHONPATH`` ``module_search_path_env`` > ``PYTHONPROFILEIMPORTTIME`` ``import_time`` > ``PYTHONPYCACHEPREFIX,`` ``pycache_prefix`` > ``PYTHONTRACEMALLOC`` ``tracemalloc`` > ``PYTHONUNBUFFERED`` ``buffered_stdio`` > ``PYTHONVERBOSE`` ``verbose`` > ``PYTHONWARNINGS`` ``warnoptions`` > ================================= ==================================== > > ``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are > specific to Windows. > > > Annex: Python 3.7 API > ===================== > > Python 3.7 has 4 functions in its C API to initialize and finalize > Python: > > * ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python > * ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python > > Python 3.7 can be configured using `Global Configuration Variables`_, > `Environment Variables`_, and the following functions: > > * ``PyImport_AppendInittab()`` > * ``PyImport_ExtendInittab()`` > * ``PyMem_SetAllocator()`` > * ``PyMem_SetupDebugHooks()`` > * ``PyObject_SetArenaAllocator()`` > * ``Py_SetPath()`` > * ``Py_SetProgramName()`` > * ``Py_SetPythonHome()`` > * ``Py_SetStandardStreamEncoding()`` > * ``PySys_AddWarnOption()`` > * ``PySys_AddXOption()`` > * ``PySys_ResetWarnOptions()`` > > There is also a high-level ``Py_Main()`` function. > > > Python Issues > ============= > > Issues that will be fixed by this PEP, directly or indirectly: > > * `bpo-1195571 `_: "simple > callback system for Py_FatalError" > * `bpo-11320 `_: > "Usage of API method Py_SetPath causes errors in Py_Initialize() > (Posix ony)" > * `bpo-13533 `_: "Would like > Py_Initialize to play friendly with host app" > * `bpo-14956 `_: "custom PYTHONPATH > may break apps embedding Python" > * `bpo-19983 `_: "When interrupted > during startup, Python should not call abort() but exit()" > * `bpo-22213 `_: "Make pyvenv style > virtual environments easier to configure when embedding Python". This > PEP more or > * `bpo-22257 `_: "PEP 432: Redesign > the interpreter startup sequence" > * `bpo-29778 `_: "_Py_CheckPython3 > uses uninitialized dllpath when embedder sets module path with > Py_SetPath" > * `bpo-30560 `_: "Add > Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal > errors". > * `bpo-31745 `_: "Overloading > "Py_GetPath" does not work" > * `bpo-32573 `_: "All sys attributes > (.argv, ...) should exist in embedded environments". > * `bpo-34725 `_: > "Py_GetProgramFullPath() odd behaviour in Windows" > * `bpo-36204 `_: "Deprecate calling > Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?" > * `bpo-33135 `_: "Define field > prefixes for the various config structs". The PEP now defines well > how warnings options are handled. > > Issues of the PEP implementation: > > * `bpo-16961 `_: "No regression > tests for -E and individual environment vars" > * `bpo-20361 `_: "-W command line > options and PYTHONWARNINGS environmental variable should not override > -b / -bb command line options" > * `bpo-26122 `_: "Isolated mode > doesn't ignore PYTHONHASHSEED" > * `bpo-29818 `_: > "Py_SetStandardStreamEncoding leads to a memory error in debug mode" > * `bpo-31845 `_: > "PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect" > * `bpo-32030 `_: "PEP 432: Rewrite > Py_Main()" > * `bpo-32124 `_: "Document functions > safe to be called before Py_Initialize()" > * `bpo-33042 `_: "New 3.7 startup > sequence crashes PyInstaller" > * `bpo-33932 `_: "Calling > Py_Initialize() twice now triggers a fatal error (Python 3.7)" > * `bpo-34008 `_: "Do we support > calling Py_Main() after Py_Initialize()?" > * `bpo-34170 `_: "Py_Initialize(): > computing path configuration must not have side effect (PEP 432)" > * `bpo-34589 `_: "Py_Initialize() > and Py_Main() should not enable C locale coercion" > * `bpo-34639 `_: > "PYTHONCOERCECLOCALE is ignored when using -E or -I option" > * `bpo-36142 `_: "Add a new > _PyPreConfig step to Python initialization to setup memory allocator > and encodings" > * `bpo-36202 `_: "Calling > Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake" > * `bpo-36301 `_: "Add > _Py_PreInitialize() function" > * `bpo-36443 `_: "Disable > coerce_c_locale and utf8_mode by default in _PyPreConfig?" > * `bpo-36444 `_: "Python > initialization: remove _PyMainInterpreterConfig" > * `bpo-36471 `_: "PEP 432, PEP 587: > Add _Py_RunMain()" > * `bpo-36763 `_: "PEP 587: Rework > initialization API to prepare second version of the PEP" > * `bpo-36775 `_: "Rework filesystem > codec implementation" > * `bpo-36900 `_: "Use _PyCoreConfig > rather than global configuration variables" > > Issues related to this PEP: > > * `bpo-12598 `_: "Move sys variable > initialization from import.c to sysmodule.c" > * `bpo-15577 `_: "Real argc and argv > in embedded interpreter" > * `bpo-16202 `_: "sys.path[0] > security issues" > * `bpo-18309 `_: "Make python > slightly more relocatable" > * `bpo-25631 `_: "Segmentation fault > with invalid Unicode command-line arguments in embedded Python" > * `bpo-26007 `_: "Support embedding > the standard library in an executable" > * `bpo-31210 `_: "Can not import > modules if sys.prefix contains DELIM". > * `bpo-31349 `_: "Embedded > initialization ignores Py_SetProgramName()" > * `bpo-33919 `_: "Expose > _PyCoreConfig structure to Python" > * `bpo-35173 `_: "Re-use already > existing functionality to allow Python 2.7.x (both embedded and > standalone) to locate the module path according to the shared library" > > > Version History > =============== > > * Version 3: > > * ``PyConfig``: Add ``configure_c_stdio`` and ``parse_argv``, > rename ``_frozen`` to ``pathconfig_warnings``. > * Rename functions using bytes strings and wide character strings. For > example, ``Py_PreInitializeFromWideArgs`` becomes > ``Py_PreInitializeFromArgs``, and ``PyConfig_SetArgv`` becomes > ``PyConfig_SetBytesArgv``. > * Add ``PyWideStringList_Insert()`` function. > * New "Path configuration", "Isolate Python", "Python Issues" > and "Version History" sections. > * ``PyConfig_SetString()`` and ``PyConfig_SetBytesString()`` now > requires the configuration as the first argument. > * Rename ``Py_UnixMain()`` to ``Py_BytesMain()`` > > * Version 2: Add ``PyConfig`` methods (ex: ``PyConfig_Read()``), add > ``PyWideStringList_Append()``, rename ``PyWideCharList`` to > ``PyWideStringList``. > * Version 1: Initial version. > > Copyright > ========= > > This document has been placed in the public domain. > From storchaka at gmail.com Thu May 16 02:35:59 2019 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 16 May 2019 09:35:59 +0300 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: 16.05.19 04:23, Victor Stinner ????: > The first implementation of my API used sys.unraisablehook(exc_type, > exc_value, exc_tb, obj). The problem is that Serhiy Storchaka asked me > to add a new error message field which breaks the API: the API is not > future-proof. > > I modified my API to create an object to pack arguments. The new API > becomes sys.unraisablehook(unraisable) where unraisable has 4 fields: > exc_type, exc_value, exc_tb, obj. This API is now future-proof: adding > a new field will not break existing custom hooks! I prefer the former design, when the hook takes 5 arguments: exc_type, exc_value, exc_tb, obj and msg. Any additional human readable information can be encoded in msg, and machine readable information can be encoded in msg or obj. Currently we have no plans for adding more details, and I do not think that we will need to do this in future. Packing arguments into a single extendable object just complicates the code and increases the chance of raising an exception or crashing. Even obj and msg could be merged into a single object, and the hook could check whether it is a string or callable. But passing them as separate arguments is fine to me, I just think that no more complication is necessary. From vstinner at redhat.com Thu May 16 05:57:24 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 11:57:24 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: Le jeu. 16 mai 2019 ? 04:44, Nathaniel Smith a ?crit : > What happens if the hook raises an exception? Aha, thanks for asking the question! If there is a custom hook and the hook fails, the default hook logs the exception of the custom hook. Technically, even if the default hook fails, the default hook is called again to handle its own exception :-) The fallback uses a direct access to the C implementation of the default hook which reduces the risk of bugs. My implementation contains an unit test to ensure that if a custom hook raises an exception, it's a logged as expected ;-) Victor -- Night gathers, and now my watch begins. It shall not end until my death. From vstinner at redhat.com Thu May 16 06:12:20 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 12:12:20 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: Le jeu. 16 mai 2019 ? 08:39, Serhiy Storchaka a ?crit : > > 16.05.19 04:23, Victor Stinner ????: > > The first implementation of my API used sys.unraisablehook(exc_type, > > exc_value, exc_tb, obj). The problem is that Serhiy Storchaka asked me > > to add a new error message field which breaks the API: the API is not > > future-proof. > > > > I modified my API to create an object to pack arguments. The new API > > becomes sys.unraisablehook(unraisable) where unraisable has 4 fields: > > exc_type, exc_value, exc_tb, obj. This API is now future-proof: adding > > a new field will not break existing custom hooks! > > I prefer the former design, when the hook takes 5 arguments: exc_type, > exc_value, exc_tb, obj and msg. Any additional human readable > information can be encoded in msg, and machine readable information can > be encoded in msg or obj. Currently we have no plans for adding more > details, and I do not think that we will need to do this in future. > Packing arguments into a single extendable object just complicates the > code and increases the chance of raising an exception or crashing. > > Even obj and msg could be merged into a single object, and the hook > could check whether it is a string or callable. But passing them as > separate arguments is fine to me, I just think that no more complication > is necessary. I came to the UnraisableHookArgs structseq idea because of my bad experience with extension warnings "hooks". Let me tell you this story :-) To enhance ResourceWarning messages, I modified Python 3.6 to add a new "source" parameter to the warnings module. It's the object which caused the ResourceWarning. This object is mostly used to display the traceback where the object has been allocated, traceback read using tracemalloc.get_object_traceback() and so required tracemalloc to be enabled. https://bugs.python.org/issue26604 The problem is that the 2 warnings "hooks" use a fixed number of positional (and positional-or-keyword) parameters: def showwarning(message, category, filename, lineno, file=None, line=None): ... def formatwarning(message, category, filename, lineno, line=None): ... Adding a new parameter would simply break *all* custom hooks in the wild... At the same time, internally, the warnings module uses a WarningMessage object to "pack" all parameters into a single object. I was able to easily add a new "source" attribute to WarningMessage without breaking the code using WarningMessage. I added new hooks which accept WarningMessage: def _showwarnmsg(msg): ... def _formatwarnmsg(msg): ... The problem is to keep the backward compatibility: decide which hook should be called... How to detect that showwarning or _showwarnmsg has been overriden? If both are overriden, which one should be modified? The implementation is tricky, and it caused a few minor regressions: https://github.com/python/cpython/commit/be7c460fb50efe3b88a00281025d76acc62ad2fd All these problems are the consequence of the initial design choice of warnings hooks: using a fixed number of parameters. This API simply cannot be extended. IMHO it's a bad design, and we can do better: WarningMessage is a better design. -- I would prefer to not reproduce the same API design mistake with sys.unraisablehook. I modified my PR to only use 4 fields: (exc_type, exc_value, exc_tb, obj). I plan to add a 5th field "err_msg" ("error_message"?) later to enhance the hook (allow to pass a custom error message to give more context where the exception comes from). Passing a single UnraisableHookArgs parameter to sys.unraisablehook allows to add a 5th field without breaking the backward compatibility. > Any additional human readable > information can be encoded in msg, and machine readable information can > be encoded in msg or obj. IMHO it's very risky to declare the current API as "complete". In my experience, we always want to enhance an API at some point to pass more information. IMHO the warnings module with my new "source" parameter is a good example. For unraisable hook, it's not hard to imagine other useful parameters that can be passed to the hook to provide more context about the exception. For example, right now we only pass one object. But there are cases where a second object would totally makes sense. -- > Packing arguments into a single extendable object just complicates the > code and increases the chance of raising an exception or crashing. Technically, UnraisableHookArgs is basically a tuple of 4 items. I consider that there is a low risk that creating the object can fail. UnraisableHookArgs creation failure *is* covered by my implementation! The default hook is called directly in C without using a temporary UnraisableHookArgs object. The exception which caused the failure is logged as well. IMHO the very little risk of UnraisableHookArgs creation failure is worth it, compared to the pain to extend an API based on a fixed number of parameters. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From andrew.svetlov at gmail.com Thu May 16 06:39:28 2019 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Thu, 16 May 2019 13:39:28 +0300 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: After the detailed explanation, UnraisableHookArgs makes sense. Forward-compatibility is important thing On Thu, May 16, 2019 at 1:12 PM Victor Stinner wrote: > > Le jeu. 16 mai 2019 ? 08:39, Serhiy Storchaka a ?crit : > > > > 16.05.19 04:23, Victor Stinner ????: > > > The first implementation of my API used sys.unraisablehook(exc_type, > > > exc_value, exc_tb, obj). The problem is that Serhiy Storchaka asked me > > > to add a new error message field which breaks the API: the API is not > > > future-proof. > > > > > > I modified my API to create an object to pack arguments. The new API > > > becomes sys.unraisablehook(unraisable) where unraisable has 4 fields: > > > exc_type, exc_value, exc_tb, obj. This API is now future-proof: adding > > > a new field will not break existing custom hooks! > > > > I prefer the former design, when the hook takes 5 arguments: exc_type, > > exc_value, exc_tb, obj and msg. Any additional human readable > > information can be encoded in msg, and machine readable information can > > be encoded in msg or obj. Currently we have no plans for adding more > > details, and I do not think that we will need to do this in future. > > Packing arguments into a single extendable object just complicates the > > code and increases the chance of raising an exception or crashing. > > > > Even obj and msg could be merged into a single object, and the hook > > could check whether it is a string or callable. But passing them as > > separate arguments is fine to me, I just think that no more complication > > is necessary. > > I came to the UnraisableHookArgs structseq idea because of my bad > experience with extension warnings "hooks". Let me tell you this story > :-) > > To enhance ResourceWarning messages, I modified Python 3.6 to add a > new "source" parameter to the warnings module. It's the object which > caused the ResourceWarning. This object is mostly used to display the > traceback where the object has been allocated, traceback read using > tracemalloc.get_object_traceback() and so required tracemalloc to be > enabled. > > https://bugs.python.org/issue26604 > > The problem is that the 2 warnings "hooks" use a fixed number of > positional (and positional-or-keyword) parameters: > > def showwarning(message, category, filename, lineno, file=None, line=None): ... > def formatwarning(message, category, filename, lineno, line=None): ... > > Adding a new parameter would simply break *all* custom hooks in the wild... > > At the same time, internally, the warnings module uses a > WarningMessage object to "pack" all parameters into a single object. I > was able to easily add a new "source" attribute to WarningMessage > without breaking the code using WarningMessage. > > I added new hooks which accept WarningMessage: > > def _showwarnmsg(msg): ... > def _formatwarnmsg(msg): ... > > The problem is to keep the backward compatibility: decide which hook > should be called... How to detect that showwarning or _showwarnmsg has > been overriden? If both are overriden, which one should be modified? > > The implementation is tricky, and it caused a few minor regressions: > > https://github.com/python/cpython/commit/be7c460fb50efe3b88a00281025d76acc62ad2fd > > All these problems are the consequence of the initial design choice of > warnings hooks: using a fixed number of parameters. This API simply > cannot be extended. IMHO it's a bad design, and we can do better: > WarningMessage is a better design. > > -- > > I would prefer to not reproduce the same API design mistake with > sys.unraisablehook. > > I modified my PR to only use 4 fields: (exc_type, exc_value, exc_tb, obj). > > I plan to add a 5th field "err_msg" ("error_message"?) later to > enhance the hook (allow to pass a custom error message to give more > context where the exception comes from). Passing a single > UnraisableHookArgs parameter to sys.unraisablehook allows to add a 5th > field without breaking the backward compatibility. > > > Any additional human readable > > information can be encoded in msg, and machine readable information can > > be encoded in msg or obj. > > IMHO it's very risky to declare the current API as "complete". In my > experience, we always want to enhance an API at some point to pass > more information. IMHO the warnings module with my new "source" > parameter is a good example. > > For unraisable hook, it's not hard to imagine other useful parameters > that can be passed to the hook to provide more context about the > exception. For example, right now we only pass one object. But there > are cases where a second object would totally makes sense. > > -- > > > Packing arguments into a single extendable object just complicates the > > code and increases the chance of raising an exception or crashing. > > Technically, UnraisableHookArgs is basically a tuple of 4 items. I > consider that there is a low risk that creating the object can fail. > > UnraisableHookArgs creation failure *is* covered by my implementation! > The default hook is called directly in C without using a temporary > UnraisableHookArgs object. The exception which caused the failure is > logged as well. > > IMHO the very little risk of UnraisableHookArgs creation failure is > worth it, compared to the pain to extend an API based on a fixed > number of parameters. > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From storchaka at gmail.com Thu May 16 06:54:04 2019 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 16 May 2019 13:54:04 +0300 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: 16.05.19 13:12, Victor Stinner ????: > I came to the UnraisableHookArgs structseq idea because of my bad > experience with extension warnings "hooks". Let me tell you this story > :-) I know this story, because I followed these issues. > For unraisable hook, it's not hard to imagine other useful parameters > that can be passed to the hook to provide more context about the > exception. For example, right now we only pass one object. But there > are cases where a second object would totally makes sense. If you have plans for adding new details in future, I propose to add a 6th parameter "context" or "extra" (always None currently). It is as extensible as packing all arguments into a single structure, but you do not need to introduce the structure type and create its instance until you need to pass additional info. From solipsis at pitrou.net Thu May 16 07:31:00 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 May 2019 13:31:00 +0200 Subject: [Python-Dev] deprecation of abstractstaticmethod and abstractclassmethod References: <51c99452-91d3-a479-2666-1af554c433e2@stoneleaf.us> <571659b9-d63e-4bf9-ba06-56e7555b9aa8@python.org> Message-ID: <20190516133100.08acad67@fsol> On Wed, 15 May 2019 13:00:16 -0700 Steve Dower wrote: > On 15May2019 1248, Nathaniel Smith wrote: > > I don't care about the deprecation either way. But can we fix the > > individual decorators so both orders work? Even if it requires a special > > case in the code, it seems worthwhile to remove a subtle user-visible > > footgun. > > The classmethod and staticmethod objects already have a getter for > __isabstractmethod__ that forwards to the underlying object, so they > should just need a setter as well, right? Or, alternatively, raise an error if the wrong ordering is applied. Failing silently isn't very helpful. Regards Antoine. From vstinner at redhat.com Thu May 16 08:02:49 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 14:02:49 +0200 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> Message-ID: (Le jeu. 16 mai 2019 ? 06:34, Gregory Szorc a ?crit : > > I know that the PEP is long, but well, it's a complex topic, and I > > chose to add many examples to make the API easier to understand. > > I saw your request for feedback on Twitter a few days back and found > this thread. > > This PEP is of interest to me because I'm the maintainer of PyOxidizer - > a project for creating single file executables embedding Python. Aha, interesting :-) > As part > of hacking on PyOxidizer, I will admit to grumbling about the current > state of the configuration and initialization mechanisms. The reliance > on global variables and the haphazard way in which you must call certain > functions before others was definitely a bit frustrating to deal with. Yeah, that's what I tried to explain in the PEP 587 Rationale. > My most important piece of feedback is: thank you for tackling this! > Your work to shore up the inner workings of interpreter state and > management is a big deal on multiple dimensions. I send my sincere > gratitude. You're welcome ;-) > PyPreConfig_INIT and PyConfig_INIT as macros that return a struct feel > weird to me. Specifically, the `PyPreConfig preconfig = > PyPreConfig_INIT;` pattern doesn't feel right. I'm sort of OK with these > being implemented as macros. But I think they should look like function > calls so the door is open to converting them to function calls in the > future. Ah yes, I noticed that some projects can only import symbols, not use directly the C API. You're right that such macro can be an issue. Would you be ok with a "PyConfig_Init(PyConfig *config);" function which would initialize all fields to theire default values? Maybe PyConfig_INIT should be renamed to PyConfig_STATIC_INIT. You can find a similar API for pthread mutex, there is a init function *and* a macro for static initialization: int pthread_mutex_init(pthread_mutex_t *restrict mutex, const pthread_mutexattr_t *restrict attr); pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; > PyPreConfig.allocator being a char* seems a bit weird. Does this imply > having to use strcmp() to determine which allocator to use? Perhaps the > allocator setting should be an int mapping to a constant instead? Yes, _PyMem_SetupAllocators() uses strcmp(). There are 6 supported values: * "default" * "debug" * "pymalloc" * "pymalloc_debug" * "malloc" * "malloc_debug" Note: pymalloc and pymalloc_debug are not supported if Python is explicitly configure using --without-pymalloc. I think that I chose to use string because the feature was first implemented using an environment variable. Actually, I *like* the idea of avoiding string in PyPreConfig because a string might need memory allocation, whereas the pre-initialization is supposed to configure memory allocation :-) I will change the type to an enum. > Relatedly, how are custom allocators registered? e.g. from Rust, I want > to use Rust's allocator. How would I do that in this API? Do I still > need to call PyMem_SetAllocator()? By default, PyPreConfig.allocator is set to NULL. In that case, _PyPreConfig_Write() leaves the memory allocator unmodified. As PyImport_AppendInittab() and PyImport_ExtendInittab(), PyMem_SetAllocator() remains relevant and continue to work as previously. Example to set your custom allocator: --- PyInitError err = Py_PreInitialize(NULL); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } PyMem_SetAllocator(PYMEM_DOMAIN_MEM, my_cool_allocator); --- Well, it also works in the opposite order, but I prefer to call PyMem_SetAllocator() after the pre-initialization to make it more explicit :-) --- PyMem_SetAllocator(PYMEM_DOMAIN_MEM, my_cool_allocator); PyInitError err = Py_PreInitialize(NULL); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } --- > I thought a point of this proposal > was to consolidate per-interpreter config settings? Right. But PyMem_SetAllocator() uses PyMemAllocatorDomain enum and PyMemAllocatorEx structure which are not really "future-proof". For example, I already replaced PyMemAllocator with PyMemAllocatorEx to add "calloc". We might extend it later one more time to add allocator with a specific memory alignement (even if the issue is now closed): https://bugs.python.org/issue18835 I consider that PyMem_SetAllocator() is too specific to be added to PyPreConfig. Are you fine with that? > I'm a little confused about the pre-initialization functions that take > command arguments. Is this intended to only be used for parsing the > arguments that `python` recognizes? Presumably a custom application > embedding Python would never use these APIs unless it wants to emulate > the behavior of `python`? (I suppose this can be clarified in the API > docs once this is implemented.) Yes, Py_PreInitializeFromArgs() parses -E, -I, -X dev and -X utf8 options: https://www.python.org/dev/peps/pep-0587/#command-line-arguments Extract of my "Isolate Python" section: "The default configuration is designed to behave as a regular Python. To embed Python into an application, it's possible to tune the configuration to better isolated the embedded Python from the system: (...)" https://www.python.org/dev/peps/pep-0587/#isolate-python I wasn't sure if I should mention parse_argv=0 in this section or not. According to what you wrote, I should :-) *Maybe* rather than documenting how to isolate Python, we might even provide a function for that? void PyConfig_Isolate(PyConfig *config) { config->isolated = 1; config->parse_argv = 0; } I didn't propose that because so far, I'm not sure that everybody has the same opinion on what "isolation" means. Does it only mean ignore environment variables? Or also ignore configuration files? What about the path configuration? That's why I propose to start without such opiniated PyConfig_Isolate() function :-) > What about PyImport_FrozenModules? This is a global variable related to > Python initialization (it contains _frozen_importlib and > _frozen_importlib_external) but it is not accounted for in the PEP. > I rely on this struct in PyOxidizer to replace the importlib modules with > custom versions so we can do 0-copy in-memory import of Python bytecode > for the entirety of the standard library. Should the PyConfig have a > reference to the _frozen[] to use? Should the _frozen struct be made > part of the public API? First of all, PEP 587 is designed to be easily extendable :-) I added _config_version field to even provide backward ABI compatibility. Honestly, I never looked at PyImport_FrozenModules. It seems to fall into the same category than "importtab": kind of corner case use case which cannot be easily generalized into PyConfig structure. As I would say the same that what I wrote about PyImport_AppendInittab(): PyImport_FrozenModules symbol remains relevant and continue to work as expected. I understand that it must be set before the initialization, and it seems safe to set it even before the pre-initialization since it's a static array. Note: I renamed PyConfig._frozen to PyConfig.pathconfig_warnings: it's an int and it's unrelated to PyImport_FrozenModules. > I rely on this struct in PyOxidizer to replace the importlib modules with > custom versions so we can do 0-copy in-memory import of Python bytecode > for the entirety of the standard library. Wait, that sounds like a cool feature! Would it make sense to make this feature upstream? If yes, maybe send a separated email to python-dev and/or open an issue. > The PEP mentions a private PyConfig._install_importlib member. I'm > curious what this is because it may be relevant to PyOxidizer. FWIW I > /might/ be interested in a mechanism to better control importlib > initialization because PyOxidizer is currently doing dirty things at > run-time to register the custom 0-copy meta path importer. I /think/ my > desired API would be a mechanism to control the name(s) of the frozen > module(s) to use to bootstrap importlib. Or there would be a way to > register the names of additional frozen modules to import and run as > part of initializing importlib (before any .py-based stdlib modules are > imported). Then PyOxidizer wouldn't need to hack up the source code to > importlib, compile custom bytecode, and inject it via > PyImport_FrozenModules. I concede this may be out of scope for the PEP. > But if the API is being reworked, I'd certainly welcome making it easier > for tools like PyOxidizer to work their crazy module importing magic :) PEP 587 is an incomplete implementation of the PEP 432. We are discussing with Nick Coghlan, Steve Dower and some others about having 2 phases for the Python initialization: "core" and "main". The "core" phase would provide a bare minimum working Python: builtin exceptions and types, maybe builtin imports, and that's basically all. It would allow to configure Python using the newly created interpreter, for example configure Python by running Python code. The problem is that these 2 phases are not well defined yet, it's still under discussion. Nick and me agreed to start with PEP 587 as a first milestone, and see later how to implement "core" and "main" phases. If the private field "_init_main" of the PEP 587 is set to 0, Py_InitializeFromConfig() stops at the "core" phase (in fact, it's already implemented!). But I didn't implement yet a _Py_InitializeMain() function to "finish" the initialization. Let's say that it exists, we would get: --- PyConfig config = PyConfig_INIT; config._init_main = 0; PyInitError err = Py_InitializeFromConfig(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } /* add your code to customize Python here */ /* calling PyRun_SimpleString() here is safe */ /* finish Python initialization */ PyInitError err = _Py_InitializeMain(&config); if (Py_INIT_FAILED(err)) { Py_ExitInitError(err); } --- Would it solve your use case? Sorry, I didn't understand properly what you mean by "controlling the names of the frozen modules to use to bootstrap importlib". > I really like the new Py_RunMain() API and associated PyConfig members. > I also invented this wheel in PyOxidizer and the new API should result > in me deleting some code that I wish I didn't have to write in the first > place :) Great! > I invented a data structure for representing a Python interpreter > configuration. And the similarities to PyConfig are striking. I think > that's a good sign :) He he :-) > It might be useful to read through that file - > especially the init function (line with `pub fn init`) to see if > anything I'm doing pushes the boundaries of the proposed API. Feel free > to file GitHub issues if you see obvious bugs with PyOxidizer's Python > initialization logic while you're at it :) Your link didn't work, but I found: https://github.com/indygreg/PyOxidizer/blob/master/pyoxidizer/src/pyembed/pyinterp.rs "write_modules_directory_env" seems very specific to your needs. Apart of that, I confirm that PythonConfig is very close to PEP 587 PyConfig! I notice that you also avoided double negation, thanks ;-) /* Pre-initialization functions we could support: * * PyObject_SetArenaAllocator() * PySys_AddWarnOption() * PySys_AddXOption() * PySys_ResetWarnOptions() */ Apart PyObject_SetArenaAllocator(), PyConfig implements the 3 other functions. Again, ss PyMem_SetAllocator(), PyObject_SetArenaAllocator() remains relevant and can be used with the pre-initialization. PySys_SetObject("argv", obj) is covered by PyConfig.argv. PySys_SetObject("argvb", obj): I'm not sure why you are doing that, it's easy to retrieve sys.argv as bytes, it's now even documented: https://docs.python.org/dev/library/sys.html#sys.argv --- Sorry, I'm not an importlib expert. I'm not sure what could be done in PEP 587 for your specific importlib changes. > Also, one thing that tripped me up a few times when writing PyOxidizer > was managing the lifetimes of memory that various global variables point > to. The short version is I was setting Python globals to point to memory > allocated by Rust and I managed to crash Python by freeing memory before > it should have been. Since the new API seems to preserve support for > global variables, I'm curious if there are changes to how memory must be > managed. It would be really nice to get to a state where you only need > to ensure the PyConfig instance and all its referenced memory only needs > to outlive the interpreter it configures. That would make the memory > lifetimes story intuitive and easily compatible with Rust. For the specific case of PyConfig, you have to call PyConfig_Clear(config) after you called Py_InitializeFromConfig(). Python keeps a copy of your configuration (and it completes the missing fields, if needed). I modified a lot of functions to ensure that Python cleanups more globals at exit in Py_Finalize() and at the end of Py_Main() / Py_RunMain(). I'm not sure if it replies to your question. If you want a more specific, can you please give more concrete examples of globals? There is also an on-going refactoring to move globals into _PyRuntimeState and PyInterpreterState: change needed to support subinterpreters, see Eric Snow's PEP 554. > One feature that I think is missing from the proposal (and this is > related to the previous paragraph) is the ability to prevent config > fallback to things that aren't PyConfig and PyPreConfig. There is > `PyConfig.parse_argv` to disable command argument parsing and > `PyConfig.use_environment` to disable environment variable fallback. But > AFAICT there is no option to disable configuration file fallback nor > global variable fallback. If you embed Python, you control global configuration variables, no? I chose to design PyConfig to inherit global configuration variables because it allows to support both ways to configure Python using a single implementation. Would you prefer an explicit PyConfig_SetDefaults(config) which would completely ignore global configuration variables? See Lib/test/test_embed.py unit tests which uses Programs/_testembed.c: https://github.com/python/cpython/blob/master/Programs/_testembed.c python._pth (Windows only), pybuilddir.txt (Unix only) and pyvenv.cfg configuration files are only used by the function building the "Path Configuration". Using PEP 587, you can now completely ignore this function: https://www.python.org/dev/peps/pep-0587/#path-configuration > Again, this proposal is terrific overall and so much better than what we > have today. The wall of text I just wrote is disproportionate in size to > the quality of the PEP. I almost feel bad writing so much feedback for > such a terrific PEP ;) > > Excellent work, Victor. I can't wait to see these changes materialize! Thanks :-) Thanks for your very interesting feedback. It's really helpful to see how the API is used "for real" :-) Victor From p.f.moore at gmail.com Thu May 16 09:30:37 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 16 May 2019 14:30:37 +0100 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> Message-ID: On Thu, 16 May 2019 at 13:05, Victor Stinner wrote: > > PyPreConfig_INIT and PyConfig_INIT as macros that return a struct feel > > weird to me. Specifically, the `PyPreConfig preconfig = > > PyPreConfig_INIT;` pattern doesn't feel right. I'm sort of OK with these > > being implemented as macros. But I think they should look like function > > calls so the door is open to converting them to function calls in the > > future. > > Ah yes, I noticed that some projects can only import symbols, not use > directly the C API. You're right that such macro can be an issue. I've not been following this PEP particularly, but I can confirm that the Vim bindings for Python also have this restriction (at least on Windows). To allow binding to the Python interpreter at runtime, and only on demand, the interface does an explicit LoadLibrary/GetProcAddress call for each C API function that's used. That means macros are unavailable (short of wholesale copying of the Python headers). (It's also a painfully laborious bit of code, and it would be nice if there were a better way of doing it, but I've never had the time/motivation to try to improve this, so that's just how it's stayed). Paul From vstinner at redhat.com Thu May 16 09:47:22 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 15:47:22 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: Le jeu. 16 mai 2019 ? 12:57, Serhiy Storchaka a ?crit : > > For unraisable hook, it's not hard to imagine other useful parameters > > that can be passed to the hook to provide more context about the > > exception. For example, right now we only pass one object. But there > > are cases where a second object would totally makes sense. > > If you have plans for adding new details in future, I propose to add a > 6th parameter "context" or "extra" (always None currently). It is as > extensible as packing all arguments into a single structure, but you do > not need to introduce the structure type and create its instance until > you need to pass additional info. In my implementation, UnraisableHookArgs is a C "structseq" type. In short, it's a tuple. make_unraisable_hook_args() basically builds a tuple of 4 items and uses PyTuple_SET_ITEM() to set the items. _PyErr_WriteUnraisableDefaultHook() uses PyStructSequence_GET_ITEM() to get items. The code pack and unpack UnraisableHookArgs is simple and reliable. An "unraisable exception" doesn't mean that Python is collapsing. It only means that the code is unable to report the exception to the caller: there is no reason why allocating a 4-tuple or calling a simple Python function (sys.unraisablehook) would fail. -- I dislike the compromise of having an API in 2 parts: positional parameters for some parameters, and a structure for some other parameters. I prefer to pack all arguments into a single structure. -- IMHO it's readable to get attributes from an object in a Python hook: it doesn't surprised me, OOP is common in Python :-) Simplified example of a pure Python reimplementation of the default hook: def myhook(unraisable): if unraisable.obj is not None: print(f"Exception ignored in: {unraisable.obj!r}") if unraisable.exc_tb is not None: traceback.print_tb(unraisable.exc_tb) print(f"{unraisable.exc_type.__name__}: {unraisable.exc_value}") Compared to positional arguments: def myhook(exc_type, exc_value, exc_tb, obj, extras): if obj is not None: print(f"Exception ignored in: {obj!r}") if exc_tb is not None: print_tb(exc_tb, file=file) print(f"{exc_type.__name__}: {exc_value}") Again, the warnings module uses a similar WarningsMsg structure and I'm not shocked by the implementation. Simplified code from Lib/warnings.py: def _formatwarnmsg_impl(msg): category = msg.category.__name__ s = f"{msg.filename}:{msg.lineno}: {category}: {msg.message}\n" if msg.line is None: line = linecache.getline(msg.filename, msg.lineno) else: line = msg.line if line: s += " %s\n" % line.strip() if msg.source is not None: ... return s Victor -- Night gathers, and now my watch begins. It shall not end until my death. From thomas at python.org Thu May 16 10:10:06 2019 From: thomas at python.org (Thomas Wouters) Date: Thu, 16 May 2019 16:10:06 +0200 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> Message-ID: On Thu, May 16, 2019 at 2:03 PM Victor Stinner wrote: > (Le jeu. 16 mai 2019 ? 06:34, Gregory Szorc a > ?crit : > > > I know that the PEP is long, but well, it's a complex topic, and I > > > chose to add many examples to make the API easier to understand. > > > > I saw your request for feedback on Twitter a few days back and found > > this thread. > > > > This PEP is of interest to me because I'm the maintainer of PyOxidizer - > > a project for creating single file executables embedding Python. > > Aha, interesting :-) > Just for some context to everyone: Gregory's PyOxidizer is very similar to Hermetic Python, the thing we use at Google for all Python programs in our mono-repo. We had a short email discussion facilitated by Augie Fackler, who wants to use PyOxidizer for Mercurial, about how Hermetic Python works. At the PyCon sprints last week, I sat down with Victor, Steve Dower and Eric Snow, showing them how Hermetic Python embeds CPython, and what hoops it has to jump through and what issues we encountered. I think most of those issues would also apply to PyOxidizer, lthough it sounds like Gregory solved some of the issues a bit differently. (Hermetic Python was originally written for Python 2.7, so it doesn't try to deal with importlib's bootstrapping, for example.) I have some comments and questions about the PEP as well, some of which overlap with Gregory's or Victor's answers: [...] > > PyPreConfig_INIT and PyConfig_INIT as macros that return a struct feel > > weird to me. Specifically, the `PyPreConfig preconfig = > > PyPreConfig_INIT;` pattern doesn't feel right. I'm sort of OK with these > > being implemented as macros. But I think they should look like function > > calls so the door is open to converting them to function calls in the > > future. > > Ah yes, I noticed that some projects can only import symbols, not use > directly the C API. You're right that such macro can be an issue. > > Would you be ok with a "PyConfig_Init(PyConfig *config);" function > which would initialize all fields to theire default values? Maybe > PyConfig_INIT should be renamed to PyConfig_STATIC_INIT. > > You can find a similar API for pthread mutex, there is a init function > *and* a macro for static initialization: > > int pthread_mutex_init(pthread_mutex_t *restrict mutex, > const pthread_mutexattr_t *restrict attr); > > pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; > This was going to be my suggestion as well: for any non-trivial macro, we should have a function for it instead. I would also point out that PEP 587 has a code example that uses PyWideStringList_INIT, but that macro isn't mention anywhere else. The PEP is a bit unclear as to the semantics of PyWideStringList as a whole: the example uses a static array with length, but doesn't explain what would happen with statically allocated data like that if you call the Append or Extend functions. It also doesn't cover how e.g. argv parsing would remove items from the list. (I would also suggest the PEP shouldn't use the term 'list', at least not unqualified, if it isn't an actual Python list.) I understand the desire to make static allocation and initialisation possible, but since you only need PyWideStringList for PyConfig, not PyPreConfig (which sets the allocator), perhaps having a PyWideStringList_Init(), which copies memory, and PyWideStringList_Clear() to clear it, would be better? > What about PyImport_FrozenModules? This is a global variable related to > > Python initialization (it contains _frozen_importlib and > > _frozen_importlib_external) but it is not accounted for in the PEP. > > I rely on this struct in PyOxidizer to replace the importlib modules with > > custom versions so we can do 0-copy in-memory import of Python bytecode > > for the entirety of the standard library. Should the PyConfig have a > > reference to the _frozen[] to use? Should the _frozen struct be made > > part of the public API? > > > First of all, PEP 587 is designed to be easily extendable :-) I added > _config_version field to even provide backward ABI compatibility. > > Honestly, I never looked at PyImport_FrozenModules. It seems to fall > into the same category than "importtab": kind of corner case use case > which cannot be easily generalized into PyConfig structure. > > As I would say the same that what I wrote about > PyImport_AppendInittab(): PyImport_FrozenModules symbol remains > relevant and continue to work as expected. I understand that it must > be set before the initialization, and it seems safe to set it even > before the pre-initialization since it's a static array. > > Note: I renamed PyConfig._frozen to PyConfig.pathconfig_warnings: it's > an int and it's unrelated to PyImport_FrozenModules. > > > I rely on this struct in PyOxidizer to replace the importlib modules with > > custom versions so we can do 0-copy in-memory import of Python bytecode > > for the entirety of the standard library. > > Wait, that sounds like a cool feature! Would it make sense to make > this feature upstream? If yes, maybe send a separated email to > python-dev and/or open an issue. > > The PEP mentions a private PyConfig._install_importlib member. I'm > > curious what this is because it may be relevant to PyOxidizer. FWIW I > > /might/ be interested in a mechanism to better control importlib > > initialization because PyOxidizer is currently doing dirty things at > > run-time to register the custom 0-copy meta path importer. I /think/ my > > desired API would be a mechanism to control the name(s) of the frozen > > module(s) to use to bootstrap importlib. Or there would be a way to > > register the names of additional frozen modules to import and run as > > part of initializing importlib (before any .py-based stdlib modules are > > imported). Then PyOxidizer wouldn't need to hack up the source code to > > importlib, compile custom bytecode, and inject it via > > PyImport_FrozenModules. I concede this may be out of scope for the PEP. > > But if the API is being reworked, I'd certainly welcome making it easier > > for tools like PyOxidizer to work their crazy module importing magic :) > > PEP 587 is an incomplete implementation of the PEP 432. We are > discussing with Nick Coghlan, Steve Dower and some others about having > 2 phases for the Python initialization: "core" and "main". The "core" > phase would provide a bare minimum working Python: builtin exceptions > and types, maybe builtin imports, and that's basically all. It would > allow to configure Python using the newly created interpreter, for > example configure Python by running Python code. > > The problem is that these 2 phases are not well defined yet, it's > still under discussion. Nick and me agreed to start with PEP 587 as a > first milestone, and see later how to implement "core" and "main" > phases. > > If the private field "_init_main" of the PEP 587 is set to 0, > Py_InitializeFromConfig() stops at the "core" phase (in fact, it's > already implemented!). But I didn't implement yet a > _Py_InitializeMain() function to "finish" the initialization. Let's > say that it exists, we would get: > > --- > PyConfig config = PyConfig_INIT; > config._init_main = 0; > PyInitError err = Py_InitializeFromConfig(&config); > if (Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > > /* add your code to customize Python here */ > /* calling PyRun_SimpleString() here is safe */ > > /* finish Python initialization */ > PyInitError err = _Py_InitializeMain(&config); > if (Py_INIT_FAILED(err)) { > Py_ExitInitError(err); > } > --- > > Would it solve your use case? > FWIW, I understand the need here: for Hermetic Python, we solved it by adding a new API similar to PyImport_AppendInittab, but instead registering a generic callback hook to be called *during* the initialisation process: after the base runtime and the import mechanism are initialised (at which point you can create Python objects), but before *any* modules are imported. We use that callback to insert a meta-importer that satisfies all stdlib imports from an embedded archive. (Using a meta-importer allows us to bypass the fileysystem altogether, even for what would otherwise be failed path lookups.) As I mentioned, Hermetic Python was originally written for Python 2.7, but this approach works fine with a frozen importlib as well. The idea of 'core' and 'main' initialisation will likely work for this, as well. Other questions/comments about PEP 587: I really like the PyInitError struct. I would like more functions to use it, e.g. the PyRrun_* "very high level" API, which currently calls exit() for you on SystemExit, and returns -1 without any other information on error. For those, I'm not entirely sure 'Init' makes sense in the name... but I can live with it. A couple of things are documented as performing pre-initialisation (PyConfig_SetBytesString, PyConfig_SetBytesArgv). I understand why, but I feel like that might be confusing and error-prone. Would it not be better to have them fail if pre-initialisation hasn't been performed yet? The buffered_stdio field of PyConfig mentions stdout and stderr, but not stdin. Does it not affect stdin? (Many of the fields could do with a bit more explicit documentation, to be honest.) The configure_c_stdio field of PyConfig sounds like it might not set sys.stdin/stdout/stderr. That would be new behaviour, but configure_c_stdio doesn't have an existing equivalence, so I'm not sure if that's what you meant or not. The dll_path field of PyConfig says "Windows only". Does that meant the struct doesn't have that field except in a Windows build? Or is it ignored, instead? If it doesn't have that field at all, what #define can be used to determine if the PyConfig struct will have it or not? It feels a bit weird to have both 'inspect' and 'interactive' in PyConfig. Is there a substantive difference between them? Is this just so you can easily tell if any of run_module / run_command / run_filename are set? "module_search_path_env" sounds like an awkward and somewhat misleading name for the translation of PYTHONPATH. Can we not just use, say, pythonpath_env? I expect the intended audience to know that PYTHONPATH != sys.path. The module_search_paths field in PyConfig doesn't mention if it's setting or adding to the calculated sys.path. As a whole, the path-calculation bits are a bit under-documented. Since this is an awkward bit of CPython, it wouldn't hurt to mention what "the default path configuration" does (i.e. search for python's home starting at program_name, add fixed subdirs to it, etc.) Path configuration is mentioned as being able to issue warnings, but it doesn't mention *how*. It can't be the warnings module at this stage. I presume it's just printing to stderr. Regarding Py_RunMain(): does it do the right thing when something calls PyErr_Print() with SystemExit set? (I mentioned last week that PyErr_Print() will call C's exit() in that case, which is obviously terrible for embedders.) Regarding isolated_mode and the site module, should we make stronger guarantees about site.py's behaviour being optional? The problem with site is that it does four things that aren't configurable, one of which is usually very desirable, one of which probably doesn't matter to embedders, and two that are iffy: sys.path deduplication and canonicalisation (and fixing up __file__/__cached__ attributes of already-imported modules); adding site-packages directories; looking for and importing sitecustomize.py; executing .pth files. The site module doesn't easily allow doing only some of these. (user-site directories are an exception, as they have their own flag, so I'm not listing that here.) With Hermetic Python we don't care about any of these (for a variety of different reasons), but I'm always a little worried that future Python versions would add behaviour to site that we *do* need. (As a side note, here's an issue I forgot to talk about last week: with Hermetic Python's meta-importers we have an ancillary regular import hook for correctly dealing with packages with modified __path__, so that for example 'xml' from the embedded stdlib zip can still import '_xmlplus' from the filesystem or a separate zip, and append its __path__ entries to its own. To do that, we use a special prefix to use for the embedded archive meta-importers; we don't want to use a file because they are not files on disk. The prefixes used to be something like ''. This works fine, and with correct ordering of import hooks nothing will try to find files named ''... until user code imports site for some reason, which then canonicalises sys.path, replacing the magic prefixes with '/path/to/cwd/'. We've since made the magic prefixes start with /, but I'm not happy with it :P) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu May 16 11:23:08 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 08:23:08 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> Message-ID: <57ab48b9-fd1a-9300-d28b-8c3164b1d583@python.org> Thanks for adding your input, Gregory! It's much appreciated. I'll shuffle your comments around a bit, as I'd rather address the themes than each individual point. On 15May2019 2134, Gregory Szorc wrote: > PyPreConfig_INIT and PyConfig_INIT as macros that return a struct feel > weird to me. Specifically, the `PyPreConfig preconfig = > PyPreConfig_INIT;` pattern doesn't feel right. I see Victor agreed here, but I think this is the right pattern for PreConfig. The "_INIT" macro pattern is common throughout as a way to initialize a stack-allocated struct - we really can't change it to be anything than "{ .member = static value }" without breaking users, but if you have another way to initialize it correctly then that is fine. The important factor here is that this struct has to be allocated _without_ any memory management provided by Python. That said, I don't particularly like this approach for PyConfig. As you said: > Also, one thing that tripped me up a few times when writing PyOxidizer > was managing the lifetimes of memory that various global variables point > to. My preference here is for PreConfig to get far enough that we can construct the full configuration as regular Python objects (e.g. using PyDict_New, PyUnicode_FromString, etc.)[1] rather than a brand new C struct with a new set of functions. That will cost copies/allocations at startup, but it will also ensure that the lifetime of the configuration info is managed by the runtime. I assume you already have code/helpers for constructing Python strings and basic data structures, so I wonder whether it would be helpful to be able to use them to create the configuration info? ([1]: Yes, this requires implementation changes so they work pre-interpreter and cross-interpreter. This work probably has to happen anyway, so I don't see any harm in assuming it will happen.) > I'm a little confused about the pre-initialization functions that take > command arguments. Is this intended to only be used for parsing the > arguments that `python` recognizes? Presumably a custom application > embedding Python would never use these APIs unless it wants to emulate > the behavior of `python`? (I suppose this can be clarified in the API > docs once this is implemented.) > One feature that I think is missing from the proposal (and this is > related to the previous paragraph) is the ability to prevent config > fallback to things that aren't PyConfig and PyPreConfig. This is certainly my intent, and I *think* Victor is coming around to it too ;) My preference is that embedding by default does not use any information outside of the configuration provided by the host application. Then our "python" host application can read the environment/argv/etc. and convert it into configuration. Since some embedders also want to do this, we can provide helper functions to replicate the behaviour. Does this sound like a balance that would suit your needs? Would you expect an embedded Python to be isolated by default? Or would you assume that it's going to pick up configuration from various places and that you (as an embedder) need to explicitly suppress that? (Some parts of the stdlib and some 3rd-party libraries use their own environment variables at runtime. We may not be able to convert all of those to configuration, but at least they're read lazily, and so Python code can override them by setting the variables.) > What about PyImport_FrozenModules? > FWIW I /might/ be interested in a mechanism to better control importlib > initialization because PyOxidizer is currently doing dirty things at > run-time to register the custom 0-copy meta path importer. Controlling imports early in initialization is one of our broader goals here, and one that I would particularly like to figure out before we commit to a new public API. Registering new importers should not have to be "dirty". Cheers, Steve From steve.dower at python.org Thu May 16 11:33:19 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 08:33:19 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: References: Message-ID: On 15May2019 1610, Victor Stinner wrote: > Thanks to the constructive discussions, I enhanced my PEP 587. I don't > plan any further change, the PEP is now ready to review (and maybe > even for pronouncement, hi Thomas! :-)). My view is that while this is a fantastic PEP and the groundwork that already exists as private API is excellent, it is too early to commit to a new public API and we need to do more analysis work. We should not accept this PEP at this time. So far, the API being exposed here has not been tested with embedders. We have very little feedback on whether it meets their needs or would help them simplify or make their projects more robust. I have concerns about the number of new functions being added, the new patterns being proposed, and both forward and backwards compatibility as we inevitably make changes. (I have discussed all of these in person with Victor, Nick, and Thomas at PyCon last week, which is why I'm not doing a point-by-point here.) As soon as we publish a PEP declaring a new embedding API, users will assume that it's here to stay. I don't believe that is true, as there is much more we can and should do to improve embedding. But we don't get to totally revise the public API on each release without alienating users, which is why I would rather hold the public API changes until 3.9, investigate and design them properly. It does our users a disservice to make major changes like this without due process. Cheers, Steve From steve.dower at python.org Thu May 16 11:45:11 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 08:45:11 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> Message-ID: <97a82b7a-0410-be1d-cda4-88fc4f6ec1b3@python.org> On 16May2019 0710, Thomas Wouters wrote: > A couple of things are documented as performing pre-initialisation > (PyConfig_SetBytesString,?PyConfig_SetBytesArgv). I understand why, but > I feel like that might be confusing and error-prone. Would it not be > better to have them fail if pre-initialisation hasn't been performed yet? I agree. Anything other than setting up the struct for pre-initialization settings doesn't need to work here. > The dll_path field of PyConfig says "Windows only". Does that meant the > struct doesn't have that field except in a Windows build? Or is it > ignored, instead? If it doesn't have that field at all, what #define can > be used to determine if the PyConfig struct will have it or not? This field doesn't need to be here. It exists because it was used in getpathp.c, and Victor's internal refactoring has kept it around through all the field movement. If we properly design initialization instead of just refactoring until it's public, I bet this field will go away. > "module_search_path_env" sounds like an awkward and somewhat misleading > name for the translation of PYTHONPATH. Can we not just use, say, > pythonpath_env? I expect the intended audience to know that PYTHONPATH > != sys.path. Again, this doesn't need to be its own configuration field, but because of the refactoring approach taken here it's flowed out to public API. A "init config from environment" can load this value and put it into the "sys.path-equivalent field" in the config. > The module_search_paths field in PyConfig doesn't mention if it's > setting or adding to the calculated sys.path. As a whole, the > path-calculation bits are a bit under-documented. Since this is an > awkward bit of CPython, it wouldn't hurt to mention what "the default > path configuration" does (i.e. search for python's home starting at > program_name, add fixed subdirs to it, etc.) Again, let's design this part properly instead of exposing what we've had for years :) > Regarding Py_RunMain(): does it do the right thing when something calls > PyErr_Print() with SystemExit set? (I mentioned last week that > PyErr_Print() will call C's exit() in that case, which is obviously > terrible for embedders.) Can we just fix PyErr_Print() to not exit? Surely we only depend on it in one or two places (sys.excepthook?) and it's almost certainly not helping anyone else. > Regarding isolated_mode and the site module, should we make stronger > guarantees about site.py's behaviour being optional? Yes, I've been forgetting about this too. There's a lot of configuration that's split between site.py and initialization, so it's very hard to understand what will be ready when you leave out site.py. Straightening this out would help (everyone except virtualenv, probably ;) ) Cheers, Steve From steve.dower at python.org Thu May 16 11:56:22 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 08:56:22 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> On 16May2019 0647, Victor Stinner wrote: > Le jeu. 16 mai 2019 ? 12:57, Serhiy Storchaka a ?crit : >>> For unraisable hook, it's not hard to imagine other useful parameters >>> that can be passed to the hook to provide more context about the >>> exception. For example, right now we only pass one object. But there >>> are cases where a second object would totally makes sense. >> >> If you have plans for adding new details in future, I propose to add a >> 6th parameter "context" or "extra" (always None currently). It is as >> extensible as packing all arguments into a single structure, but you do >> not need to introduce the structure type and create its instance until >> you need to pass additional info. > > In my implementation, UnraisableHookArgs is a C "structseq" type. In > short, it's a tuple. make_unraisable_hook_args() basically builds a > tuple of 4 items and uses PyTuple_SET_ITEM() to set the items. > _PyErr_WriteUnraisableDefaultHook() uses PyStructSequence_GET_ITEM() > to get items. > > The code pack and unpack UnraisableHookArgs is simple and reliable. > > An "unraisable exception" doesn't mean that Python is collapsing. It > only means that the code is unable to report the exception to the > caller: there is no reason why allocating a 4-tuple or calling a > simple Python function (sys.unraisablehook) would fail. > > -- > > I dislike the compromise of having an API in 2 parts: positional > parameters for some parameters, and a structure for some other > parameters. I prefer to pack all arguments into a single structure. I really like this API, and I agree with Victor that we don't really need more than the exception info. For future expansion, we can pass in a different exception, no? I'm not even convinced we need the obj argument, or that it's a good idea - this is yet another place where it's likely to be dead/dying already. And what can you do with it? Resurrection seems like a really bad idea, as does diving into a custom __repr__. There's no useful recovery mechanism here that I'm aware of, so I'd be in favor of just passing through the exception and nothing else. Cheers, Steve From steve.dower at python.org Thu May 16 12:02:29 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 09:02:29 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: On 16May2019 0856, Steve Dower wrote: > On 16May2019 0647, Victor Stinner wrote: >> Le jeu. 16 mai 2019 ? 12:57, Serhiy Storchaka a >> ?crit : >>>> For unraisable hook, it's not hard to imagine other useful parameters >>>> that can be passed to the hook to provide more context about the >>>> exception. For example, right now we only pass one object. But there >>>> are cases where a second object would totally makes sense. >>> >>> If you have plans for adding new details in future, I propose to add a >>> 6th parameter "context" or "extra" (always None currently). It is as >>> extensible as packing all arguments into a single structure, but you do >>> not need to introduce the structure type and create its instance until >>> you need to pass additional info. >> >> In my implementation, UnraisableHookArgs is a C "structseq" type. In >> short, it's a tuple. make_unraisable_hook_args() basically builds a >> tuple of 4 items and uses PyTuple_SET_ITEM() to set the items. >> _PyErr_WriteUnraisableDefaultHook() uses PyStructSequence_GET_ITEM() >> to get items. >> >> The code pack and unpack UnraisableHookArgs is simple and reliable. >> >> An "unraisable exception" doesn't mean that Python is collapsing. It >> only means that the code is unable to report the exception to the >> caller: there is no reason why allocating a 4-tuple or calling a >> simple Python function (sys.unraisablehook) would fail. >> >> -- >> >> I dislike the compromise of having an API in 2 parts: positional >> parameters for some parameters, and a structure for some other >> parameters. I prefer to pack all arguments into a single structure. > > I really like this API, and I agree with Victor that we don't really > need more than the exception info. For future expansion, we can pass in > a different exception, no? > > I'm not even convinced we need the obj argument, or that it's a good > idea - this is yet another place where it's likely to be dead/dying > already. And what can you do with it? Resurrection seems like a really > bad idea, as does diving into a custom __repr__. There's no useful > recovery mechanism here that I'm aware of, so I'd be in favor of just > passing through the exception and nothing else. Actually, if the default implementation prints the exception message, how is this different from sys.excepthook? Specifically, from the point of customizing the hooks. If I were going to replace unraisablehook to do something on specific exceptions, I'm almost certainly going to replace excepthook as well, no? The only difference is whether there are any try/except blocks in between, and by definition I think the answer is no (in both cases, or we wouldn't have reached the hook). Do you have scenarios in mind where you would need these to do different things? And would those be just as well served by a flag on the exception object rather than an entirely separate hook? Cheers, Steve From steve.dower at python.org Thu May 16 12:12:26 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 09:12:26 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: On 16May2019 0902, Steve Dower wrote: > Actually, if the default implementation prints the exception message, > how is this different from sys.excepthook? Specifically, from the point > of customizing the hooks. > > If I were going to replace unraisablehook to do something on specific > exceptions, I'm almost certainly going to replace excepthook as well, > no? The only difference is whether there are any try/except blocks in > between, and by definition I think the answer is no (in both cases, or > we wouldn't have reached the hook). > > Do you have scenarios in mind where you would need these to do different > things? And would those be just as well served by a flag on the > exception object rather than an entirely separate hook? Finally, the changes aren't in yet but PEP 578 is accepted, so perhaps this could also be satisfied by an audit hook for an exception that's about to be ignored? Cheers, Steve From vstinner at redhat.com Thu May 16 13:25:06 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 19:25:06 +0200 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> Message-ID: Le jeu. 16 mai 2019 ? 16:10, Thomas Wouters a ?crit : >> Would you be ok with a "PyConfig_Init(PyConfig *config);" function >> which would initialize all fields to theire default values? Maybe >> PyConfig_INIT should be renamed to PyConfig_STATIC_INIT. >> >> You can find a similar API for pthread mutex, there is a init function >> *and* a macro for static initialization: >> >> int pthread_mutex_init(pthread_mutex_t *restrict mutex, >> const pthread_mutexattr_t *restrict attr); >> >> pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; > > > This was going to be my suggestion as well: for any non-trivial macro, we should have a function for it instead. Ok, I will do that. > I would also point out that PEP 587 has a code example that uses PyWideStringList_INIT, but that macro isn't mention anywhere else. Oh, I forgot to better document it. Well, the macro is trivial: #define _PyWstrList_INIT (_PyWstrList){.length = 0, .items = NULL} For consistency, I prefer to not initialize manually these fields, but use a macro instead. (Variables are allocated on the stack and so *must* be initialized.) > The PEP is a bit unclear as to the semantics of PyWideStringList as a whole: the example uses a static array with length, but doesn't explain what would happen with statically allocated data like that if you call the Append or Extend functions. It also doesn't cover how e.g. argv parsing would remove items from the list. (I would also suggest the PEP shouldn't use the term 'list', at least not unqualified, if it isn't an actual Python list.) Calling PyWideStringList_Append() or PyWideStringList_Insert() on a "constant" list will crash: don't do that :-) I tried to explain the subtle details of "constant" vs "dynamic" configurations in "Initialization with constant PyConfig" and "Memory allocations and Py_DecodeLocale()" functions. A "constant" PyWideStringList must not be used with a "dynamic" PyConfig: otherwise, PyConfig_Clear() will crash as well. I would prefer to have separated "const PyWideStringList" and "const PyConfig" types, but the C language doesn't convert "wchat_*" to "const wchar_t*" when you do that. We would need duplicated PyConstantWideStringList and PyConstantConfig structures, which would require to be "casted" to PyWideStringList and PyConfig internally to reuse the same code for constant and dynamic configuration. If you consider that the specific case of "constant configuration" adds too much burden / complexity, we mgiht remove it and always require to use dynamic configuration. Right now, Programs/_testembed.c almost uses only "constant" configuration. Using dynamic memory would make the code longer: need to handle memory allocation failures. > I understand the desire to make static allocation and initialisation possible, but since you only need PyWideStringList for PyConfig, not PyPreConfig (which sets the allocator), perhaps having a PyWideStringList_Init(), which copies memory, and PyWideStringList_Clear() to clear it, would be better? Do you mean to always require to build dynamic lists? Said differently, not allow to write something like the following code? static wchar_t* argv[] = { L"python3", L"-c", L"pass", L"arg2", }; _PyCoreConfig config = _PyCoreConfig_INIT; config.argv.length = Py_ARRAY_LENGTH(argv); config.argv.items = argv; >> If the private field "_init_main" of the PEP 587 is set to 0, >> Py_InitializeFromConfig() stops at the "core" phase (in fact, it's >> already implemented!). But I didn't implement yet a >> _Py_InitializeMain() function to "finish" the initialization. Let's >> say that it exists, we would get: >> >> --- >> PyConfig config = PyConfig_INIT; >> config._init_main = 0; >> PyInitError err = Py_InitializeFromConfig(&config); >> if (Py_INIT_FAILED(err)) { >> Py_ExitInitError(err); >> } >> >> /* add your code to customize Python here */ >> /* calling PyRun_SimpleString() here is safe */ >> >> /* finish Python initialization */ >> PyInitError err = _Py_InitializeMain(&config); >> if (Py_INIT_FAILED(err)) { >> Py_ExitInitError(err); >> } >> --- >> >> Would it solve your use case? > > > FWIW, I understand the need here: for Hermetic Python, we solved it by adding a new API similar to PyImport_AppendInittab, but instead registering a generic callback hook to be called *during* the initialisation process: after the base runtime and the import mechanism are initialised (at which point you can create Python objects), but before *any* modules are imported. We use that callback to insert a meta-importer that satisfies all stdlib imports from an embedded archive. (Using a meta-importer allows us to bypass the fileysystem altogether, even for what would otherwise be failed path lookups.) > > As I mentioned, Hermetic Python was originally written for Python 2.7, but this approach works fine with a frozen importlib as well. The idea of 'core' and 'main' initialisation will likely work for this, as well. Well, even if it's not part of the PEP 587, I just implemented it anyway while fixing a bug: https://github.com/python/cpython/commit/9ef5dcaa0b3c7c7ba28dbb3ec0c9507d9d05e3a9 Example: static int test_init_main(void) { _PyCoreConfig config = _PyCoreConfig_INIT; configure_init_main(&config); config._init_main = 0; _PyInitError err = _Py_InitializeFromConfig(&config); if (_Py_INIT_FAILED(err)) { _Py_ExitInitError(err); } /* sys.stdout don't exist yet: it is created by _Py_InitializeMain() */ int res = PyRun_SimpleString( "import sys; " "print('Run Python code before _Py_InitializeMain', " "file=sys.stderr)"); if (res < 0) { exit(1); } err = _Py_InitializeMain(); if (_Py_INIT_FAILED(err)) { _Py_ExitInitError(err); } return _Py_RunMain(); } As you can see, it's possible execute Python between "core" and "main" initialization phases. Moreover, I even fixed Python to be able to use "import sys" before the "main" initialization phase ;-) (Only builtin and frozen modules are available at this stage.) Again, I'm not comfortable to make PyConfig._init_main and _Py_InitializeMain() public, because I consider that they are too experimental and we don't have enough time to discuss what is the "core" initialization phase exactly. > Other questions/comments about PEP 587: > > I really like the PyInitError struct. I would like more functions to use it, e.g. the PyRrun_* "very high level" API, which currently calls exit() for you on SystemExit, and returns -1 without any other information on error. For those, I'm not entirely sure 'Init' makes sense in the name... but I can live with it. PyInitError structure can be renamed PyError, but it should only be used with functions which can exit Python. In short, are you talking "The Very High Level Layer" of the C API? https://docs.python.org/dev/c-api/veryhigh.html One issue is that I dislike adding new functions to the C API, but it seems like we should add a few to provide a better API for embedded Python. libpython must never exit the process! (or only when you explicity asks that :-)) Note: PyRun_SimpleStringFlags() is a wrapper which makes PyRun_StringFlags() usage easier. PyRun_StringFlags() doesn't handle the exception and so let you decide how to handle it. > A couple of things are documented as performing pre-initialisation (PyConfig_SetBytesString, PyConfig_SetBytesArgv). I understand why, but I feel like that might be confusing and error-prone. Would it not be better to have them fail if pre-initialisation hasn't been performed yet? It's easier to modify the code to fail with an error if Python is not pre-initialized. I propose to implicitly pre-initialize Python to make the API easier to use. In practice, you rarely have to explicitly pre-initialize Python. The default PyPreConfig is just fine for almost all use cases, especially since Nick Coghlan and me decided to disable C locale coercion and UTF-8 Mode by default. You now have to opt-in to enable these encoding features. > The buffered_stdio field of PyConfig mentions stdout and stderr, but not stdin. Does it not affect stdin? Extract of create_stdio(): /* stdin is always opened in buffered mode, first because it shouldn't make a difference in common use cases, second because TextIOWrapper depends on the presence of a read1() method which only exists on buffered streams. */ Note: Unbuffered stdin doesn't magically make the producer on the other side of a pipe flushing its (stdout/stderr) buffer more frequently :-) > (Many of the fields could do with a bit more explicit documentation, to be honest.) Well, 2 years ago, almost no configuration parameter was documented :-) I helped to document "Global configuration variables" at: https://docs.python.org/dev/c-api/init.html#global-configuration-variables I had to reverse engineer the code to be able to document it :-D Right now, my reference documentation lives in Include/cpython/coreconfig.h. Some parameters are better documented there, than in the PEP. I can try to enhance the documentation in the PEP. > The configure_c_stdio field of PyConfig sounds like it might not set sys.stdin/stdout/stderr. That would be new behaviour, but configure_c_stdio doesn't have an existing equivalence, so I'm not sure if that's what you meant or not. In Python 3.7, only Py_Main() configured C standard streams. I moved the code into _PyCoreConfig_Write() which is called by _Py_InitializeFromConfig() and so by Py_Initialize() as well. My intent is to be able to get the same behavior using Py_Initialize() + Py_RunMain(), than using Py_Main(). Said differently, Python 3.8 now always configures C standard streams. Maybe I should modify the configure_c_stdio default value to 0, and only enable it by default in Py_Main()? Honestly, I'm a little bit confused here. I'm not sure what is the expected behavior. Usually, in case of doubt, I look at the behavior before my refactoring. The old behaviour was that only Py_Main() configured C standard streams. Maybe I should restore this behavior. But to build a customized Python which should behave as the regular Python, you would like opt-in for configure_c_stdio=1. Maybe we need a function to set the configuration to get "regular Python" behavior? Something like: PyConfig_SetRegularPythonBehavior()? (sorry for the silly name!) > The dll_path field of PyConfig says "Windows only". Does that meant the struct doesn't have that field except in a Windows build? Or is it ignored, instead? If it doesn't have that field at all, what #define can be used to determine if the PyConfig struct will have it or not? The field doesn't exist on non-Windows platforms. I chose to expose it to let the developer chooses where Python looks for DLL. But Steve just said (in an email below) that there is no reason to make it configurable. In that case, I will make it internal again. It seems like I misunderstood the purpose of this parameter. > It feels a bit weird to have both 'inspect' and 'interactive' in PyConfig. Is there a substantive difference between them? Is this just so you can easily tell if any of run_module / run_command / run_filename are set? In Python 3.7, there are Py_InspectFlag and Py_InteractiveFlag. If "interactive" parameter is non-zero, C standard streams are configured as buffered. It is also used to decide if stdin is considered as interactive or not: /* Return non-zero is stdin is a TTY or if -i command line option is used */ static int stdin_is_interactive(const _PyCoreConfig *config) { return (isatty(fileno(stdin)) || config->interactive); } The "inspect" parameter is used to decide if we start a REPL or not. The "-i" command line option sets inspect (Py_InspectFlag) and interactive (Py_InteractiveFlag) to 1. These flags are exposed at Python level as sys.flags.inspect and sys.flags.interactive. ... Honestly, I'm not sure if there is a real difference between these two flags, but they are exposed and exist for years... so I decided to keep them. > "module_search_path_env" sounds like an awkward and somewhat misleading name for the translation of PYTHONPATH. Can we not just use, say, pythonpath_env? I expect the intended audience to know that PYTHONPATH != sys.path. Sure, I can rename it. > The module_search_paths field in PyConfig doesn't mention if it's setting or adding to the calculated sys.path. As a whole, the path-calculation bits are a bit under-documented. Py_InitializeFromConfig() sets sys.path from module_search_paths. sys.path doesn't exist before Py_InitializeFromConfig() is called. > Since this is an awkward bit of CPython, it wouldn't hurt to mention what "the default path configuration" does (i.e. search for python's home starting at program_name, add fixed subdirs to it, etc.) Oh, that's a big task :-) Nobody knows what getpath.c and getpathp.c do :-D > Path configuration is mentioned as being able to issue warnings, but it doesn't mention *how*. It can't be the warnings module at this stage. I presume it's just printing to stderr. First, I didn't know, but I just saw that it's only on Unix (getpath.c). On Windows (getpathp.c), no warning is emitted. The warning is written into C stderr. The flag isn't new: it's based on Py_FrozenFlag. When I looked at how Python is embedded, I was surprised by the number of applications setting Py_FrozenFlag to 1 to suppress these warnings. > Regarding Py_RunMain(): does it do the right thing when something calls PyErr_Print() with SystemExit set? (I mentioned last week that PyErr_Print() will call C's exit() in that case, which is obviously terrible for embedders.) I spent a significant amount of time to ensure that Py_InitializeFromConfig() and Py_RunMain() don't exit directly, but return a proper failure or exit code. For example, Python 3.6 contains around 319 calls to Py_FatalError(). The master branch contains around 181 calls to Py_FatalError(): still a lot, but I converted 138 calls to _Py_INIT_ERR() ;-) The work is not complete: I just checked, Py_RunMain() still calls directly PyErr_Print() at many places. Well, the code can be fixed, and it's not directly related to the PEP, is it? The issue already existed in Python 3.7 with Py_Main(). > Regarding isolated_mode and the site module, should we make stronger guarantees about site.py's behaviour being optional? The problem with site is that it does four things that aren't configurable, one of which is usually very desirable, one of which probably doesn't matter to embedders, and two that are iffy: sys.path deduplication and canonicalisation (and fixing up __file__/__cached__ attributes of already-imported modules); adding site-packages directories; looking for and importing sitecustomize.py; executing .pth files. The site module doesn't easily allow doing only some of these. (user-site directories are an exception, as they have their own flag, so I'm not listing that here.) With Hermetic Python we don't care about any of these (for a variety of different reasons), but I'm always a little worried that future Python versions would add behaviour to site that we *do* need. Honestly, I would prefer to simply remove the site module, I dislike it because it makes Python startup way slower :-) ... But well, it does a few important things :-) About the PEP 587: PyConfig.user_site_directory is exported as sys.flags.no_user_site (negative value) which is used by the site module. I'm not sure if you are asking me to modify my PEP, or if it's more a general remark. The PEP 587 gives control on how sys.path is initialized. In the "Isolate Python" section, I suggest to set the "isolated" parameter to 1 which imply setting user_site_directory to 0. So sys.path isn't modified afterwards. What you pass to PyConfig is what you get in sys.path in this case. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From matusvalo at gmail.com Thu May 16 14:04:25 2019 From: matusvalo at gmail.com (matus valo) Date: Thu, 16 May 2019 20:04:25 +0200 Subject: [Python-Dev] bpo-9584: Added brace expressions to the glob and fnmatch. Message-ID: Hi All, I have implemented PR fixing bpo-9584 few months ago. Recently, I was encouraged that conversation on DEV mailing list should be started here on DEV mailing list. From the conversation in bug report, I understood that the main problem is that straightforward implementation can break a backward compatibility. E.g.: * Patched python: >>> import os >>> import glob >>> os.makedirs('a{b,c}d/e') >>> os.listdir('a{b,c}d') ['e'] >>> glob.glob('a{b,c}d/*') [] >>> * Unpatched python: >>> import os, glob >>> os.makedirs('a{b,c}d/e') >>> os.listdir('a{b,c}d') ['e'] >>> glob.glob('a{b,c}d/*') ['a{b,c}d/e'] How can I proceed with the PR? Please advise. Thanks, Matus Valo -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Thu May 16 14:10:55 2019 From: barry at python.org (Barry Warsaw) Date: Thu, 16 May 2019 11:10:55 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: <0AB0D675-4CA8-46D2-93A9-2F83E6AF3FA6@python.org> On May 16, 2019, at 03:12, Victor Stinner wrote: > I came to the UnraisableHookArgs structseq idea because of my bad > experience with extension warnings "hooks". Let me tell you this story > :-) Thanks for that! > def showwarning(message, category, filename, lineno, file=None, line=None): ... > def formatwarning(message, category, filename, lineno, line=None): ? While I?m fine with the API you propose, you could consider keyword-only arguments to help future proof the call signature. I don?t like it as much because it?s not as backward compatibility proof, but it?s an option to perhaps consider. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From vstinner at redhat.com Thu May 16 14:38:17 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 20:38:17 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: <0AB0D675-4CA8-46D2-93A9-2F83E6AF3FA6@python.org> References: <0AB0D675-4CA8-46D2-93A9-2F83E6AF3FA6@python.org> Message-ID: > While I?m fine with the API you propose, you could consider keyword-only arguments to help future proof the call signature. I don?t like it as much because it?s not as backward compatibility proof, but it?s an option to perhaps consider. Let's say that we have: hook(*, exc_type, exc_value, exc_tb, obj). Now we pass a new err_msg keyword argument: the call fails with a TypeError :-( Or do you mean using **kwargs? I don't think **kwargs is a good API :-( Well, the same can be done using *args and positional arguments to ignore new unknown arguments, but I also dislike such API. Victor -- Night gathers, and now my watch begins. It shall not end until my death. -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Thu May 16 14:58:33 2019 From: encukou at gmail.com (Petr Viktorin) Date: Thu, 16 May 2019 20:58:33 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: Message-ID: <314887b2-8fb9-447d-7293-2babb8ba5f00@gmail.com> On 5/16/19 3:23 AM, Victor Stinner wrote: [...] > I modified my API to create an object to pack arguments. The new API > becomes sys.unraisablehook(unraisable) where unraisable has 4 fields: > exc_type, exc_value, exc_tb, obj. [...] I always thought the classic (exc_type, exc_value, exc_tb) triple is a holdover from older Python versions, and all the information is now in the exception instance. Is the triple ever different from (type(exc), exc, exc.__traceback__)? (possibly with a getattr for __traceback__) Should new APIs use it? From njs at pobox.com Thu May 16 15:29:00 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 16 May 2019 12:29:00 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: On Thu, May 16, 2019, 09:07 Steve Dower wrote: > > Actually, if the default implementation prints the exception message, > how is this different from sys.excepthook? Specifically, from the point > of customizing the hooks. > sys.excepthook means the program has fully unwound and is about to exit. This is pretty different from an exception inside a __del__ or background thread or whatever, where the program definitely hasn't unwound and is likely to continue. And I'm pretty sure they have behavioral differences already, like if you pass in a SystemExit exception then sys.excepthook doesn't print anything, but PyErr_WriteUnraiseable prints a traceback. So making them two separate hooks seems right to me. Some people will override both; that's fine. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu May 16 15:41:13 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 16 May 2019 21:41:13 +0200 Subject: [Python-Dev] Need review Message-ID: <20190516214113.07623f9a@fsol> Hello, I need a review on the PEP 574 implementation: https://github.com/python/cpython/pull/7076 I would really like it to be in 3.8 and so ideally it should be merged within two weeks. Regards Antoine. From vstinner at redhat.com Thu May 16 16:22:00 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 22:22:00 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: <314887b2-8fb9-447d-7293-2babb8ba5f00@gmail.com> References: <314887b2-8fb9-447d-7293-2babb8ba5f00@gmail.com> Message-ID: Le jeu. 16 mai 2019 ? 20:58, Petr Viktorin a ?crit : > I always thought the classic (exc_type, exc_value, exc_tb) triple is a > holdover from older Python versions, and all the information is now in > the exception instance. > Is the triple ever different from (type(exc), exc, exc.__traceback__)? > (possibly with a getattr for __traceback__) I added assertions in PyErr_WriteTraceback(): assert(Py_TYPE(v) == t); assert(PyException_GetTraceback(v) == tb); "Py_TYPE(v) == t" fails in test_exceptions.test_memory_error_in_PyErr_PrintEx() for example. PyErr_NoMemory() calls PyErr_SetNone(PyExc_MemoryError), it sets tstate->curexc_type to PyExc_MemoryError, but tstate->curexc_value is set to NULLL. "PyException_GetTraceback(v) == tb" fails in test_exceptions.test_unraisable() for example: "PyTraceBack_Here(f);" in the "error:" label of ceval.c creates a traceback object and sets it to tstate->curexec_traceback, but it doesn't set the __traceback__ attribute of the current exception. > Should new APIs use it? I tried to add a "PyErr_NormalizeException(&t, &v, &tb);" call in PyErr_WriteUnraisable(): it creates an exception object (exc_value) for the PyErr_NoMemory() case, but it still doesn't set the __traceback__ attribute of the exception for the PyTraceBack_Here() case. It seems like PyErr_WriteUnraisable() cannot avoid having 3 variables (exc_type, exc_value, exc_tb), since they are not consistent as you may expect. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From vstinner at redhat.com Thu May 16 17:04:49 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 23:04:49 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: Le jeu. 16 mai 2019 ? 17:56, Steve Dower a ?crit : > I really like this API, and I agree with Victor that we don't really > need more than the exception info. For future expansion, we can pass in > a different exception, no? Sorry, I don't understand. I explained that we need more than (exc_type, exc_value, exc_tb). "obj" is part of the C function PyErr_WriteUnraisable(). We have to pass it to the hook. Otherwise, how do you want to guess where the exception comes from? Without any context, it can be super painful to debug such exception :-( That's why I said that I like Serhiy's idea of extending the API to allow to also pass an error message. Unraisable exceptions are commonly raised during garbage collection or in finalizers. So it's commonly happens when you don't "expect" them :-) > I'm not even convinced we need the obj argument, or that it's a good > idea - this is yet another place where it's likely to be dead/dying > already. And what can you do with it? Resurrection seems like a really > bad idea, as does diving into a custom __repr__. There's no useful > recovery mechanism here that I'm aware of, so I'd be in favor of just > passing through the exception and nothing else. Well, first of all, I would advice to *not* keep "obj" alive after the execution of the hook. Keep repr(obj) if you want, but not a reference to the object. Same for the exception value, keeping it alive is a high risk of creating annoying reference cycles ;-) -- In a finalizer, "obj" is not always the object being finalized. Example: when an _asyncio.Future is finalized, loop.call_exception_handler() is called if the future wasn't consumed. If this call fails, PyErr_WriteUnraisable(func) is called. func isn't the future, but the call_exception_handler() method. Example: on a garbage collection, callbacks of weak references are called. If a callback fails, PyErr_WriteUnraisable(callback) is called. It's not the collected object nor the weak reference. It's also common that PyErr_WriteUnraisable(NULL) is called. In this case, you have no context where the exception comes from :-( For that, I experimented a custom hook which logs also the current Python stack. That's useful in many cases! socket.socket finalizer is a case where the finalized object (the socket) is passed to the hook. IMHO it's fine to resurrect a socket in that case. Finalization isn't deallocation. The objet remains consistent. The finalization protocol ensures that the finalizer is only called once. In practice, I wouldn't advice to resurrect objects :-) I would expect that a hook only calls repr(hook) and then forgets the object. The current PyErr_WriteUnraisable() implementation does exactly that: it formats repr(obj) into sys.stderr. *If* someone finds a case where PyErr_WriteUnraisable() can resurect an object that is freed just after the call, I would suggest to fix the code to not pass the object to PyErr_WriteUnraisable(). Using PEP 442 finalizers, I don't think that it should happen. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From pablogsal at gmail.com Thu May 16 17:12:29 2019 From: pablogsal at gmail.com (Pablo Galindo Salgado) Date: Thu, 16 May 2019 22:12:29 +0100 Subject: [Python-Dev] Parser module in the stdlib Message-ID: Hi everyone, TLDR ===== I propose to remove the current parser module and expose pgen2 as a standard library module. Some context =========== The parser module has been "deprecated" (technically we recommend to prefer the ast module instead) since Python2.5 but is still in the standard library. Is a 1222-line C module that needs to be kept in sync with the grammar and the Python parser sometimes by hand. It has also been broken for several years: I recently fixed a bug that was introduced in Python 3.5 that caused the parse module to not being able to parse if-else blocks ( https://bugs.python.org/issue36256). The interface it provides is a very raw view of the CST. For instance: >>> parser.sequence2st(parser.suite("def f(x,y,z): pass").totuple()) provides an object with methods compile, isexpr, issuite, tolist, totuple. The last two produce containers with the numerical values of the grammar elements (tokens, dfas...etc) >>> parser.suite("def f(x,y,z): pass").tolist() [257, [269, [295, [263, [1, 'def'], [1, 'f'], [264, [7, '('], [265, [266, [1, 'x']], [12, ','], [266, [1, 'y']], [12, ','], [266, [1, 'z']]], [8, ')']], [11, ':'], [304, [270, [271, [277, [1, 'pass']]], [4, '']]]]]], [4, ''], [0, '']] This is a very raw interface and is very tied to the particularities of CPython without almost any abstraction. On the other hand, there is a Python implementation of the Python parser and parser generator in lib2to3 (pgen2). This module is not documented and is usually considered an implementation detail of lib2to3 but is extremely useful. Several 3rd party packages (black, fissix...) are using it directly or have their own forks due to the fact that it can get outdated with respect to the Python3 grammar as it was originally used only for Python2 to Python3 migration. It has the ability to consume LL1 grammars in EBNF form and produces an LL1 parser for them (by creating parser tables that the same module can consume). Many people use the module currently to support or parse supersets of Python (like Python2/3 compatible grammars, cython-like changes...etc). Proposition ======== I propose to remove finally the parser module as it has been "deprecated" for a long time, is almost clear that nobody uses it and has very limited usability and replace it (maybe with a different name) with pgen2 (maybe with a more generic interface that is detached to lib2to3 particularities). This will not only help a lot current libraries that are using forks or similar solutions but also will help to keep synchronized the shipped grammar (that is able to parse Python2 and Python3 code) with the current Python one (as now will be more justified to keep them in sync). What do people think about? Do you have any concerns? Do you think is a good/bad idea? Regards from sunny London, Pablo Galindo -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu May 16 17:16:49 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 14:16:49 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: On 16May2019 1404, Victor Stinner wrote: > Le jeu. 16 mai 2019 ? 17:56, Steve Dower a ?crit : >> I really like this API, and I agree with Victor that we don't really >> need more than the exception info. For future expansion, we can pass in >> a different exception, no? > > Sorry, I don't understand. I explained that we need more than > (exc_type, exc_value, exc_tb). > > "obj" is part of the C function PyErr_WriteUnraisable(). We have to > pass it to the hook. Otherwise, how do you want to guess where the > exception comes from? Without any context, it can be super painful to > debug such exception :-( You go on to say "pass an error message" and "keep repr(obj) if you want", but how is this different from creating an exception that contains the custom message, the repr of the object, and chains the exception that triggered it? If you just have a message and no exception at this point, firstly it's going against what the hook is documented as doing, but secondly why not convert the message into an exception? Maybe I just missed where you explained how it isn't possible to represent an error message as an exception, but I'm pretty sure I can't be the only one who will have missed it, so maybe you could explain just that one point? Thanks, Steve From vstinner at redhat.com Thu May 16 17:41:39 2019 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 16 May 2019 23:41:39 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: Le jeu. 16 mai 2019 ? 23:17, Steve Dower a ?crit : > You go on to say "pass an error message" and "keep repr(obj) if you > want", but how is this different from creating an exception that > contains the custom message, the repr of the object, and chains the > exception that triggered it? Well, "unraisable exceptions" are raised when something goes wrong. I'm not comfortable with creating a new exception instance and chaining it to the previous exception, because it could go wrong and raise a 3rd exception. Issue a new error while trying to log an error can be annoying :-( Moreover, when I looked at details of the "unraisable exception" (see my reply to Petr Viktorin), I saw that the exception is not well defined as you might expect. (exc_type, exc_value, exc_tb) can *not* be replaced with (type(exc_value), exc_value, exc_value.__traceback__). Sometimes, exc_value is None. Sometimes, exc_tb is a traceback object, but exc_value.__traceback__ is None. So I'm not comfortable neither to chain such badly shaped exception into a new exception. I prefer to pass "raw" values and let the hook decides how to handle them :-) Said differently, I prefer to design the hook to restrict the risks of creating a new error. At least in the code which calls the hook. Inside the hook. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From njs at pobox.com Thu May 16 17:46:32 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 16 May 2019 14:46:32 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: On Thu, May 16, 2019 at 2:17 PM Steve Dower wrote: > You go on to say "pass an error message" and "keep repr(obj) if you > want", but how is this different from creating an exception that > contains the custom message, the repr of the object, and chains the > exception that triggered it? A clever hook might want the actual object, so it can pretty-print it, or open an interactive debugger and let it you examine it, or something. Morally this is similar to calling repr(obj), but it doesn't literally call repr(obj). -n -- Nathaniel J. Smith -- https://vorpus.org From vstinner at redhat.com Thu May 16 18:12:51 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 17 May 2019 00:12:51 +0200 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: Le jeu. 16 mai 2019 ? 23:15, Pablo Galindo Salgado a ?crit : > The parser module has been "deprecated" (technically we recommend to prefer the ast module instead) since Python2.5 but is still in the standard library. Importing it doesn't emit a DeprecationgWarning. It's only deprecated in the documentation: https://docs.python.org/dev/library/parser.html I searched for "import parser" in Python on GitHub. I looked at the 10 pages of results: I cannot find any code which looks to import "parser" from the Python standard library. Only "import parser" in a test suite of PyPy, something unrelated. > >>> parser.suite("def f(x,y,z): pass").tolist() > [257, [269, [295, [263, [1, 'def'], [1, 'f'], [264, [7, '('], [265, [266, [1, 'x']], [12, ','], [266, [1, 'y']], [12, ','], [266, [1, 'z']]], [8, ')']], [11, ':'], [304, [270, [271, [277, [1, 'pass']]], [4, '']]]]]], [4, ''], [0, '']] I don't understand how anyone can read or use Concrete Syntax Tree (CST) directly :-( I would expect that you need another module on top of that to use a CST. But I'm not aware of anything in the stdlib, so I understand that... nobody uses this "parser" module. I never heard about the "parser" module in the stdlib before this email :-) > Several 3rd party packages (black, fissix...) are using it directly or have their own forks due to the fact that it can get outdated with respect to the Python3 > grammar as it was originally used only for Python2 to Python3 migration. It has the ability to consume LL1 grammars in EBNF form and produces an LL1 parser for them (by creating parser tables > that the same module can consume). Many people use the module currently to support or parse supersets of Python (like Python2/3 compatible grammars, cython-like changes...etc). * Did someone review the pgen2 implementation? * Does it have a good API? * Does it come with documentation? * Who is maintaining it? Who will maintain it? > I propose to remove finally the parser module as it has been "deprecated" for a long time, is almost clear that nobody uses it and has very limited usability and replace it (maybe with a different name) > with pgen2 (maybe with a more generic interface that is detached to lib2to3 particularities). I'm not sure that these two things have to be done at the same time. Maybe we can start by issuing a DeprecationWarning on "import parser" and announce the future removal in What's New in Python 3.8, just in case, and wait one more release before removing it. But I'm also fine with removing it right now. As I wrote, I never heard about this module previously... If your proposal is to reuse "parser" name, I'm not sure that it's a good idea, since it existed for years and had a different API. Why not keeping "pgen2" name, you said that it's already used under this name in the wild. Note: One thing to consider is that https://pypi.org/project/pgen2/ name is already used. It might conflict if the "new" pgen2 (pgen3? :-D) would have a little bit different API. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From vstinner at redhat.com Thu May 16 18:17:00 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 17 May 2019 00:17:00 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: Le jeu. 16 mai 2019 ? 23:46, Nathaniel Smith a ?crit : > A clever hook might want the actual object, so it can pretty-print it, > or open an interactive debugger and let it you examine it, or > something. Morally this is similar to calling repr(obj), but it > doesn't literally call repr(obj). Good point. By the way, I started to use tracemalloc.get_object_traceback(obj) to display where the object comes from more and more often :-D It helps me at least in debugging. So yeah, I would prefer to pass directly the object to the hook. For example, I'm using it with the "source" parameter of a ResourceWarning: https://pythondev.readthedocs.io/debug_tools.html#resourcewarning But also when a buffer underflow or overflow is detected by debug hooks on Python memory allocators: https://pythondev.readthedocs.io/debug_tools.html#pythonmalloc-debug It might also help me sometimes to be able to call tracemalloc.get_object_traceback(obj) from a custom unraisablehook(). Victor -- Night gathers, and now my watch begins. It shall not end until my death. From njs at pobox.com Thu May 16 18:41:26 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 16 May 2019 15:41:26 -0700 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: On Thu, May 16, 2019 at 2:13 PM Pablo Galindo Salgado wrote: > I propose to remove finally the parser module as it has been "deprecated" for a long time, is almost clear that nobody uses it and has very limited usability and replace it (maybe with a different name) > with pgen2 (maybe with a more generic interface that is detached to lib2to3 particularities). This will not only help a lot current libraries that are using forks or similar solutions but also will help to keep > synchronized the shipped grammar (that is able to parse Python2 and Python3 code) with the current Python one (as now will be more justified to keep them in sync). Will the folks using forks be happy to switch to the stdlib version? For example I can imagine that if black wants to process 3.7 input code while running on 3.6, it might prefer a parser on PyPI even if the stdlib version were public, since the PyPI version can be updated independently of the host Python. -n -- Nathaniel J. Smith -- https://vorpus.org From pablogsal at gmail.com Thu May 16 18:48:48 2019 From: pablogsal at gmail.com (Pablo Galindo Salgado) Date: Thu, 16 May 2019 23:48:48 +0100 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: > Will the folks using forks be happy to switch to the stdlib version? >For example I can imagine that if black wants to process 3.7 input >code while running on 3.6, it might prefer a parser on PyPI even if >he stdlib version were public, since the PyPI version can be updated >independently of the host Python. The tool can parse arbitrary grammars, the one that is packed into is just one of them. I think it would be useful, among other things because the standard library lacks currently a proper CST solution. The ast module is heavily leveraged for things like formatters, static code analyzers...etc but CST can be very useful as ?ukasz describes here: https://bugs.python.org/issue33337 I think is missing an important gap in the stdlib and the closest thing we have (the current parser module) is not useful for any of that. Also, the core to generating the hypothetical new package (with some new API over it may be) is already undocumented as an implementation detail of lib2to3 (and some people are already using it directly). On Thu, 16 May 2019 at 23:41, Nathaniel Smith wrote: > On Thu, May 16, 2019 at 2:13 PM Pablo Galindo Salgado > wrote: > > I propose to remove finally the parser module as it has been > "deprecated" for a long time, is almost clear that nobody uses it and has > very limited usability and replace it (maybe with a different name) > > with pgen2 (maybe with a more generic interface that is detached to > lib2to3 particularities). This will not only help a lot current libraries > that are using forks or similar solutions but also will help to keep > > synchronized the shipped grammar (that is able to parse Python2 and > Python3 code) with the current Python one (as now will be more justified to > keep them in sync). > > Will the folks using forks be happy to switch to the stdlib version? > For example I can imagine that if black wants to process 3.7 input > code while running on 3.6, it might prefer a parser on PyPI even if > the stdlib version were public, since the PyPI version can be updated > independently of the host Python. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu May 16 18:55:13 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 15:55:13 -0700 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: On 16May2019 1548, Pablo Galindo Salgado wrote: > > Will the folks using forks be happy to switch to the stdlib version? >>For example I can imagine that if black wants to process 3.7 input >>code while running on 3.6, it might prefer a parser on PyPI even if >>he stdlib version were public, since the PyPI version can be updated >>independently of the host Python. > The tool can parse arbitrary grammars, the one that is packed into is > just one of them. > > I think it would be useful, among other things because the standard library > lacks currently a proper CST solution. The ast module is heavily > leveraged for > things like formatters, static code analyzers...etc but CST can be very > useful as > ?ukasz describes here: > > https://bugs.python.org/issue33337 > > I think is missing an important gap in the stdlib and the closest thing > we have > (the current parser module) is not useful for any of that. Also, the > core to generating > the hypothetical new package (with some new API over it may be) is > already undocumented > as an implementation detail of lib2to3 (and some people are already > using it directly). We still have the policy of not removing modules that exist in the Python 2 standard library. But 3.9 won't be covered by that :) But I'm in favor of having a proper CST module that matches the version of Python it's in. It doesn't help people on earlier versions (yet), but given how closely tied it is to the Python version you're on I think it makes sense in the stdlib. Cheers, Steve From steve.dower at python.org Thu May 16 19:00:07 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 16 May 2019 16:00:07 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> Message-ID: <8332cadb-0434-e633-798e-7fe28ed5c09a@python.org> On 16May2019 1441, Victor Stinner wrote: > Le jeu. 16 mai 2019 ? 23:17, Steve Dower a ?crit : >> You go on to say "pass an error message" and "keep repr(obj) if you >> want", but how is this different from creating an exception that >> contains the custom message, the repr of the object, and chains the >> exception that triggered it? > > Well, "unraisable exceptions" are raised when something goes wrong. > I'm not comfortable with creating a new exception instance and > chaining it to the previous exception, because it could go wrong and > raise a 3rd exception. Issue a new error while trying to log an error > can be annoying :-( I mean, aren't they all? :) That's why they're exceptional. If we're not in a position to raise a new exception, then it's not safe to call into user's Python code. If it's safe to call into their Python code, then we have to be in a position to raise a new exception. You said earlier that we can safely do this, and now you're saying we can't safely do it. Which is it? > Moreover, when I looked at details of the "unraisable exception" (see > my reply to Petr Viktorin), I saw that the exception is not well > defined as you might expect. (exc_type, exc_value, exc_tb) can *not* > be replaced with (type(exc_value), exc_value, > exc_value.__traceback__). Sometimes, exc_value is None. Sometimes, > exc_tb is a traceback object, but exc_value.__traceback__ is None. So > I'm not comfortable neither to chain such badly shaped exception into > a new exception. Yeah, this is the normal optimization for raising and handling exceptions that never reach Python code. Exceptions are supposed to be normalized before re-entering the eval loop. > I prefer to pass "raw" values and let the hook decides how to handle them :-) This is fine, it's how we handle exceptions in most other places (such as __exit__, logging, sys, traceback and all through the C API), and skipping normalization for a last-chance "the-world-may-be-burning" hook isn't the worst thing ever. Cheers, Steve From greg at krypto.org Thu May 16 21:09:51 2019 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 16 May 2019 18:09:51 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: <8332cadb-0434-e633-798e-7fe28ed5c09a@python.org> References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> <8332cadb-0434-e633-798e-7fe28ed5c09a@python.org> Message-ID: I like the feature, we should have it. It'll be useful for debugging and probably more. Which brings me to the annoying paint color question: These exceptions were most definitely raised. Thus the term "unraisable" is wrong. I believe you really mean "uncatchable". -gps On Thu, May 16, 2019 at 4:00 PM Steve Dower wrote: > On 16May2019 1441, Victor Stinner wrote: > > Le jeu. 16 mai 2019 ? 23:17, Steve Dower a > ?crit : > >> You go on to say "pass an error message" and "keep repr(obj) if you > >> want", but how is this different from creating an exception that > >> contains the custom message, the repr of the object, and chains the > >> exception that triggered it? > > > > Well, "unraisable exceptions" are raised when something goes wrong. > > I'm not comfortable with creating a new exception instance and > > chaining it to the previous exception, because it could go wrong and > > raise a 3rd exception. Issue a new error while trying to log an error > > can be annoying :-( > > I mean, aren't they all? :) That's why they're exceptional. > > If we're not in a position to raise a new exception, then it's not safe > to call into user's Python code. If it's safe to call into their Python > code, then we have to be in a position to raise a new exception. > > You said earlier that we can safely do this, and now you're saying we > can't safely do it. Which is it? > > > Moreover, when I looked at details of the "unraisable exception" (see > > my reply to Petr Viktorin), I saw that the exception is not well > > defined as you might expect. (exc_type, exc_value, exc_tb) can *not* > > be replaced with (type(exc_value), exc_value, > > exc_value.__traceback__). Sometimes, exc_value is None. Sometimes, > > exc_tb is a traceback object, but exc_value.__traceback__ is None. So > > I'm not comfortable neither to chain such badly shaped exception into > > a new exception. > > Yeah, this is the normal optimization for raising and handling > exceptions that never reach Python code. Exceptions are supposed to be > normalized before re-entering the eval loop. > > > I prefer to pass "raw" values and let the hook decides how to handle > them :-) > > This is fine, it's how we handle exceptions in most other places (such > as __exit__, logging, sys, traceback and all through the C API), and > skipping normalization for a last-chance "the-world-may-be-burning" hook > isn't the worst thing ever. > > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Thu May 16 21:23:34 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 17 May 2019 03:23:34 +0200 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <8c678174-b68b-3e70-9f82-b4c7e274185c@python.org> <8332cadb-0434-e633-798e-7fe28ed5c09a@python.org> Message-ID: Le ven. 17 mai 2019 ? 03:10, Gregory P. Smith a ?crit : > I like the feature, we should have it. It'll be useful for debugging and probably more. > > Which brings me to the annoying paint color question: These exceptions were most definitely raised. Thus the term "unraisable" is wrong. I believe you really mean "uncatchable". "unraisablehook" name comes from the existing PyErr_WriteUnraisable(). I understood that "unraisable" means that the function calling PyErr_WriteUnraisable() cannot "raise" the exception to its caller. The exception has to be "swallowed" by the function. -- If you like the feature, please review my PR :-) https://github.com/python/cpython/pull/13187 Victor -- Night gathers, and now my watch begins. It shall not end until my death. From njs at pobox.com Thu May 16 21:32:39 2019 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 16 May 2019 18:32:39 -0700 Subject: [Python-Dev] bpo-36829: Add sys.unraisablehook() In-Reply-To: References: <314887b2-8fb9-447d-7293-2babb8ba5f00@gmail.com> Message-ID: On Thu, May 16, 2019 at 1:23 PM Victor Stinner wrote: > > Le jeu. 16 mai 2019 ? 20:58, Petr Viktorin a ?crit : > > I always thought the classic (exc_type, exc_value, exc_tb) triple is a > > holdover from older Python versions, and all the information is now in > > the exception instance. > > Is the triple ever different from (type(exc), exc, exc.__traceback__)? > > (possibly with a getattr for __traceback__) > > I added assertions in PyErr_WriteTraceback(): > > assert(Py_TYPE(v) == t); > assert(PyException_GetTraceback(v) == tb); > > "Py_TYPE(v) == t" fails in > test_exceptions.test_memory_error_in_PyErr_PrintEx() for example. > PyErr_NoMemory() calls PyErr_SetNone(PyExc_MemoryError), it sets > tstate->curexc_type to PyExc_MemoryError, but tstate->curexc_value is > set to NULLL. This makes some sense ? if you can't allocate memory, then better not allocate an exception instance to report that! So this is legitimately a special case. But... it looks like everywhere else, the way we handle this when transitioning into Python code is to create an instance. For example, that test does 'except MemoryError as e', so an instance does need to be created then. The comments suggest that there's some trick where we have pre-allocated MemoryError() instances? But either way, if we can afford to call a Python hook (which requires at least allocating a frame!), then we can probably also afford to materialize the MemoryError instance. So I feel like maybe we shouldn't be passing None to the unraisable hook, even if PyErr_NoMemory() did initially set that? Also, in practice, the only time I've ever seen MemoryError is from attempting to do a single massively-huge allocation. It's never meant that regular allocation of small objects will fail. > "PyException_GetTraceback(v) == tb" fails in > test_exceptions.test_unraisable() for example: "PyTraceBack_Here(f);" > in the "error:" label of ceval.c creates a traceback object and sets > it to tstate->curexec_traceback, but it doesn't set the __traceback__ > attribute of the current exception. Isn't this just a bug that should be fixed? > > Should new APIs use it? > > I tried to add a "PyErr_NormalizeException(&t, &v, &tb);" call in > PyErr_WriteUnraisable(): it creates an exception object (exc_value) > for the PyErr_NoMemory() case, but it still doesn't set the > __traceback__ attribute of the exception for the PyTraceBack_Here() > case. > > It seems like PyErr_WriteUnraisable() cannot avoid having 3 variables > (exc_type, exc_value, exc_tb), since they are not consistent as you > may expect. I'm actually fine with it having three arguments -- even if it's technically unnecessary, it's currently 100% consistent across these low-level APIs, and it doesn't hurt anything, so we might as well stick with it for consistency. -n -- Nathaniel J. Smith -- https://vorpus.org From bcannon at gmail.com Thu May 16 22:04:37 2019 From: bcannon at gmail.com (Brett Cannon) Date: Thu, 16 May 2019 19:04:37 -0700 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: On Thu., May 16, 2019, 15:56 Steve Dower, wrote: > On 16May2019 1548, Pablo Galindo Salgado wrote: > > > Will the folks using forks be happy to switch to the stdlib version? > >>For example I can imagine that if black wants to process 3.7 input > >>code while running on 3.6, it might prefer a parser on PyPI even if > >>he stdlib version were public, since the PyPI version can be updated > >>independently of the host Python. > > The tool can parse arbitrary grammars, the one that is packed into is > > just one of them. > > > > I think it would be useful, among other things because the standard > library > > lacks currently a proper CST solution. The ast module is heavily > > leveraged for > > things like formatters, static code analyzers...etc but CST can be very > > useful as > > ?ukasz describes here: > > > > https://bugs.python.org/issue33337 > > > > I think is missing an important gap in the stdlib and the closest thing > > we have > > (the current parser module) is not useful for any of that. Also, the > > core to generating > > the hypothetical new package (with some new API over it may be) is > > already undocumented > > as an implementation detail of lib2to3 (and some people are already > > using it directly). > > We still have the policy of not removing modules that exist in the > Python 2 standard library. But 3.9 won't be covered by that :) > Correct. ? We should deprecate in 3.8 for removal in 3.9. > But I'm in favor of having a proper CST module that matches the version > of Python it's in. It doesn't help people on earlier versions (yet), but > given how closely tied it is to the Python version you're on I think it > makes sense in the stdlib. > +1. I think someone brought up the API, and probably talking to projects that are using pgen2 out of lib2to3 would be good. -Brett > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Fri May 17 09:06:34 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Fri, 17 May 2019 22:06:34 +0900 Subject: [Python-Dev] Consolidating channel of record [was: 581 (Using GitHub issues for CPython) is accepted] In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> <23772.29178.404867.929820@turnbull.sk.tsukuba.ac.jp> Message-ID: <23774.45530.164132.186287@turnbull.sk.tsukuba.ac.jp> Chris Barker - NOAA Federal writes: > Frankly, multiple long meandering threads in s single mailing list > are not s very good archive either. True, but I have no idea how to address that administratively, except to have a *very* strong moderator. > Ideally, the PEP is updated with a summary of the issues discussed > as the discussion unfolds. The point of "channel of record" is not that it would replace the PEPs themselves. It's that PEP authors are people with time limitations, too. For example, there's at least one person currently requesting a review on a PEP who has in the past loudly and publicly announced their unsubscription from python-ideas and IIRC even python-dev. I think it's a good idea to have *one* place that is the principal source for PEP protagonists who don't have time for or simply dislike participating in some (or even all) of the channels du jour. With a designated channel of record, if discussions take place elsewhere, the PEP protagonist *may*, but would be under *no obligation* to, follow them or search for them. In the current situation, there's an implicit obligation for the protagonist to check all the channels, maybe even to do a Google search. Or at least some of the posters seem to feel that way. > As the community seems to be moving to a wider variety of fora, > this will become all the more critical. It would also be possible, I suppose, for the PEP protagonist to designate a channel in the PEP. That has its problems too (they can choose a channel that is inconvenient for the majority of interested participants, for example), but with a designated channel system of either kind the problems are predictable and bounded. Steve From qiang.fang at zoho.com.cn Fri May 17 04:00:11 2019 From: qiang.fang at zoho.com.cn (Q) Date: Fri, 17 May 2019 16:00:11 +0800 Subject: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation Message-ID: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> A lot of the Python code we use in production are used directly as imports in other python distributions (such as the python comes with the finite element software Abaqus and MSC Marc), many packages (such as matplotlib, numpy) that may have varying versioned dependencies. I was wondering if this could be expanded to allow a version to be set within a package and have that propagate to all modules in that package. For example in the root init.py if I set multiversion(tornado, 2.2.1) then all modules in that package will use tornado 2.2.1 when I import tornado. See a relevant issue on github: https://github.com/mitsuhiko/multiversion/issues/1 Thank you! Qiang -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiang.fang at zoho.com.cn Fri May 17 04:09:45 2019 From: qiang.fang at zoho.com.cn (Q) Date: Fri, 17 May 2019 16:09:45 +0800 Subject: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation In-Reply-To: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> References: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> Message-ID: <16ac4d7010e.1180c0d4d4832.5644021354870491002@zoho.com.cn> A lot of the Python code we use in production are used directly as imports in other python distributions (such as the python comes with the finite element software Abaqus and MSC Marc), many packages (such as matplotlib, numpy) that may have varying versioned dependencies, which makes it is a pain to get all these dependencies met in those close source python distributions. The work around is to let a version to be set within a package and have that propagate to all modules in that package. For example in the root init.py if I set tonado version to be 2.2.1 then all modules in that package will use tornado 2.2.1 when I import tornado. See a relevant issue of a similar package on github: https://github.com/mitsuhiko/multiversion/issues/1 Thank you! Qiang -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Fri May 17 11:25:15 2019 From: bcannon at gmail.com (Brett Cannon) Date: Fri, 17 May 2019 08:25:15 -0700 Subject: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation In-Reply-To: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> References: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> Message-ID: Thanks for the idea but there are currently no plans to support such a feature. If you would like to see it then you will need to write a PEP with a proof-of-concept to demonstrate how you would expect such a feature to work. On Fri., May 17, 2019, 07:55 Q via Python-Dev, wrote: > A lot of the Python code we use in production are used directly as imports > in other python > distributions (such as the python comes with the finite element software > Abaqus and MSC Marc), many > packages (such as matplotlib, numpy) that may have varying versioned > dependencies. > > I was wondering if this could be expanded to allow a version to be set > within a package and have > that propagate to all modules in that package. For example in the root > init.py if I set > multiversion(tornado, 2.2.1) then all modules in that package will use > tornado 2.2.1 when I import > tornado. > > See a relevant issue on github: > https://github.com/mitsuhiko/multiversion/issues/1 > > Thank you! > Qiang > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From dholth at gmail.com Fri May 17 11:38:55 2019 From: dholth at gmail.com (Daniel Holth) Date: Fri, 17 May 2019 11:38:55 -0400 Subject: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation In-Reply-To: References: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> Message-ID: This sounds exactly like what people used to do with eggs. You could have multiple versions of a package on the path as eggs and then require a version at runtime. The approach has problems. Ruby also abandoned a strategy where random app code depends on package management code at runtime. One better strategy is to set up a python path in a wrapper script. On Fri, May 17, 2019, 11:27 Brett Cannon wrote: > Thanks for the idea but there are currently no plans to support such a > feature. If you would like to see it then you will need to write a PEP with > a proof-of-concept to demonstrate how you would expect such a feature to > work. > > On Fri., May 17, 2019, 07:55 Q via Python-Dev, > wrote: > >> A lot of the Python code we use in production are used directly as >> imports in other python >> distributions (such as the python comes with the finite element software >> Abaqus and MSC Marc), many >> packages (such as matplotlib, numpy) that may have varying versioned >> dependencies. >> >> I was wondering if this could be expanded to allow a version to be set >> within a package and have >> that propagate to all modules in that package. For example in the root >> init.py if I set >> multiversion(tornado, 2.2.1) then all modules in that package will use >> tornado 2.2.1 when I import >> tornado. >> >> See a relevant issue on github: >> https://github.com/mitsuhiko/multiversion/issues/1 >> >> Thank you! >> Qiang >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri May 17 14:08:03 2019 From: status at bugs.python.org (Python tracker) Date: Fri, 17 May 2019 18:08:03 +0000 (UTC) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20190517180803.A2F9352B235@bugs.ams1.psf.io> ACTIVITY SUMMARY (2019-05-10 - 2019-05-17) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7067 (-17) closed 41633 (+91) total 48700 (+74) Open issues with patches: 2861 Issues opened (56) ================== #36084: Threading: add builtin TID attribute to Thread objects https://bugs.python.org/issue36084 reopened by vstinner #36801: Wait for connection_lost in StreamWriter.drain https://bugs.python.org/issue36801 reopened by asvetlov #36802: Revert back StreamWriter awrite/aclose but provide await write https://bugs.python.org/issue36802 reopened by vstinner #36817: Add = to f-strings for easier debugging. https://bugs.python.org/issue36817 reopened by pablogsal #36876: Global C variables are a problem. https://bugs.python.org/issue36876 opened by eric.snow #36877: [meta] Move fields from _PyRuntimeState to PyInterpreterState. https://bugs.python.org/issue36877 opened by eric.snow #36878: ast.parse with type_comments=True should allow extra text afte https://bugs.python.org/issue36878 opened by msullivan #36879: bug with round() and "numpy floats" https://bugs.python.org/issue36879 opened by Scaler #36880: Returning None from a callback with restype py_object decremen https://bugs.python.org/issue36880 opened by dgelessus #36881: isinstance raises TypeError for metaclass with metaclass=ABCMe https://bugs.python.org/issue36881 opened by emanspeaks #36882: ???ctypes?????????C?????????c_char_p?????????????????? https://bugs.python.org/issue36882 opened by fy2099 #36885: Make makeunicode.py script more readable https://bugs.python.org/issue36885 opened by scoder #36887: Add integer square root, math.isqrt https://bugs.python.org/issue36887 opened by mark.dickinson #36888: Create a way to check that the parent process is alive for dea https://bugs.python.org/issue36888 opened by tomMoral #36889: Merge StreamWriter and StreamReader into just asyncio.Stream https://bugs.python.org/issue36889 opened by asvetlov #36890: python-3.7.3-macosx10.6.pkg verification error on macOS 10.6 S https://bugs.python.org/issue36890 opened by steve_314 #36891: Additional startup plugin for vendors https://bugs.python.org/issue36891 opened by vx1920 #36892: "Modules" section in Tutorial contains incorrect description a https://bugs.python.org/issue36892 opened by musou1500 #36893: email.headerregistry.Address blocks Unicode local part addr_sp https://bugs.python.org/issue36893 opened by dracos #36894: test_multiprocessing_spawn regression on Windows https://bugs.python.org/issue36894 opened by pitrou #36895: time.clock() marked for removal in 3.8 ?????still there. https://bugs.python.org/issue36895 opened by mbussonn #36896: clarify in types.rst that FunctionTypes & co constructors don https://bugs.python.org/issue36896 opened by mbussonn #36897: shlex doesn't differentiate escaped characters in output https://bugs.python.org/issue36897 opened by Matthew Gamble #36898: Add parameter @case_sensitive to glob and rglob in pathlib https://bugs.python.org/issue36898 opened by Chuang Men #36901: Fix leaks in /PC/bdist_wininst/install.c https://bugs.python.org/issue36901 opened by cheryl.sabella #36904: Implement _PyStack_UnpackDict() with a single allocation https://bugs.python.org/issue36904 opened by jdemeyer #36905: test_typing.GetTypeHintTests.test_get_type_hints_modules_forwa https://bugs.python.org/issue36905 opened by xtreak #36906: Compile time textwrap.dedent() equivalent for str or bytes lit https://bugs.python.org/issue36906 opened by gregory.p.smith #36907: Crash due to borrowed references in _PyStack_UnpackDict() https://bugs.python.org/issue36907 opened by jdemeyer #36908: "This module is always available" is confusing https://bugs.python.org/issue36908 opened by nedbat #36910: Certain Malformed email causes email.parser to throw Attribute https://bugs.python.org/issue36910 opened by msapiro #36913: Missing documentation for decorators https://bugs.python.org/issue36913 opened by tomerv #36917: ast.NodeVisitor no longer calls visit_Str https://bugs.python.org/issue36917 opened by Anthony Sottile #36918: ValueError warning in test_urllib due to io.IOBase destructor https://bugs.python.org/issue36918 opened by xtreak #36919: Exception from 'compile' reports a newline char not present in https://bugs.python.org/issue36919 opened by BCSharp #36921: Deprecate yield from and @coroutine in asyncio https://bugs.python.org/issue36921 opened by asvetlov #36922: Implement Py_TPFLAGS_METHOD_DESCRIPTOR https://bugs.python.org/issue36922 opened by jdemeyer #36924: Simplify implementation of classmethod_descriptor.__call__ https://bugs.python.org/issue36924 opened by jdemeyer #36926: Implement methoddescr_call without _PyMethodDef_RawFastCallDic https://bugs.python.org/issue36926 opened by jdemeyer #36927: traceback docstrings should explicitly state return values ins https://bugs.python.org/issue36927 opened by Aaron Hall #36929: Other Python _io implementations may not expose _io in their t https://bugs.python.org/issue36929 opened by tekknolagi #36932: asyncio-task.rst could use proper deprecated-removed directive https://bugs.python.org/issue36932 opened by mbussonn #36933: sys.set_coroutine_wrapper documented as to be removed in 3.8 ( https://bugs.python.org/issue36933 opened by mbussonn #36935: bpo-35813 introduced usage of the deprecated PyErr_SetFromWind https://bugs.python.org/issue36935 opened by ZackerySpytz #36936: CALL_FUNCTION_KW opcode: keyword names must be non-empty https://bugs.python.org/issue36936 opened by jdemeyer #36937: New _PyObject_MakeTpCall() function https://bugs.python.org/issue36937 opened by jdemeyer #36940: Update Py_FrozenMain() for _PyCoreConfig (PEP 587) https://bugs.python.org/issue36940 opened by vstinner #36941: Windows build changes for Windows ARM64 https://bugs.python.org/issue36941 opened by Paul Monson #36942: Windows code changes for Windows ARM64 https://bugs.python.org/issue36942 opened by Paul Monson #36943: Windows test changes for Windows ARM64 https://bugs.python.org/issue36943 opened by Paul Monson #36944: Add support for ARM64 to libffi https://bugs.python.org/issue36944 opened by Paul Monson #36945: Add _PyPreConfig.configure_locale: allow to leave LC_CTYPE unc https://bugs.python.org/issue36945 opened by vstinner #36946: Possible signed integer overflow in slice handling https://bugs.python.org/issue36946 opened by ZackerySpytz #36947: [Good first issue] Fix 3.3.3.1 Metaclasses Documentation https://bugs.python.org/issue36947 opened by Nicholas Matthews #36948: NameError in urllib.request.URLopener.retrieve https://bugs.python.org/issue36948 opened by xtreak #36949: WeakSet.__repr__ and __str__ do not show contents of the set https://bugs.python.org/issue36949 opened by steve.dower Most recent 15 issues with no replies (15) ========================================== #36949: WeakSet.__repr__ and __str__ do not show contents of the set https://bugs.python.org/issue36949 #36948: NameError in urllib.request.URLopener.retrieve https://bugs.python.org/issue36948 #36944: Add support for ARM64 to libffi https://bugs.python.org/issue36944 #36943: Windows test changes for Windows ARM64 https://bugs.python.org/issue36943 #36940: Update Py_FrozenMain() for _PyCoreConfig (PEP 587) https://bugs.python.org/issue36940 #36933: sys.set_coroutine_wrapper documented as to be removed in 3.8 ( https://bugs.python.org/issue36933 #36932: asyncio-task.rst could use proper deprecated-removed directive https://bugs.python.org/issue36932 #36926: Implement methoddescr_call without _PyMethodDef_RawFastCallDic https://bugs.python.org/issue36926 #36924: Simplify implementation of classmethod_descriptor.__call__ https://bugs.python.org/issue36924 #36922: Implement Py_TPFLAGS_METHOD_DESCRIPTOR https://bugs.python.org/issue36922 #36921: Deprecate yield from and @coroutine in asyncio https://bugs.python.org/issue36921 #36905: test_typing.GetTypeHintTests.test_get_type_hints_modules_forwa https://bugs.python.org/issue36905 #36901: Fix leaks in /PC/bdist_wininst/install.c https://bugs.python.org/issue36901 #36896: clarify in types.rst that FunctionTypes & co constructors don https://bugs.python.org/issue36896 #36889: Merge StreamWriter and StreamReader into just asyncio.Stream https://bugs.python.org/issue36889 Most recent 15 issues waiting for review (15) ============================================= #36948: NameError in urllib.request.URLopener.retrieve https://bugs.python.org/issue36948 #36946: Possible signed integer overflow in slice handling https://bugs.python.org/issue36946 #36945: Add _PyPreConfig.configure_locale: allow to leave LC_CTYPE unc https://bugs.python.org/issue36945 #36943: Windows test changes for Windows ARM64 https://bugs.python.org/issue36943 #36942: Windows code changes for Windows ARM64 https://bugs.python.org/issue36942 #36941: Windows build changes for Windows ARM64 https://bugs.python.org/issue36941 #36937: New _PyObject_MakeTpCall() function https://bugs.python.org/issue36937 #36936: CALL_FUNCTION_KW opcode: keyword names must be non-empty https://bugs.python.org/issue36936 #36935: bpo-35813 introduced usage of the deprecated PyErr_SetFromWind https://bugs.python.org/issue36935 #36932: asyncio-task.rst could use proper deprecated-removed directive https://bugs.python.org/issue36932 #36929: Other Python _io implementations may not expose _io in their t https://bugs.python.org/issue36929 #36927: traceback docstrings should explicitly state return values ins https://bugs.python.org/issue36927 #36926: Implement methoddescr_call without _PyMethodDef_RawFastCallDic https://bugs.python.org/issue36926 #36924: Simplify implementation of classmethod_descriptor.__call__ https://bugs.python.org/issue36924 #36922: Implement Py_TPFLAGS_METHOD_DESCRIPTOR https://bugs.python.org/issue36922 Top 10 most discussed issues (10) ================================= #36867: Make semaphore_tracker track other system resources https://bugs.python.org/issue36867 11 msgs #36887: Add integer square root, math.isqrt https://bugs.python.org/issue36887 11 msgs #36763: PEP 587: Rework initialization API to prepare second version o https://bugs.python.org/issue36763 10 msgs #36917: ast.NodeVisitor no longer calls visit_Str https://bugs.python.org/issue36917 10 msgs #36870: test_asyncio: test_drain_raises() fails randomly on Windows https://bugs.python.org/issue36870 9 msgs #36880: Returning None from a callback with restype py_object decremen https://bugs.python.org/issue36880 9 msgs #36895: time.clock() marked for removal in 3.8 ?????still there. https://bugs.python.org/issue36895 9 msgs #36908: "This module is always available" is confusing https://bugs.python.org/issue36908 8 msgs #36945: Add _PyPreConfig.configure_locale: allow to leave LC_CTYPE unc https://bugs.python.org/issue36945 8 msgs #35495: argparse does not honor default argument for nargs=argparse.RE https://bugs.python.org/issue35495 7 msgs Issues closed (89) ================== #1257: atexit errors should result in nonzero exit code https://bugs.python.org/issue1257 closed by vstinner #2921: enable embedding: declare/#define only py* symbols in #include https://bugs.python.org/issue2921 closed by vstinner #3020: doctest should have lib2to3 integration https://bugs.python.org/issue3020 closed by scoder #3620: test_smtplib is flaky https://bugs.python.org/issue3620 closed by SilentGhost #6584: gzip module has no custom exception https://bugs.python.org/issue6584 closed by serhiy.storchaka #10991: trace fails when test imported a temporary file https://bugs.python.org/issue10991 closed by brett.cannon #13320: _remove_visual_c_ref in distutils.msvc9compiler causes DLL loa https://bugs.python.org/issue13320 closed by vstinner #13493: Import error with embedded python on AIX 6.1 https://bugs.python.org/issue13493 closed by vstinner #14228: Don't display traceback when import site is interrupted by CTR https://bugs.python.org/issue14228 closed by vstinner #14353: Proper gettext support in locale module https://bugs.python.org/issue14353 closed by benjamin.peterson #16100: Compiling vim with Python 3.3 support fails https://bugs.python.org/issue16100 closed by zach.ware #16961: No regression tests for -E and individual environment vars https://bugs.python.org/issue16961 closed by vstinner #17679: sysconfig generation uses some env variables multiple times https://bugs.python.org/issue17679 closed by vstinner #19175: Erroneous reference to "integer" in format string grammar https://bugs.python.org/issue19175 closed by cheryl.sabella #19216: stat cache for import bootstrap https://bugs.python.org/issue19216 closed by vstinner #21202: Naming a file` io.py` causes cryptic error message https://bugs.python.org/issue21202 closed by vstinner #21563: Segv during call to builtin_execfile in application embedding https://bugs.python.org/issue21563 closed by vstinner #22747: Interpreter fails in initialize on systems where HAVE_LANGINFO https://bugs.python.org/issue22747 closed by vstinner #23603: Embedding Python3.4 - PyUnicode_Check fails (MinGW-W64) https://bugs.python.org/issue23603 closed by vstinner #24280: Unable to install Python https://bugs.python.org/issue24280 closed by vstinner #24758: unittest.mock.Mock's new "unsafe" feature needs a better error https://bugs.python.org/issue24758 closed by xtreak #26122: Isolated mode doesn't ignore PYTHONHASHSEED https://bugs.python.org/issue26122 closed by vstinner #26480: add a flag that will not give the set a sys.stdin https://bugs.python.org/issue26480 closed by vstinner #26707: plistlib fails to parse bplist with 0x80 UID values https://bugs.python.org/issue26707 closed by serhiy.storchaka #26891: CPython doesn't work when you disable refcounting https://bugs.python.org/issue26891 closed by vstinner #29818: Py_SetStandardStreamEncoding leads to a memory error in debug https://bugs.python.org/issue29818 closed by vstinner #30464: gammavariate has a wrong comment https://bugs.python.org/issue30464 closed by mark.dickinson #30633: Python 3.6.1 installation issues on OpenSuse 42.1: ModuleNotFo https://bugs.python.org/issue30633 closed by vstinner #31473: Debug hooks on memory allocators are not thread safe (serialno https://bugs.python.org/issue31473 closed by vstinner #32177: spammers mine emails from bugs.python.org https://bugs.python.org/issue32177 closed by SilentGhost #32231: -bb option should override -W options https://bugs.python.org/issue32231 closed by vstinner #32832: doctest should support custom ps1/ps2 prompts https://bugs.python.org/issue32832 closed by cheryl.sabella #32995: Add a glossary entry for context variables https://bugs.python.org/issue32995 closed by cheryl.sabella #33123: Path.unlink should have a missing_ok parameter https://bugs.python.org/issue33123 closed by pitrou #33303: ElementTree Comment text isn't escaped https://bugs.python.org/issue33303 closed by scoder #33882: doc Mention breakpoint() in debugger-related FAQ https://bugs.python.org/issue33882 closed by matrixise #34152: performance of some list slice assignment margin cases can be https://bugs.python.org/issue34152 closed by pablogsal #34408: possible null pointer dereference in pystate.c https://bugs.python.org/issue34408 closed by pablogsal #34424: Unicode names break email header https://bugs.python.org/issue34424 closed by cheryl.sabella #34484: Unicode HOWTO incorrectly refers to Private Use Area for surro https://bugs.python.org/issue34484 closed by akuchling #34509: Starting to use gcc-8 on CI https://bugs.python.org/issue34509 closed by zach.ware #34579: test_embed.InitConfigTests fail on AIX https://bugs.python.org/issue34579 closed by vstinner #34682: Typo reports on docs@ https://bugs.python.org/issue34682 closed by cheryl.sabella #35138: timeit documentation should have example with function argumen https://bugs.python.org/issue35138 closed by vstinner #35589: BaseSelectorEventLoop.sock_sendall() performance regression: e https://bugs.python.org/issue35589 closed by asvetlov #36008: [good first issue] Update documentation for 3.8 https://bugs.python.org/issue36008 closed by cheryl.sabella #36471: PEP 432, PEP 587: Add _Py_RunMain() https://bugs.python.org/issue36471 closed by vstinner #36474: RecursionError resets trace function set via sys.settrace https://bugs.python.org/issue36474 closed by blueyed #36594: Undefined behavior due to incorrect usage of %p in format stri https://bugs.python.org/issue36594 closed by brett.cannon #36618: clang expects memory aligned on 16 bytes, but pymalloc aligns https://bugs.python.org/issue36618 closed by vstinner #36620: Documentation missing parameter for Itertools.zip_longest https://bugs.python.org/issue36620 closed by rhettinger #36684: codecov.io code coverage has not updated since 2019-04-13 https://bugs.python.org/issue36684 closed by inada.naoki #36725: Reference leak regression with Python3.8a3 https://bugs.python.org/issue36725 closed by vstinner #36728: Remove PyEval_ReInitThreads() from the public C API https://bugs.python.org/issue36728 closed by vstinner #36748: Optimize textio write buffering https://bugs.python.org/issue36748 closed by inada.naoki #36749: PPC64 AIX 3.x: compilation issue, linker fails to locate symbo https://bugs.python.org/issue36749 closed by vstinner #36751: Changes in the inspect module for PEP 570 https://bugs.python.org/issue36751 closed by pablogsal #36760: subprocess.run fails with capture_output=True and stderr=STDOU https://bugs.python.org/issue36760 closed by gregory.p.smith #36789: Unicode HOWTO incorrectly states that UTF-8 contains no zero b https://bugs.python.org/issue36789 closed by cheryl.sabella #36791: sum() relies on C signed overflow behaviour https://bugs.python.org/issue36791 closed by serhiy.storchaka #36799: Typo in ctypes documentation https://bugs.python.org/issue36799 closed by matrixise #36807: IDLE doesn't call os.fsync() https://bugs.python.org/issue36807 closed by terry.reedy #36818: Add PyInterpreterState.runtime. https://bugs.python.org/issue36818 closed by vstinner #36822: Minor grammatical fix in glossary.rst https://bugs.python.org/issue36822 closed by pablogsal #36845: ipaddres.IPv4Network and ipaddress.IPv6Network tuple construct https://bugs.python.org/issue36845 closed by inada.naoki #36875: argparse does not ship with translations https://bugs.python.org/issue36875 closed by terry.reedy #36883: ???ctypes?????????C?????????c_char_p?????????????????? https://bugs.python.org/issue36883 closed by anthonypjshaw #36884: DeprecationWarning in test_asyncio.test_pep492.StreamReaderTes https://bugs.python.org/issue36884 closed by asvetlov #36886: Failed to construct CodeType on Python-3.8.0a4 https://bugs.python.org/issue36886 closed by SilentGhost #36899: datetime utcfromtimestamp ignores astimezone https://bugs.python.org/issue36899 closed by SilentGhost #36900: Use _PyCoreConfig rather than global configuration variables https://bugs.python.org/issue36900 closed by vstinner #36902: spam https://bugs.python.org/issue36902 closed by SilentGhost #36903: ResourceWarning in test_logging.HandlerTest.test_post_fork_chi https://bugs.python.org/issue36903 closed by xtreak #36909: LastUpdatedOrderedDict recipe uses super() unnecessarily https://bugs.python.org/issue36909 closed by rhettinger #36911: ast.parse outputs ast.Strs which do not differentiate between https://bugs.python.org/issue36911 closed by eric.smith #36912: Can't left-align strings using f-strings or format() https://bugs.python.org/issue36912 closed by gvanrossum #36914: zipimport does not load ZIP files produced by GitHub https://bugs.python.org/issue36914 closed by SilentGhost #36915: regrtest: when interrupted, temporary directory is not removed https://bugs.python.org/issue36915 closed by vstinner #36916: Swallow unhandled exception report introduced by 36802 https://bugs.python.org/issue36916 closed by asvetlov #36920: inspect.getcallargs sees optional arg to builtin as required https://bugs.python.org/issue36920 closed by brett.cannon #36923: Implemented __str__ for zip and map objects https://bugs.python.org/issue36923 closed by SilentGhost #36925: python3.7.3 can't work after MacOS upgrade from v10.14.4 to v1 https://bugs.python.org/issue36925 closed by ned.deily #36928: linkt threading.settrace to sys.settrace https://bugs.python.org/issue36928 closed by SilentGhost #36930: Windows proxy settings automatically used ... partly https://bugs.python.org/issue36930 closed by iwetzel #36931: json lib doesnt want to load from file https://bugs.python.org/issue36931 closed by zach.ware #36934: C API Function PyLong_AsDouble Returning Wrong Value https://bugs.python.org/issue36934 closed by sajjadfx #36938: Py_XDECREF on PyUnicodeobject raises SIGSEGV signal https://bugs.python.org/issue36938 closed by serhiy.storchaka #36939: Allow to use shutil.copytree for existing destination director https://bugs.python.org/issue36939 closed by jack1142 #1465646: test_grp & test_pwd fail https://bugs.python.org/issue1465646 closed by matrixise From qiang.fang at zoho.com.cn Fri May 17 22:26:04 2019 From: qiang.fang at zoho.com.cn (Q) Date: Sat, 18 May 2019 10:26:04 +0800 Subject: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation In-Reply-To: References: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> Message-ID: <16ac8c2b2b4.ffb0004b5281.6365385802178906075@zoho.com.cn> >From the viewpoint of the package user, people don't want to change every import to "require a version at runtime". If "set up a python path in a wrapper script" is a better strategy, can you please give an example for the following use case: The abaqus python distribution has libA 1.0, the external package to be installed is LibB, which requires libA2.0, and we don't want to change every import in LibB to be import libA 1.0, because the source code may have not been provided. In emacs, you can always advice a command to change its behaviour, which makes it very user friendly, and that what all the following suggestion is about. I have no idea at all how it could be implemented though. https://github.com/mitsuhiko/multiversion/issues/1 "The work around is to let a version to be set within a package and have that propagate to all modules in that package. For example in the root init.py if I set tonado version to be 2.2.1 then all modules in that package will use tornado 2.2.1 when I import tornado." ---- ? ???, 2019-05-17 23:38:55 Daniel Holth ?? ---- This sounds exactly like what people used to do with eggs. You could have multiple versions of a package on the path as eggs and then require a version at runtime. The approach has problems. Ruby also abandoned a strategy where random app code depends on package management code at runtime. One better strategy is to set up a python path in a wrapper script. On Fri, May 17, 2019, 11:27 Brett Cannon wrote: Thanks for the idea but there are currently no plans to support such a feature. If you would like to see it then you will need to write a PEP with a proof-of-concept to demonstrate how you would expect such a feature to work. On Fri., May 17, 2019, 07:55 Q via Python-Dev, wrote: A lot of the Python code we use in production are used directly as imports in other python distributions (such as the python comes with the finite element software Abaqus and MSC Marc), many packages (such as matplotlib, numpy) that may have varying versioned dependencies. I was wondering if this could be expanded to allow a version to be set within a package and have that propagate to all modules in that package. For example in the root init.py if I set multiversion(tornado, 2.2.1) then all modules in that package will use tornado 2.2.1 when I import tornado. See a relevant issue on github: https://github.com/mitsuhiko/multiversion/issues/1 Thank you! Qiang _______________________________________________ Python-Dev mailing list mailto:Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org _______________________________________________ Python-Dev mailing list mailto:Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From qiang.fang at zoho.com.cn Fri May 17 22:32:14 2019 From: qiang.fang at zoho.com.cn (Q) Date: Sat, 18 May 2019 10:32:14 +0800 Subject: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation In-Reply-To: <16ac8c2b2b4.ffb0004b5281.6365385802178906075@zoho.com.cn> References: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> <16ac8c2b2b4.ffb0004b5281.6365385802178906075@zoho.com.cn> Message-ID: <16ac8c85883.1228864315288.5365657524062277459@zoho.com.cn> Sorry, there is an error in my last post. It's corrected as: The abaqus python distribution has libA 1.0, the external package to be installed is LibB, which requires libA2.0, and we don't want to change every import in LibB to be import /somewhere/libA2.0, because the source code may have not been provided. ---- ? ???, 2019-05-18 10:26:04 Q ?? ---- >From the viewpoint of the package user, people don't want to change every import to "require a version at runtime". If "set up a python path in a wrapper script" is a better strategy, can you please give an example for the following use case: The abaqus python distribution has libA 1.0, the external package to be installed is LibB, which requires libA2.0, and we don't want to change every import in LibB to be import libA 1.0, because the source code may have not been provided. In emacs, you can always advice a command to change its behaviour, which makes it very user friendly, and that what all the following suggestion is about. I have no idea at all how it could be implemented though. https://github.com/mitsuhiko/multiversion/issues/1 "The work around is to let a version to be set within a package and have that propagate to all modules in that package. For example in the root init.py if I set tonado version to be 2.2.1 then all modules in that package will use tornado 2.2.1 when I import tornado." ---- ? ???, 2019-05-17 23:38:55 Daniel Holth ?? ---- This sounds exactly like what people used to do with eggs. You could have multiple versions of a package on the path as eggs and then require a version at runtime. The approach has problems. Ruby also abandoned a strategy where random app code depends on package management code at runtime. One better strategy is to set up a python path in a wrapper script. On Fri, May 17, 2019, 11:27 Brett Cannon wrote: Thanks for the idea but there are currently no plans to support such a feature. If you would like to see it then you will need to write a PEP with a proof-of-concept to demonstrate how you would expect such a feature to work. On Fri., May 17, 2019, 07:55 Q via Python-Dev, wrote: A lot of the Python code we use in production are used directly as imports in other python distributions (such as the python comes with the finite element software Abaqus and MSC Marc), many packages (such as matplotlib, numpy) that may have varying versioned dependencies. I was wondering if this could be expanded to allow a version to be set within a package and have that propagate to all modules in that package. For example in the root init.py if I set multiversion(tornado, 2.2.1) then all modules in that package will use tornado 2.2.1 when I import tornado. See a relevant issue on github: https://github.com/mitsuhiko/multiversion/issues/1 Thank you! Qiang _______________________________________________ Python-Dev mailing list mailto:Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/brett%40python.org _______________________________________________ Python-Dev mailing list mailto:Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/dholth%40gmail.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From thatiparthysreenivas at gmail.com Sat May 18 02:38:41 2019 From: thatiparthysreenivas at gmail.com (Srinivas Reddy Thatiparthy) Date: Sat, 18 May 2019 12:08:41 +0530 Subject: [Python-Dev] Merge ? Message-ID: Pls take a look here https://github.com/python/cpython/pull/9882 If it not okay to document that >>> mock.assert_has_calls([]) does not raise an exception; then i will go ahead and close the pr. Best Regards, Srinivas Reddy Thatiparthy. -------------- next part -------------- An HTML attachment was scrubbed... URL: From senthil at uthcode.com Sat May 18 05:12:51 2019 From: senthil at uthcode.com (Senthil Kumaran) Date: Sat, 18 May 2019 10:12:51 +0100 Subject: [Python-Dev] [python-committers] PEP 581 (Using GitHub issues for CPython) is accepted In-Reply-To: References: <7407D51F-AC46-4423-AF4A-63940F80D629@python.org> <20190515104830.5b307230@fsol> Message-ID: On Wed, May 15, 2019 at 6:44 PM Ezio Melotti wrote: > > I share the same concerns: 1) the PEP contains several factual errors. I pointed this out during > the core-sprints last year and more recently Berker pointed out some > on GitHub: https://github.com/python/peps/pull/1013 ; > 4) Berker is/was writing a competing PEP, in order to address the > problems of PEP 581 more effectively since his comments on GitHub > haven't been addressed; > This concerns me a bit. The PEP/announcement acknowledges the work Ezio and Berker. However, it does not express if the PEP had addressed their review comments or had the current maintainers on-board with the proposal. I was of the assumption that if the current maintainers (/domain experts) were on-board, then it will be easier for the rest of us to adopt the new changes. -------------- next part -------------- An HTML attachment was scrubbed... URL: From dan at danryan.co Sat May 18 03:18:22 2019 From: dan at danryan.co (Dan Ryan) Date: Sat, 18 May 2019 03:18:22 -0400 Subject: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation In-Reply-To: <16ac8c85883.1228864315288.5365657524062277459@zoho.com.cn> References: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> <16ac8c2b2b4.ffb0004b5281.6365385802178906075@zoho.com.cn> <16ac8c85883.1228864315288.5365657524062277459@zoho.com.cn> Message-ID: <002301d50d49$e36ddf60$aa499e20$@danryan.co> It probably makes the most sense to formally outline all of the behavioral changes, expectations, and constraints you are asking for as Brett said because dependency and package management are tricky and it's not clear (to me at least) what the request is, what problem is being encountered, and how it is proposed that it be resolved. First glance makes it sound a bit like the a prerequisite for this request would be allowing multiple versions of the same package to be installed in a given python environment to begin with, or else some runtime hacking of paths would be necessary. I may be misunderstanding, but that's why a formal approach for something like this might make some sense Dan Ryan gh: @techalchemy // e: dan at danryan.co From: Q [mailto:qiang.fang at zoho.com.cn] Sent: Friday, May 17, 2019 10:32 PM To: Daniel Holth Cc: Brett Cannon; Python-Dev Subject: Re: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation Sorry, there is an error in my last post. It's corrected as: The abaqus python distribution has libA 1.0, the external package to be installed is LibB, which requires libA2.0, and we don't want to change every import in LibB to be import /somewhere/libA2.0, because the source code may have not been provided. ---- ? ???, 2019-05-18 10:26:04 Q ?? ---- >From the viewpoint of the package user, people don't want to change every import to "require a version at runtime". If "set up a python path in a wrapper script" is a better strategy, can you please give an example for the following use case: The abaqus python distribution has libA 1.0, the external package to be installed is LibB, which requires libA2.0, and we don't want to change every import in LibB to be import libA 1.0, because the source code may have not been provided. In emacs, you can always advice a command to change its behaviour, which makes it very user friendly, and that what all the following suggestion is about. I have no idea at all how it could be implemented though. https://github.com/mitsuhiko/multiversion/issues/1 "The work around is to let a version to be set within a package and have that propagate to all modules in that package. For example in the root init.py if I set tonado version to be 2.2.1 then all modules in that package will use tornado 2.2.1 when I import tornado." ---- ? ???, 2019-05-17 23:38:55 Daniel Holth ?? ---- This sounds exactly like what people used to do with eggs. You could have multiple versions of a package on the path as eggs and then require a version at runtime. The approach has problems. Ruby also abandoned a strategy where random app code depends on package management code at runtime. One better strategy is to set up a python path in a wrapper script. On Fri, May 17, 2019, 11:27 Brett Cannon wrote: Thanks for the idea but there are currently no plans to support such a feature. If you would like to see it then you will need to write a PEP with a proof-of-concept to demonstrate how you would expect such a feature to work. On Fri., May 17, 2019, 07:55 Q via Python-Dev, wrote: A lot of the Python code we use in production are used directly as imports in other python distributions (such as the python comes with the finite element software Abaqus and MSC Marc), many packages (such as matplotlib, numpy) that may have varying versioned dependencies. I was wondering if this could be expanded to allow a version to be set within a package and have that propagate to all modules in that package. For example in the root init.py if I set multiversion(tornado, 2.2.1) then all modules in that package will use tornado 2.2.1 when I import tornado. See a relevant issue on github: https://github.com/mitsuhiko/multiversion/issues/1 Thank you! Qiang From gregory.szorc at gmail.com Sun May 19 14:58:31 2019 From: gregory.szorc at gmail.com (Gregory Szorc) Date: Sun, 19 May 2019 11:58:31 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> Message-ID: <3a18be48-c63f-c70b-2f84-3a657a482df5@gmail.com> On 5/16/2019 10:25 AM, Victor Stinner wrote: > Le jeu. 16 mai 2019 ? 16:10, Thomas Wouters a ?crit : >>> Would you be ok with a "PyConfig_Init(PyConfig *config);" function >>> which would initialize all fields to theire default values? Maybe >>> PyConfig_INIT should be renamed to PyConfig_STATIC_INIT. >>> >>> You can find a similar API for pthread mutex, there is a init function >>> *and* a macro for static initialization: >>> >>> int pthread_mutex_init(pthread_mutex_t *restrict mutex, >>> const pthread_mutexattr_t *restrict attr); >>> >>> pthread_mutex_t mutex = PTHREAD_MUTEX_INITIALIZER; >> >> >> This was going to be my suggestion as well: for any non-trivial macro, we should have a function for it instead. > > Ok, I will do that. > > >> I would also point out that PEP 587 has a code example that uses PyWideStringList_INIT, but that macro isn't mention anywhere else. > > Oh, I forgot to better document it. Well, the macro is trivial: > > #define _PyWstrList_INIT (_PyWstrList){.length = 0, .items = NULL} > > For consistency, I prefer to not initialize manually these fields, but > use a macro instead. > > (Variables are allocated on the stack and so *must* be initialized.) > > >> The PEP is a bit unclear as to the semantics of PyWideStringList as a whole: the example uses a static array with length, but doesn't explain what would happen with statically allocated data like that if you call the Append or Extend functions. It also doesn't cover how e.g. argv parsing would remove items from the list. (I would also suggest the PEP shouldn't use the term 'list', at least not unqualified, if it isn't an actual Python list.) > > Calling PyWideStringList_Append() or PyWideStringList_Insert() on a > "constant" list will crash: don't do that :-) > > I tried to explain the subtle details of "constant" vs "dynamic" > configurations in "Initialization with constant PyConfig" and "Memory > allocations and Py_DecodeLocale()" functions. > > A "constant" PyWideStringList must not be used with a "dynamic" > PyConfig: otherwise, PyConfig_Clear() will crash as well. > > I would prefer to have separated "const PyWideStringList" and "const > PyConfig" types, but the C language doesn't convert "wchat_*" to > "const wchar_t*" when you do that. We would need duplicated > PyConstantWideStringList and PyConstantConfig structures, which would > require to be "casted" to PyWideStringList and PyConfig internally to > reuse the same code for constant and dynamic configuration. > > If you consider that the specific case of "constant configuration" > adds too much burden / complexity, we mgiht remove it and always > require to use dynamic configuration. > > Right now, Programs/_testembed.c almost uses only "constant" > configuration. Using dynamic memory would make the code longer: need > to handle memory allocation failures. > > >> I understand the desire to make static allocation and initialisation possible, but since you only need PyWideStringList for PyConfig, not PyPreConfig (which sets the allocator), perhaps having a PyWideStringList_Init(), which copies memory, and PyWideStringList_Clear() to clear it, would be better? > > Do you mean to always require to build dynamic lists? Said > differently, not allow to write something like the following code? > > static wchar_t* argv[] = { > L"python3", > L"-c", > L"pass", > L"arg2", > }; > > _PyCoreConfig config = _PyCoreConfig_INIT; > config.argv.length = Py_ARRAY_LENGTH(argv); > config.argv.items = argv; > > >>> If the private field "_init_main" of the PEP 587 is set to 0, >>> Py_InitializeFromConfig() stops at the "core" phase (in fact, it's >>> already implemented!). But I didn't implement yet a >>> _Py_InitializeMain() function to "finish" the initialization. Let's >>> say that it exists, we would get: >>> >>> --- >>> PyConfig config = PyConfig_INIT; >>> config._init_main = 0; >>> PyInitError err = Py_InitializeFromConfig(&config); >>> if (Py_INIT_FAILED(err)) { >>> Py_ExitInitError(err); >>> } >>> >>> /* add your code to customize Python here */ >>> /* calling PyRun_SimpleString() here is safe */ >>> >>> /* finish Python initialization */ >>> PyInitError err = _Py_InitializeMain(&config); >>> if (Py_INIT_FAILED(err)) { >>> Py_ExitInitError(err); >>> } >>> --- >>> >>> Would it solve your use case? >> >> >> FWIW, I understand the need here: for Hermetic Python, we solved it by adding a new API similar to PyImport_AppendInittab, but instead registering a generic callback hook to be called *during* the initialisation process: after the base runtime and the import mechanism are initialised (at which point you can create Python objects), but before *any* modules are imported. We use that callback to insert a meta-importer that satisfies all stdlib imports from an embedded archive. (Using a meta-importer allows us to bypass the fileysystem altogether, even for what would otherwise be failed path lookups.) >> >> As I mentioned, Hermetic Python was originally written for Python 2.7, but this approach works fine with a frozen importlib as well. The idea of 'core' and 'main' initialisation will likely work for this, as well. > > Well, even if it's not part of the PEP 587, I just implemented it > anyway while fixing a bug: > https://github.com/python/cpython/commit/9ef5dcaa0b3c7c7ba28dbb3ec0c9507d9d05e3a9 > > Example: > > static int test_init_main(void) > { > _PyCoreConfig config = _PyCoreConfig_INIT; > configure_init_main(&config); > config._init_main = 0; > > _PyInitError err = _Py_InitializeFromConfig(&config); > if (_Py_INIT_FAILED(err)) { > _Py_ExitInitError(err); > } > > /* sys.stdout don't exist yet: it is created by _Py_InitializeMain() */ > int res = PyRun_SimpleString( > "import sys; " > "print('Run Python code before _Py_InitializeMain', " > "file=sys.stderr)"); > if (res < 0) { > exit(1); > } > > err = _Py_InitializeMain(); > if (_Py_INIT_FAILED(err)) { > _Py_ExitInitError(err); > } > > return _Py_RunMain(); > } > > As you can see, it's possible execute Python between "core" and "main" > initialization phases. Moreover, I even fixed Python to be able to use > "import sys" before the "main" initialization phase ;-) (Only builtin > and frozen modules are available at this stage.) > > Again, I'm not comfortable to make PyConfig._init_main and > _Py_InitializeMain() public, because I consider that they are too > experimental and we don't have enough time to discuss what is the > "core" initialization phase exactly. It sounds like PyOxidizer and Hermetic Python are on the same page and we're working towards a more official solution. But I want to make sure by explicitly stating what PyOxidizer is doing. Essentially, to facilitate in-memory import, we need to register a custom sys.meta_path importer *before* any file-based imports are attempted. In other words, we don't want the PathFinder registered on sys.meta_path to be used. Since we don't have a clear "hook point" to run custom code between init_importlib() (which is where the base importlib system is initialized) and _PyCodecRegistry_Init() (which has the first import of a .py-backed module - "encodings"), my current hack in PyOxidizer is to compile a modified version of the importlib._bootstrap_external module which contains my custom MemoryImporter. I inject this custom version at run-time by updating the global frozen modules table and the Python initialization mechanism executes my custom code when the _frozen_importlib_external module is imported/executed as part of init_importlib_external(). The CPython API can facilitate making this less hacky by enabling embedders to run custom code between importlib initialization but before any path-based modules are imported. I /think/ providing a 2-phase initialization that stops between _Py_InitializeCore() and _Py_InitializeMainInterpreter() would get the job done for PyOxidizer today. But it may be a bit more complicated than that for others (or possibly me in the future) since importlib is initialized in both these phases. The "external" importlib bits providing the path-based importer are initialized later during _Py_InitializeMainInterpreter. It's quite possible we would want to muck around with the state of external importers in our initialization "hook" and if they aren't loaded yet... Furthermore, new_interpreter() makes both importlib initialization function calls at the same time. We would need to inject a custom meta path importer for sub-interpreters, so we would need a code injection point in new_interpreter() as well. Installing a custom importlib._bootstrap_external/_frozen_importlib_external module and globally changing the code that runs during importlib init gets the job. But it is hacky. FWIW my notes on how this all works (in Python 3.7) are at https://github.com/indygreg/PyOxidizer/blob/57c823d6ca2321d12067bf36603a9a0ad2320c75/docs/technotes.rst, https://github.com/indygreg/PyOxidizer/blob/57c823d6ca2321d12067bf36603a9a0ad2320c75/README.rst#how-it-works, and https://gregoryszorc.com/blog/2018/12/18/distributing-standalone-python-applications/. Again, it sounds like we're working towards a robust solution. I just wanted to brain dump to make sure we are. >> Other questions/comments about PEP 587: >> >> I really like the PyInitError struct. I would like more functions to use it, e.g. the PyRrun_* "very high level" API, which currently calls exit() for you on SystemExit, and returns -1 without any other information on error. For those, I'm not entirely sure 'Init' makes sense in the name... but I can live with it. > > PyInitError structure can be renamed PyError, but it should only be > used with functions which can exit Python. In short, are you talking > "The Very High Level Layer" of the C API? > https://docs.python.org/dev/c-api/veryhigh.html > > One issue is that I dislike adding new functions to the C API, but it > seems like we should add a few to provide a better API for embedded > Python. libpython must never exit the process! (or only when you > explicity asks that :-)) > > Note: PyRun_SimpleStringFlags() is a wrapper which makes > PyRun_StringFlags() usage easier. PyRun_StringFlags() doesn't handle > the exception and so let you decide how to handle it. > > >> A couple of things are documented as performing pre-initialisation (PyConfig_SetBytesString, PyConfig_SetBytesArgv). I understand why, but I feel like that might be confusing and error-prone. Would it not be better to have them fail if pre-initialisation hasn't been performed yet? > > It's easier to modify the code to fail with an error if Python is not > pre-initialized. > > I propose to implicitly pre-initialize Python to make the API easier > to use. In practice, you rarely have to explicitly pre-initialize > Python. The default PyPreConfig is just fine for almost all use cases, > especially since Nick Coghlan and me decided to disable C locale > coercion and UTF-8 Mode by default. You now have to opt-in to enable > these encoding features. > > >> The buffered_stdio field of PyConfig mentions stdout and stderr, but not stdin. Does it not affect stdin? > > Extract of create_stdio(): > > /* stdin is always opened in buffered mode, first because it shouldn't > make a difference in common use cases, second because TextIOWrapper > depends on the presence of a read1() method which only exists on > buffered streams. > */ > > Note: Unbuffered stdin doesn't magically make the producer on the > other side of a pipe flushing its (stdout/stderr) buffer more > frequently :-) > > >> (Many of the fields could do with a bit more explicit documentation, to be honest.) > > Well, 2 years ago, almost no configuration parameter was documented > :-) I helped to document "Global configuration variables" at: > https://docs.python.org/dev/c-api/init.html#global-configuration-variables > > I had to reverse engineer the code to be able to document it :-D > > Right now, my reference documentation lives in > Include/cpython/coreconfig.h. Some parameters are better documented > there, than in the PEP. I can try to enhance the documentation in the > PEP. > > >> The configure_c_stdio field of PyConfig sounds like it might not set sys.stdin/stdout/stderr. That would be new behaviour, but configure_c_stdio doesn't have an existing equivalence, so I'm not sure if that's what you meant or not. > > In Python 3.7, only Py_Main() configured C standard streams. > > I moved the code into _PyCoreConfig_Write() which is called by > _Py_InitializeFromConfig() and so by Py_Initialize() as well. > > My intent is to be able to get the same behavior using Py_Initialize() > + Py_RunMain(), than using Py_Main(). > > Said differently, Python 3.8 now always configures C standard streams. > Maybe I should modify the configure_c_stdio default value to 0, and > only enable it by default in Py_Main()? > > Honestly, I'm a little bit confused here. I'm not sure what is the > expected behavior. Usually, in case of doubt, I look at the behavior > before my refactoring. The old behaviour was that only Py_Main() > configured C standard streams. Maybe I should restore this behavior. > > But to build a customized Python which should behave as the regular > Python, you would like opt-in for configure_c_stdio=1. > > Maybe we need a function to set the configuration to get "regular > Python" behavior? > > Something like: PyConfig_SetRegularPythonBehavior()? (sorry for the silly name!) > > >> The dll_path field of PyConfig says "Windows only". Does that meant the struct doesn't have that field except in a Windows build? Or is it ignored, instead? If it doesn't have that field at all, what #define can be used to determine if the PyConfig struct will have it or not? > > The field doesn't exist on non-Windows platforms. > > I chose to expose it to let the developer chooses where Python looks for DLL. > > But Steve just said (in an email below) that there is no reason to > make it configurable. In that case, I will make it internal again. It > seems like I misunderstood the purpose of this parameter. > > >> It feels a bit weird to have both 'inspect' and 'interactive' in PyConfig. Is there a substantive difference between them? Is this just so you can easily tell if any of run_module / run_command / run_filename are set? > > In Python 3.7, there are Py_InspectFlag and Py_InteractiveFlag. > > If "interactive" parameter is non-zero, C standard streams are > configured as buffered. It is also used to decide if stdin is > considered as interactive or not: > > /* Return non-zero is stdin is a TTY or if -i command line option is used */ > static int > stdin_is_interactive(const _PyCoreConfig *config) > { > return (isatty(fileno(stdin)) || config->interactive); > } > > The "inspect" parameter is used to decide if we start a REPL or not. > > The "-i" command line option sets inspect (Py_InspectFlag) and > interactive (Py_InteractiveFlag) to 1. > > These flags are exposed at Python level as sys.flags.inspect and > sys.flags.interactive. > > ... Honestly, I'm not sure if there is a real difference between these > two flags, but they are exposed and exist for years... so I decided to > keep them. > > >> "module_search_path_env" sounds like an awkward and somewhat misleading name for the translation of PYTHONPATH. Can we not just use, say, pythonpath_env? I expect the intended audience to know that PYTHONPATH != sys.path. > > Sure, I can rename it. > > >> The module_search_paths field in PyConfig doesn't mention if it's setting or adding to the calculated sys.path. As a whole, the path-calculation bits are a bit under-documented. > > Py_InitializeFromConfig() sets sys.path from module_search_paths. > > sys.path doesn't exist before Py_InitializeFromConfig() is called. > > >> Since this is an awkward bit of CPython, it wouldn't hurt to mention what "the default path configuration" does (i.e. search for python's home starting at program_name, add fixed subdirs to it, etc.) > > Oh, that's a big task :-) Nobody knows what getpath.c and getpathp.c do :-D > > >> Path configuration is mentioned as being able to issue warnings, but it doesn't mention *how*. It can't be the warnings module at this stage. I presume it's just printing to stderr. > > First, I didn't know, but I just saw that it's only on Unix > (getpath.c). On Windows (getpathp.c), no warning is emitted. > > The warning is written into C stderr. > > The flag isn't new: it's based on Py_FrozenFlag. When I looked at how > Python is embedded, I was surprised by the number of applications > setting Py_FrozenFlag to 1 to suppress these warnings. > > >> Regarding Py_RunMain(): does it do the right thing when something calls PyErr_Print() with SystemExit set? (I mentioned last week that PyErr_Print() will call C's exit() in that case, which is obviously terrible for embedders.) > > I spent a significant amount of time to ensure that > Py_InitializeFromConfig() and Py_RunMain() don't exit directly, but > return a proper failure or exit code. For example, Python 3.6 contains > around 319 calls to Py_FatalError(). The master branch contains around > 181 calls to Py_FatalError(): still a lot, but I converted 138 calls > to _Py_INIT_ERR() ;-) > > The work is not complete: I just checked, Py_RunMain() still calls > directly PyErr_Print() at many places. Well, the code can be fixed, > and it's not directly related to the PEP, is it? The issue already > existed in Python 3.7 with Py_Main(). > > >> Regarding isolated_mode and the site module, should we make stronger guarantees about site.py's behaviour being optional? The problem with site is that it does four things that aren't configurable, one of which is usually very desirable, one of which probably doesn't matter to embedders, and two that are iffy: sys.path deduplication and canonicalisation (and fixing up __file__/__cached__ attributes of already-imported modules); adding site-packages directories; looking for and importing sitecustomize.py; executing .pth files. The site module doesn't easily allow doing only some of these. (user-site directories are an exception, as they have their own flag, so I'm not listing that here.) With Hermetic Python we don't care about any of these (for a variety of different reasons), but I'm always a little worried that future Python versions would add behaviour to site that we *do* need. > > Honestly, I would prefer to simply remove the site module, I dislike > it because it makes Python startup way slower :-) ... But well, it > does a few important things :-) > > About the PEP 587: PyConfig.user_site_directory is exported as > sys.flags.no_user_site (negative value) which is used by the site > module. > > I'm not sure if you are asking me to modify my PEP, or if it's more a > general remark. The PEP 587 gives control on how sys.path is > initialized. > > In the "Isolate Python" section, I suggest to set the "isolated" > parameter to 1 which imply setting user_site_directory to 0. So > sys.path isn't modified afterwards. What you pass to PyConfig is what > you get in sys.path in this case. Regarding site.py, I agree it is problematic for embedding scenarios. Some features of site.py can be useful. Others aren't. It would be useful to have more granular control over which bits of site.run are run. My naive suggestion would be to add individual flags to control which functions site.py:main() runs. That way embedders can cherry-pick site.py features without having to manually import the module and call functions within. That feels much more robust for long-term maintainability. While I was reading replies on this thread, a few other points regarding embedding crept to my mind. I apologize in advance for the feature creep... Regarding Python calling exit(), this is problematic for embedding scenarios. This thread called attention to exit() during interpreter initialization. But it is also a problem elsewhere. For example, PyErr_PrintEx() will call Py_Exit() if the exception is a SystemExit. There's definitely room to improve the exception handling mechanism to give embedders better control when SystemExit is raised. As it stands, we need to check for SystemExit manually and reimplement _Py_HandleSystemExit() to emulate its behavior for e.g. exception value handling (fun fact: you can pass non-None, non-integer values to sys.exit/SystemExit). Regarding builtin and frozen modules, they both are backed by global arrays (PyImport_Inittab and PyImport_FrozenModules, respectively). In my initial reply I mentioned that this seemed problematic because I think a goal of initialization configuration overhaul should be to remove dependence on global variables. In addition to that concern, I'd like to point out that the way things work today is the BuiltinImporter and FrozenImporter meta path importers test for module availability by iterating these global arrays and calling _PyUnicode_EqualToASCIIString() on each member. This is done via import.c:is_builtin() and import.c:find_frozen(), respectively (both C functions are exported to Python and called as part of their respective find_spec() implementations). BuiltinImporter and FrozenImporter are always registered as the first two sys.meta_path importers. This behavior means every "import" results in an array walk + string compare over these two arrays. Yes, this is done from C and should be reasonably fast. But it feels inefficient to me. Slow import performance is a common complaint against Python and anything that can be done to minimize overhead could be useful. Having a more efficient member lookup for BuiltinImporter and FrozenImporter might shave off a millisecond or two from startup. This would require some kind of derived data structure. Unfortunately, as long as there is a global data structure that can be mutated any time (the API contract doesn't prohibit modifying these global arrays after initialization), you would need to check for "cache invalidation" on every lookup, undermining performance benefits. But if the interpreter config contained references to the builtin and frozen module arrays and refused to allow them to be modified after initialization, initialization could build a derived data structure used for efficient lookups and all the problems go away! What I'm trying to say is that moving the builtin and frozen modules arrays to the initialization config data structure is not only the right thing to do from an architectural perspective, but it can also open the door to optimizing import performance. FWIW PyOxidizer uses a Rust HashMap to index modules data, so testing for module presence is O(1)~. It's on my laundry list of TODOs to create an UberImporter that indexes every known module at startup and proxies to BuiltinImporter and FrozenImporter as necessary. The cost to satisfying a find_spec() or similar MetaPathFinder interface request would then be a Rust HashMap lookup, avoiding the O(n) traversal of sys.meta_path entries (which is Python heavy and relatively slow compared to compiled code) and further avoiding the O(n) lookups in BuiltinImporter and FrozenImporter. I don't expect this to yield the performance wins that doing away with filesystem-based module importing did (importing the entire standard library completed in ~70% of the time). But startup overhead is a problem and every little improvement can help. Perhaps there is room to add importlib APIs to facilitate indexing. Then MetaPathFinders which are immutable could contribute to a global lookup dict, facilitating much faster import operations. It wouldn't work for PathFinder. But it opens some interesting possibilities... From steve at holdenweb.com Mon May 20 05:21:14 2019 From: steve at holdenweb.com (Steve Holden) Date: Mon, 20 May 2019 10:21:14 +0100 Subject: [Python-Dev] Feature request: Change a Dependency Package Version During Package Initiation In-Reply-To: <002301d50d49$e36ddf60$aa499e20$@danryan.co> References: <16ac4ce3e2f.c1af300c4816.5384009389495434427@zoho.com.cn> <16ac8c2b2b4.ffb0004b5281.6365385802178906075@zoho.com.cn> <16ac8c85883.1228864315288.5365657524062277459@zoho.com.cn> <002301d50d49$e36ddf60$aa499e20$@danryan.co> Message-ID: I suspect this discussion would be better moved to python-ideas, since it seems clear there needs to be some work on determining the exact requirements On Sat, May 18, 2019 at 4:03 PM Dan Ryan wrote: > It probably makes the most sense to formally outline all of the behavioral > changes, expectations, and constraints you are asking for as Brett said > because dependency and package management are tricky and it's not clear (to > me at least) what the request is, what problem is being encountered, and > how it is proposed that it be resolved. > > First glance makes it sound a bit like the a prerequisite for this request > would be allowing multiple versions of the same package to be installed in > a given python environment to begin with, or else some runtime hacking of > paths would be necessary. I may be misunderstanding, but that's why a > formal approach for something like this might make some sense > > Dan Ryan > gh: @techalchemy // e: dan at danryan.co > > From: Q [mailto:qiang.fang at zoho.com.cn] > Sent: Friday, May 17, 2019 10:32 PM > To: Daniel Holth > Cc: Brett Cannon; Python-Dev > Subject: Re: [Python-Dev] Feature request: Change a Dependency Package > Version During Package Initiation > > Sorry, there is an error in my last post. It's corrected as: > The abaqus python distribution has libA 1.0, the external package to be > installed is LibB, which requires libA2.0, and we don't want to change > every import in LibB to be import /somewhere/libA2.0, because the source > code may have not been provided. > > > ---- ? ???, 2019-05-18 10:26:04 Q ?? ---- > > From the viewpoint of the package user, people don't want to change every > import to "require a version at runtime". If "set up a python path in a > wrapper script" is a better strategy, can you please give an example for > the following use case: > The abaqus python distribution has libA 1.0, the external package to be > installed is LibB, which requires libA2.0, and we don't want to change > every import in LibB to be import libA 1.0, because the source code may > have not been provided. > In emacs, you can always advice a command to change its behaviour, which > makes it very user friendly, and that what all the following suggestion is > about. I have no idea at all how it could be implemented though. > https://github.com/mitsuhiko/multiversion/issues/1 > "The work around is to let a version to be set within a package and have > that propagate to all modules in that package. For example in the root > init.py if I set tonado version to be 2.2.1 then all modules in that > package will use tornado 2.2.1 when I import tornado." > > ---- ? ???, 2019-05-17 23:38:55 Daniel Holth ?? ---- > > > > This sounds exactly like what people used to do with eggs. You could have > multiple versions of a package on the path as eggs and then require a > version at runtime. The approach has problems. Ruby also abandoned a > strategy where random app code depends on package management code at > runtime. > > One better strategy is to set up a python path in a wrapper script. > > On Fri, May 17, 2019, 11:27 Brett Cannon wrote: > > Thanks for the idea but there are currently no plans to support such a > feature. If you would like to see it then you will need to write a PEP with > a proof-of-concept to demonstrate how you would expect such a feature to > work. > > On Fri., May 17, 2019, 07:55 Q via Python-Dev, > wrote: > > A lot of the Python code we use in production are used directly as imports > in other python > distributions (such as the python comes with the finite element software > Abaqus and MSC Marc), many > packages (such as matplotlib, numpy) that may have varying versioned > dependencies. > > I was wondering if this could be expanded to allow a version to be set > within a package and have > that propagate to all modules in that package. For example in the root > init.py if I set > multiversion(tornado, 2.2.1) then all modules in that package will use > tornado 2.2.1 when I import > tornado. > > See a relevant issue on github: > https://github.com/mitsuhiko/multiversion/issues/1 > > Thank you! > Qiang > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Mon May 20 07:09:28 2019 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 20 May 2019 13:09:28 +0200 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: <3a18be48-c63f-c70b-2f84-3a657a482df5@gmail.com> References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> <3a18be48-c63f-c70b-2f84-3a657a482df5@gmail.com> Message-ID: Hi Gregory, IMHO your remarks are not directly related to the PEP 587 and can be addressed in parallel. > It sounds like PyOxidizer and Hermetic Python are on the same page and > we're working towards a more official solution. But I want to make sure > by explicitly stating what PyOxidizer is doing. > > Essentially, to facilitate in-memory import, we need to register a > custom sys.meta_path importer *before* any file-based imports are > attempted. (...) > I /think/ providing a 2-phase > initialization that stops between _Py_InitializeCore() and > _Py_InitializeMainInterpreter() would get the job done for PyOxidizer > today. (...) Extract of PEP 587: "This extracts a subset of the API design from the PEP 432 development and refactoring work that is now considered sufficiently stable to make public (allowing 3rd party embedding applications access to the same configuration APIs that the native CPython CLI is now using)." We know that my PEP 587 is incomplete, but the work will continue in Python 3.9 to support your use case. The PEP 587 introduces an experimental separation between "core" and "main" initialization phases. PyConfig._init_main=0 stops at the "core" phase, then you are free to run C and Python, _Py_InitializeMain() finishes the Python initialization ("main" phase). > > In the "Isolate Python" section, I suggest to set the "isolated" > > parameter to 1 which imply setting user_site_directory to 0. So > > sys.path isn't modified afterwards. What you pass to PyConfig is what > > you get in sys.path in this case. > > Regarding site.py, I agree it is problematic for embedding scenarios. > Some features of site.py can be useful. Others aren't. It would be > useful to have more granular control over which bits of site.run are > run. My naive suggestion would be to add individual flags to control > which functions site.py:main() runs. That way embedders can cherry-pick > site.py features without having to manually import the module and call > functions within. That feels much more robust for long-term maintainability. I agree that more work can be done on the site module. IMHO core features which are needed by everybody should be done before calling site. Maybe using a frozen "presite" module or whatever. I would be interested to make possible to use Python for most cases without the site module. > Regarding Python calling exit(), this is problematic for embedding > scenarios. I am working on that. I fixed dozens of functions. For example, Py_RunMain() should not longer exit if there is an uncaught SystemExit when calling PyErr_Print(). SystemExit is now handled separately before calling PyErr_Print(). The work is not done, but it should be way better than Python 3.6 and 3.7 state. > This thread called attention to exit() during interpreter > initialization. But it is also a problem elsewhere. For example, > PyErr_PrintEx() will call Py_Exit() if the exception is a SystemExit. > There's definitely room to improve the exception handling mechanism to > give embedders better control when SystemExit is raised. As it stands, > we need to check for SystemExit manually and reimplement > _Py_HandleSystemExit() to emulate its behavior for e.g. exception value > handling (fun fact: you can pass non-None, non-integer values to > sys.exit/SystemExit). I don't know well these functions, maybe new functions are needed. It can be done without/outside the PEP 587. > Having a more efficient member lookup for BuiltinImporter and > FrozenImporter might shave off a millisecond or two from startup. This > would require some kind of derived data structure. (...) I don't think that the structures/API to define frozen/builtin modules has to change. We can convert these lists into an hash table during the initialization of the importlib module. I'm not saying that the current API is perfect, just that IMHO it can be solved without the API. > Unfortunately, as long as there is a global data structure that can be mutated any time > (the API contract doesn't prohibit modifying these global arrays after > initialization), you would need to check for "cache invalidation" on > every lookup, undermining performance benefits. Do you really expect an application modifying these lists dynamically? Victor -- Night gathers, and now my watch begins. It shall not end until my death. From vstinner at redhat.com Mon May 20 08:05:42 2019 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 20 May 2019 14:05:42 +0200 Subject: [Python-Dev] PEP 587 "Python Initialization Configuration" version 4 Message-ID: Hi, I expected the version 3 of my PEP to be complete, but Gregory Szorc and Steve Dower spotted a few more issues ;-) The main change of the version 4 is the introduction of "Python Configuration" and "Isolated Configuration" default configuration which are well better defined. The new "Isolated Configuration" provides sane default values to isolate Python from the system. For example, to embed Python into an application. Using the environment are now opt-in options, rather than an opt-out options. For example, environment variables, command line arguments and global configuration variables are ignored by default. Building a customized Python which behaves as the regular Python becomes easier using the new Py_RunMain() function. Moreover, using the "Python Configuration", PyConfig.argv arguments are now parsed the same way the regular Python parses command line arguments, and PyConfig.xoptions are handled as -X opt command line options. I replaced all macros with functions. Macros can cause issues when used from different programming languages, whereas functions are always well supported. PyPreConfig structure doesn't allocate memory anymore (the allocator field becomes an integer, instead of a string). I removed the "Constant PyConfig" special case which introduced too many exceptions for little benefit. See the "Version History" section for the full changes. HTML version: https://www.python.org/dev/peps/pep-0587/ Full text below. Victor PEP: 587 Title: Python Initialization Configuration Author: Victor Stinner , Nick Coghlan BDFL-Delegate: Thomas Wouters Discussions-To: python-dev at python.org Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 27-Mar-2019 Python-Version: 3.8 Abstract ======== Add a new C API to configure the Python Initialization providing finer control on the whole configuration and better error reporting. It becomes possible to read the configuration and then override some computed parameters before it is applied. It also becomes possible to completely override how Python computes the module search paths (``sys.path``). The new `Isolated Configuration`_ provides sane default values to isolate Python from the system. For example, to embed Python into an application. Using the environment are now opt-in options, rather than an opt-out options. For example, environment variables, command line arguments and global configuration variables are ignored by default. Building a customized Python which behaves as the regular Python becomes easier using the new ``Py_RunMain()`` function. Moreover, using the `Python Configuration`_, ``PyConfig.argv`` arguments are now parsed the same way the regular Python parses command line arguments, and ``PyConfig.xoptions`` are handled as ``-X opt`` command line options. This extracts a subset of the API design from the PEP 432 development and refactoring work that is now considered sufficiently stable to make public (allowing 3rd party embedding applications access to the same configuration APIs that the native CPython CLI is now using). Rationale ========= Python is highly configurable but its configuration evolved organically. The initialization configuration is scattered all around the code using different ways to set them: global configuration variables (ex: ``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``), command line arguments (ex: ``-b``), configuration files (ex: ``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A straightforward and reliable way to configure Python is needed. Some configuration parameters are not accessible from the C API, or not easily. For example, there is no API to override the default values of ``sys.executable``. Some options like ``PYTHONPATH`` can only be set using an environment variable which has a side effect on Python child processes if not unset properly. Some options also depends on other options: see `Priority and Rules`_. Python 3.7 API does not provide a consistent view of the overall configuration. The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as input whereas the Python filesystem encoding is set during the initialization which can lead to mojibake. Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory allocation failure which is not convenient when Python is embedded. Moreover, ``Py_Main()`` could exit directly the process rather than returning an exit code. Proposed new API reports the error or exit code to the caller which can decide how to handle it. Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly was almost impossible in Python 3.6. The code base has been deeply reworked in Python 3.7 and then in Python 3.8 to read the configuration into a structure with no side effect. It becomes possible to clear the configuration (release memory) and read again the configuration if the encoding changed . It is required to implement properly the UTF-8 which changes the encoding using ``-X utf8`` command line option. Internally, bytes ``argv`` strings are decoded from the filesystem encoding. The ``-X dev`` changes the memory allocator (behaves as ``PYTHONMALLOC=debug``), whereas it was not possible to change the memory allocation *while* parsing the command line arguments. The new design of the internal implementation not only allowed to implement properly ``-X utf8`` and ``-X dev``, it also allows to change the Python behavior way more easily, especially for corner cases like that, and ensure that the configuration remains consistent: see `Priority and Rules`_. This PEP is a partial implementation of PEP 432 which is the overall design. New fields can be added later to ``PyConfig`` structure to finish the implementation of the PEP 432 (e.g. by adding a new partial initialization API which allows to configure Python using Python objects to finish the full initialization). However, those features are omitted from this PEP as even the native CPython CLI doesn't work that way - the public API proposal in this PEP is limited to features which have already been implemented and adopted as private APIs for us in the native CPython CLI. Python Initialization C API =========================== This PEP proposes to add the following new structures, functions and macros. New structures: * ``PyConfig`` * ``PyInitError`` * ``PyPreConfig`` * ``PyWideStringList`` New functions: * ``PyConfig_Clear(config)`` * ``PyConfig_InitIsolatedConfig()`` * ``PyConfig_InitPythonConfig()`` * ``PyConfig_Read(config)`` * ``PyConfig_SetArgv(config, argc, argv)`` * ``PyConfig_SetBytesArgv(config, argc, argv)`` * ``PyConfig_SetBytesString(config, config_str, str)`` * ``PyConfig_SetString(config, config_str, str)`` * ``PyInitError_Error(err_msg)`` * ``PyInitError_Exit(exitcode)`` * ``PyInitError_Failed(err)`` * ``PyInitError_IsError(err)`` * ``PyInitError_IsExit(err)`` * ``PyInitError_NoMemory()`` * ``PyInitError_Ok()`` * ``PyPreConfig_InitIsolatedConfig(preconfig)`` * ``PyPreConfig_InitPythonConfig(preconfig)`` * ``PyWideStringList_Append(list, item)`` * ``PyWideStringList_Insert(list, index, item)`` * ``Py_BytesMain(argc, argv)`` * ``Py_ExitInitError(err)`` * ``Py_InitializeFromConfig(config)`` * ``Py_PreInitialize(preconfig)`` * ``Py_PreInitializeFromArgs(preconfig, argc, argv)`` * ``Py_PreInitializeFromBytesArgs(preconfig, argc, argv)`` * ``Py_RunMain()`` This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type) and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these internal structures. ``PyInterpreterState.config`` becomes the new reference configuration, replacing global configuration variables and other private variables. PyWideStringList ---------------- ``PyWideStringList`` is a list of ``wchar_t*`` strings. ``PyWideStringList`` structure fields: * ``length`` (``Py_ssize_t``) * ``items`` (``wchar_t**``) Methods: * ``PyInitError PyWideStringList_Append(PyWideStringList *list, const wchar_t *item)``: Append *item* to *list*. * ``PyInitError PyWideStringList_Insert(PyWideStringList *list, Py_ssize_t index, const wchar_t *item)``: Insert *item* into *list* at *index*. If *index* is greater than *list* length, just append *item* to *list*. If *length* is non-zero, *items* must be non-NULL and all strings must be non-NULL. PyInitError ----------- ``PyInitError`` is a structure to store an error message or an exit code for the Python Initialization. For an error, it stores the C function name which created the error. Example:: PyInitError alloc(void **ptr, size_t size) { *ptr = PyMem_RawMalloc(size); if (*ptr == NULL) { return PyInitError_NoMemory(); } return PyInitError_Ok(); } int main(int argc, char **argv) { void *ptr; PyInitError err = alloc(&ptr, 16); if (PyInitError_Failed(err)) { Py_ExitInitError(err); } PyMem_Free(ptr); return 0; } ``PyInitError`` fields: * ``exitcode`` (``int``): Argument passed to ``exit()``. * ``err_msg`` (``const char*``): Error message. * ``func`` (``const char *``): Name of the function which created an error, can be ``NULL``. * private ``_type`` field: for internal usage only. Functions to create an error: * ``PyInitError_Ok()``: Success. * ``PyInitError_Error(err_msg)``: Initialization error with a message. * ``PyInitError_NoMemory()``: Memory allocation failure (out of memory). * ``PyInitError_Exit(exitcode)``: Exit Python with the specified exit code. Functions to handle an error: * ``PyInitError_Failed(err)``: Is the result an error or an exit? * ``PyInitError_IsError(err)``: Is the result an error? * ``PyInitError_IsExit(err)``: Is the result an exit? * ``Py_ExitInitError(err)``: Call ``exit(exitcode)`` if *err* is an exit, print the error and exit if *err* is an error. Must only be called with an error and an exit: if ``PyInitError_Failed(err)`` is true. Preinitialization with PyPreConfig ---------------------------------- The ``PyPreConfig`` structure is used to preinitialize Python: * Set the Python memory allocator * Configure the LC_CTYPE locale * Set the UTF-8 mode Example using the preinitialization to enable the UTF-8 Mode:: PyPreConfig preconfig; PyPreConfig_InitPythonConfig(&preconfig); preconfig.utf8_mode = 1; PyInitError err = Py_PreInitialize(&preconfig); if (PyInitError_Failed(err)) { Py_ExitInitError(err); } /* at this point, Python will speak UTF-8 */ Py_Initialize(); /* ... use Python API here ... */ Py_Finalize(); Function to initialize a pre-configuration: * ``void PyPreConfig_InitIsolatedConfig(PyPreConfig *preconfig)`` * ``void PyPreConfig_InitPythonConfig(PyPreConfig *preconfig)`` Functions to preinitialization Python: * ``PyInitError Py_PreInitialize(const PyPreConfig *preconfig)`` * ``PyInitError Py_PreInitializeFromBytesArgs(const PyPreConfig *preconfig, int argc, char * const *argv)`` * ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *preconfig, int argc, wchar_t * const * argv)`` The caller is responsible to handle error or exit using ``PyInitError_Failed()`` and ``Py_ExitInitError()``. If Python is initialized with command line arguments, the command line arguments must also be passed to preinitialize Python, since they have an effect on the pre-configuration like encodings. For example, the ``-X utf8`` command line option enables the UTF-8 Mode. ``PyPreConfig`` fields: * ``allocator`` (``int``): Name of the memory allocator (ex: ``PYMEM_ALLOCATOR_MALLOC``). Valid values: * ``PYMEM_ALLOCATOR_NOT_SET`` (``0``): don't change memory allocators (use defaults) * ``PYMEM_ALLOCATOR_DEFAULT`` (``1``): default memory allocators * ``PYMEM_ALLOCATOR_DEBUG`` (``2``): enable debug hooks * ``PYMEM_ALLOCATOR_MALLOC`` (``3``): force usage of ``malloc()`` * ``PYMEM_ALLOCATOR_MALLOC_DEBUG`` (``4``): ``malloc()`` with debug hooks * ``PYMEM_ALLOCATOR_PYMALLOC`` (``5``): Python "pymalloc" allocator * ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` (``6``): pymalloc with debug hooks * Note: ``PYMEM_ALLOCATOR_PYMALLOC`` and ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` are not supported if Python is configured using ``--without-pymalloc`` * ``configure_locale`` (``int``): Set the LC_CTYPE locale to the user preferred locale? If equals to 0, set ``coerce_c_locale`` and ``coerce_c_locale_warn`` to 0. * ``coerce_c_locale`` (``int``): If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE locale to decide if it should be coerced. * ``coerce_c_locale_warn`` (``int``): If non-zero, emit a warning if the C locale is coerced. * ``dev_mode`` (``int``): See ``PyConfig.dev_mode``. * ``isolated`` (``int``): See ``PyConfig.isolated``. * ``legacy_windows_fs_encoding`` (``int``): If non-zero, disable UTF-8 Mode, set the Python filesystem encoding to ``mbcs``, set the filesystem error handler to ``replace``. * ``parse_argv`` (``int``): If non-zero, ``Py_PreInitializeFromArgs()`` and ``Py_PreInitializeFromBytesArgs()`` parse their ``argv`` argument the same way the regular Python parses command line arguments: see `Command Line Arguments`_. * ``use_environment`` (``int``): See ``PyConfig.use_environment``. * ``utf8_mode`` (``int``): If non-zero, enable the UTF-8 mode. The ``legacy_windows_fs_encoding`` is only available on Windows. There is also a private field, for internal use only, ``_config_version`` (``int``): the configuration version, used for ABI compatibility. ``PyMem_SetAllocator()`` can be called after ``Py_PreInitialize()`` and before ``Py_InitializeFromConfig()`` to install a custom memory allocator. It can be called before ``Py_PreInitialize()`` if ``allocator`` is set to ``PYMEM_ALLOCATOR_NOT_SET`` (default value). Python memory allocation functions like ``PyMem_RawMalloc()`` must not be used before Python preinitialization, whereas calling directly ``malloc()`` and ``free()`` is always safe. ``Py_DecodeLocale()`` must not be called before the preinitialization. Initialization with PyConfig ---------------------------- The ``PyConfig`` structure contains most parameters to configure Python. Example setting the program name:: void init_python(void) { PyInitError err; PyConfig config; err = PyConfig_InitPythonConfig(&config); if (PyInitError_Failed(err)) { goto fail; } /* Set the program name. Implicitly preinitialize Python. */ err = PyConfig_SetString(&config, &config.program_name, L"/path/to/my_program"); if (PyInitError_Failed(err)) { goto fail; } err = Py_InitializeFromConfig(&config); if (PyInitError_Failed(err)) { goto fail; } PyConfig_Clear(&config); return; fail: PyConfig_Clear(&config); Py_ExitInitError(err); } ``PyConfig`` methods: * ``PyInitError PyConfig_InitPythonConfig(PyConfig *config)`` Initialize configuration with `Python Configuration`_. * ``PyInitError PyConfig_InitIsolatedConfig(PyConfig *config)``: Initialize configuration with `Isolated Configuration`_. * ``PyInitError PyConfig_SetString(PyConfig *config, wchar_t * const *config_str, const wchar_t *str)``: Copy the wide character string *str* into ``*config_str``. Preinitialize Python if needed. * ``PyInitError PyConfig_SetBytesString(PyConfig *config, wchar_t * const *config_str, const char *str)``: Decode *str* using ``Py_DecodeLocale()`` and set the result into ``*config_str``. Preinitialize Python if needed. * ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, wchar_t * const *argv)``: Set command line arguments from wide character strings. Preinitialize Python if needed. * ``PyInitError PyConfig_SetBytesArgv(PyConfig *config, int argc, char * const *argv)``: Set command line arguments: decode bytes using ``Py_DecodeLocale()``. Preinitialize Python if needed. * ``PyInitError PyConfig_Read(PyConfig *config)``: Read all Python configuration. Fields which are already initialized are left unchanged. Preinitialize Python if needed. * ``void PyConfig_Clear(PyConfig *config)``: Release configuration memory. Most ``PyConfig`` methods preinitialize Python if needed. In that case, the Python preinitialization configuration in based on the ``PyConfig``. If configuration fields which are in common with ``PyPreConfig`` are tuned, they must be set before calling a ``PyConfig`` method: * ``dev_mode`` * ``isolated`` * ``parse_argv`` * ``use_environment`` Moreover, if ``PyConfig_SetArgv()`` or ``PyConfig_SetBytesArgv()`` is used, this method must be called first, before other methods, since the preinitialization configuration depends on command line arguments (if ``parse_argv`` is non-zero). Functions to initialize Python: * ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``: Initialize Python from *config* configuration. The caller of these methods and functions is responsible to handle error or exit using ``PyInitError_Failed()`` and ``Py_ExitInitError()``. ``PyConfig`` fields: * ``argv`` (``PyWideStringList``): Command line arguments, ``sys.argv``. See ``parse_argv`` to parse ``argv`` the same way the regular Python parses Python command line arguments. If ``argv`` is empty, an empty string is added to ensure that ``sys.argv`` always exists and is never empty. * ``base_exec_prefix`` (``wchar_t*``): ``sys.base_exec_prefix``. * ``base_prefix`` (``wchar_t*``): ``sys.base_prefix``. * ``buffered_stdio`` (``int``): If equals to 0, enable unbuffered mode, making the stdout and stderr streams unbuffered. * ``bytes_warning`` (``int``): If equals to 1, issue a warning when comparing ``bytes`` or ``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If equal or greater to 2, raise a ``BytesWarning`` exception. * ``check_hash_pycs_mode`` (``wchar_t*``): ``--check-hash-based-pycs`` command line option value (see PEP 552). * ``configure_c_stdio`` (``int``): If non-zero, configure C standard streams (``stdio``, ``stdout``, ``stdout``). For example, set their mode to ``O_BINARY`` on Windows. * ``dev_mode`` (``int``): Development mode * ``dump_refs`` (``int``): If non-zero, dump all objects which are still alive at exit * ``exec_prefix`` (``wchar_t*``): ``sys.exec_prefix``. * ``executable`` (``wchar_t*``): ``sys.executable``. * ``faulthandler`` (``int``): If non-zero, call ``faulthandler.enable()``. * ``filesystem_encoding`` (``wchar_t*``): Filesystem encoding, ``sys.getfilesystemencoding()``. * ``filesystem_errors`` (``wchar_t*``): Filesystem encoding errors, ``sys.getfilesystemencodeerrors()``. * ``use_hash_seed`` (``int``), ``hash_seed`` (``unsigned long``): Randomized hash function seed. * ``home`` (``wchar_t*``): Python home directory. * ``import_time`` (``int``): If non-zero, profile import time. * ``inspect`` (``int``): Enter interactive mode after executing a script or a command. * ``install_signal_handlers`` (``int``): Install signal handlers? * ``interactive`` (``int``): Interactive mode. * ``legacy_windows_stdio`` (``int``, Windows only): If non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. * ``malloc_stats`` (``int``): If non-zero, dump memory allocation statistics at exit. * ``pythonpath_env`` (``wchar_t*``): Module search paths as a string separated by DELIM (usually ``:``). Initialized from ``PYTHONPATH`` environment variable value by default. * ``module_search_paths_set`` (``int``), ``module_search_paths`` (``PyWideStringList``): ``sys.path``. If ``module_search_paths_set`` is equal to 0, the ``module_search_paths`` is replaced by the function computing the `Path Configuration`. * ``optimization_level`` (``int``): Compilation optimization level. * ``parse_argv`` (``int``): If non-zero, parse ``argv`` the same way the regular Python command line arguments, and strip Python arguments from ``argv``: see `Command Line Arguments`_. * ``parser_debug`` (``int``): If non-zero, turn on parser debugging output (for expert only, depending on compilation options). * ``pathconfig_warnings`` (``int``): If equal to 0, suppress warnings when computing the path configuration (Unix only, Windows does not log any warning). Otherwise, warnings are written into stderr. * ``prefix`` (``wchar_t*``): ``sys.prefix``. * ``program_name`` (``wchar_t*``): Program name. * ``pycache_prefix`` (``wchar_t*``): ``.pyc`` cache prefix. * ``quiet`` (``int``): Quiet mode. For example, don't display the copyright and version messages even in interactive mode. * ``run_command`` (``wchar_t*``): ``-c COMMAND`` argument. * ``run_filename`` (``wchar_t*``): ``python3 SCRIPT`` argument. * ``run_module`` (``wchar_t*``): ``python3 -m MODULE`` argument. * ``show_alloc_count`` (``int``): Show allocation counts at exit? * ``show_ref_count`` (``int``): Show total reference count at exit? * ``site_import`` (``int``): Import the ``site`` module at startup? * ``skip_source_first_line`` (``int``): Skip the first line of the source? * ``stdio_encoding`` (``wchar_t*``), ``stdio_errors`` (``wchar_t*``): Encoding and encoding errors of ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. * ``tracemalloc`` (``int``): If non-zero, call ``tracemalloc.start(value)``. * ``user_site_directory`` (``int``): If non-zero, add user site directory to ``sys.path``. * ``verbose`` (``int``): If non-zero, enable verbose mode. * ``warnoptions`` (``PyWideStringList``): Options of the ``warnings`` module to build warnings filters. * ``write_bytecode`` (``int``): If non-zero, write ``.pyc`` files. * ``xoptions`` (``PyWideStringList``): ``sys._xoptions``. If ``parse_argv`` is non-zero, ``argv`` arguments are parsed the same way the regular Python parses command line arguments, and Python arguments are stripped from ``argv``: see `Command Line Arguments`_. The ``xoptions`` options are parsed to set other options: see `-X Options`_. ``PyConfig`` private fields, for internal use only: * ``_config_version`` (``int``): Configuration version, used for ABI compatibility. * ``_config_init`` (``int``): Function used to initalize ``PyConfig``, used for preinitialization. * ``_install_importlib`` (``int``): Install importlib? * ``_init_main`` (``int``): If equal to 0, stop Python initialization before the "main" phase (see PEP 432). More complete example modifying the default configuration, read the configuration, and then override some parameters:: PyInitError init_python(const char *program_name) { PyInitError err; PyConfig config; err = PyConfig_InitPythonConfig(&config); if (PyInitError_Failed(err)) { goto done; } /* Set the program name before reading the configuraton (decode byte string from the locale encoding). Implicitly preinitialize Python. */ err = PyConfig_SetBytesString(&config, &config.program_name, program_name); if (PyInitError_Failed(err)) { goto done; } /* Read all configuration at once */ err = PyConfig_Read(&config); if (PyInitError_Failed(err)) { goto done; } /* Append our custom search path to sys.path */ err = PyWideStringList_Append(&config.module_search_paths, L"/path/to/more/modules"); if (PyInitError_Failed(err)) { goto done; } /* Override executable computed by PyConfig_Read() */ err = PyConfig_SetString(&config, &config.executable, L"/path/to/my_executable"); if (PyInitError_Failed(err)) { goto done; } err = Py_InitializeFromConfig(&config); done: PyConfig_Clear(&config); return err; } .. note:: ``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` and ``PyImport_ExtendInittab()`` functions are still relevant and continue to work as previously. They should be set or called before the Python initialization. Isolated Configuration ---------------------- ``PyPreConfig_InitIsolatedConfig()`` and ``PyConfig_InitIsolatedConfig()`` functions create a configuration to isolate Python from the system. For example, to embed Python into an application. This configuration ignores global configuration variables, environments variables and command line arguments (``argv`` is not parsed). The C standard streams (ex: ``stdout``) and the LC_CTYPE locale are left unchanged by default. Configuration files are still used with this configuration. Set the `Path Configuration`_ ("output fields") to ignore these configuration files and avoid the function computing the default path configuration. Python Configuration -------------------- ``PyPreConfig_InitPythonConfig()`` and ``PyConfig_InitPythonConfig()`` functions create a configuration to build a customized Python which behaves as the regular Python. Environments variables and command line arguments are used to configure Python, whereas global configuration variables are ignored. This function enables C locale coercion (PEP 538) and UTF-8 Mode (PEP 540) depending on the LC_CTYPE locale, ``PYTHONUTF8`` and ``PYTHONCOERCECLOCALE`` environment variables. Example of customized Python always running in isolated mode:: int main(int argc, char **argv) { PyConfig config; PyInitError err; err = PyConfig_InitPythonConfig(&config); if (PyInitError_Failed(err)) { goto fail; } config.isolated = 1; /* Decode command line arguments. Implicitly preinitialize Python (in isolated mode). */ err = PyConfig_SetBytesArgv(&config, argc, argv); if (PyInitError_Failed(err)) { goto fail; } err = Py_InitializeFromConfig(&config); if (PyInitError_Failed(err)) { goto fail; } PyConfig_Clear(&config); return Py_RunMain(); fail: PyConfig_Clear(&config); if (!PyInitError_IsExit(err)) { /* Display the error message and exit the process with non-zero exit code */ Py_ExitInitError(err); } return err.exitcode; } This example is a basic implementation of the "System Python Executable" discussed in PEP 432. Path Configuration ------------------ ``PyConfig`` contains multiple fields for the path configuration: * Path configuration input fields: * ``home`` * ``pythonpath_env`` * ``pathconfig_warnings`` * Path configuration output fields: * ``exec_prefix`` * ``executable`` * ``prefix`` * ``module_search_paths_set``, ``module_search_paths`` It is possible to completely ignore the function computing the default path configuration by setting explicitly all path configuration output fields listed above. A string is considered as set even if it's an empty string. ``module_search_paths`` is considered as set if ``module_search_paths_set`` is set to 1. In this case, path configuration input fields are ignored as well. Set ``pathconfig_warnings`` to 0 to suppress warnings when computing the path configuration (Unix only, Windows does not log any warning). If ``base_prefix`` or ``base_exec_prefix`` fields are not set, they inherit their value from ``prefix`` and ``exec_prefix`` respectively. If ``site_import`` is non-zero, ``sys.path`` can be modified by the ``site`` module. For example, if ``user_site_directory`` is non-zero, the user site directory is added to ``sys.path`` (if it exists). See also `Configuration Files`_ used by the path configuration. Py_BytesMain() -------------- Python 3.7 provides a high-level ``Py_Main()`` function which requires to pass command line arguments as ``wchar_t*`` strings. It is non-trivial to use the correct encoding to decode bytes. Python has its own set of issues with C locale coercion and UTF-8 Mode. This PEP adds a new ``Py_BytesMain()`` function which takes command line arguments as bytes:: int Py_BytesMain(int argc, char **argv) Py_RunMain() ------------ The new ``Py_RunMain()`` function executes the command (``PyConfig.run_command``), the script (``PyConfig.run_filename``) or the module (``PyConfig.run_module``) specified on the command line or in the configuration, and then finalizes Python. It returns an exit status that can be passed to the ``exit()`` function. :: int Py_RunMain(void); See `Python Configuration`_ for an example of customized Python always running in isolated mode using ``Py_RunMain()``. Backwards Compatibility ======================= This PEP only adds a new API: it leaves the existing API unchanged and has no impact on the backwards compatibility. The Python 3.7 ``Py_Initialize()`` function now disable the C locale coercion (PEP 538) and the UTF-8 Mode (PEP 540) by default to prevent mojibake. The new API using the `Python Configuration`_ is needed to enable them automatically. Annexes ======= Comparison of Python and Isolated Configurations ------------------------------------------------ Differences between ``PyPreConfig_InitPythonConfig()`` and ``PyPreConfig_InitIsolatedConfig()``: =============================== ======= ======== PyPreConfig Python Isolated =============================== ======= ======== ``coerce_c_locale_warn`` -1 0 ``coerce_c_locale`` -1 0 ``configure_locale`` **1** 0 ``dev_mode`` -1 0 ``isolated`` -1 **1** ``legacy_windows_fs_encoding`` -1 0 ``use_environment`` -1 0 ``parse_argv`` **1** 0 ``utf8_mode`` -1 0 =============================== ======= ======== Differences between ``PyConfig_InitPythonConfig()`` and ``PyConfig_InitIsolatedConfig()``: =============================== ======= ======== PyConfig Python Isolated =============================== ======= ======== ``configure_c_stdio`` **1** 0 ``install_signal_handlers`` **1** 0 ``isolated`` 0 **1** ``parse_argv`` **1** 0 ``pathconfig_warnings`` **1** 0 ``use_environment`` **1** 0 ``user_site_directory`` **1** 0 =============================== ======= ======== Priority and Rules ------------------ Priority of configuration parameters, highest to lowest: * ``PyConfig`` * ``PyPreConfig`` * Configuration files * Command line options * Environment variables * Global configuration variables Priority of warning options, highest to lowest: * ``PyConfig.warnoptions`` * ``PyConfig.dev_mode`` (add ``"default"``) * ``PYTHONWARNINGS`` environment variables * ``-W WARNOPTION`` command line argument * ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater than 1, or add ``"default::BytesWarning``) Rules on ``PyConfig`` parameters: * If ``isolated`` is non-zero, ``use_environment`` and ``user_site_directory`` are set to 0. * If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``, ``faulthandler`` is set to 1, and ``"default"`` filter is added to ``warnoptions``. But the ``PYTHONMALLOC`` environment variable has the priority over ``dev_mode`` to set the memory allocator. * If ``base_prefix`` is not set, it inherits ``prefix`` value. * If ``base_exec_prefix`` is not set, it inherits ``exec_prefix`` value. * If the ``python._pth`` configuration file is present, ``isolated`` is set to 1 and ``site_import`` is set to 0; but ``site_import`` is set to 1 if ``python._pth`` contains ``import site``. Rules on ``PyConfig`` and ``PyPreConfig`` parameters: * If ``PyPreConfig.legacy_windows_fs_encoding`` is non-zero, set ``PyPreConfig.utf8_mode`` to 0, set ``PyConfig.filesystem_encoding`` to ``mbcs``, and set ``PyConfig.filesystem_errors`` to ``replace``. Configuration Files ------------------- Python configuration files used by the `Path Configuration`_: * ``pyvenv.cfg`` * ``python._pth`` (Windows only) * ``pybuilddir.txt`` (Unix only) Global Configuration Variables ------------------------------ Global configuration variables mapped to ``PyPreConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` ``Py_UTF8Mode`` ``utf8_mode`` ======================================== ================================ (NOT) means that the ``PyPreConfig`` value is the oposite of the global configuration variable value. ``Py_LegacyWindowsFSEncodingFlag`` is only available on Windows. Global configuration variables mapped to ``PyConfig`` fields: ======================================== ================================ Variable Field ======================================== ================================ ``Py_BytesWarningFlag`` ``bytes_warning`` ``Py_DebugFlag`` ``parser_debug`` ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` (NOT) ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_FrozenFlag`` ``pathconfig_warnings`` (NOT) ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) ``Py_InspectFlag`` ``inspect`` ``Py_InteractiveFlag`` ``interactive`` ``Py_IsolatedFlag`` ``isolated`` ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` ``Py_NoSiteFlag`` ``site_import`` (NOT) ``Py_NoUserSiteDirectory`` ``user_site_directory`` (NOT) ``Py_OptimizeFlag`` ``optimization_level`` ``Py_QuietFlag`` ``quiet`` ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` (NOT) ``Py_VerboseFlag`` ``verbose`` ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` ======================================== ================================ (NOT) means that the ``PyConfig`` value is the oposite of the global configuration variable value. ``Py_LegacyWindowsStdioFlag`` is only available on Windows. Command Line Arguments ---------------------- Usage:: python3 [options] python3 [options] -c COMMAND python3 [options] -m MODULE python3 [options] SCRIPT Command line options mapped to pseudo-action on ``PyPreConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-E`` ``use_environment = 0`` ``-I`` ``isolated = 1`` ``-X dev`` ``dev_mode = 1`` ``-X utf8`` ``utf8_mode = 1`` ``-X utf8=VALUE`` ``utf8_mode = VALUE`` ================================ ================================ Command line options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-b`` ``bytes_warning++`` ``-B`` ``write_bytecode = 0`` ``-c COMMAND`` ``run_command = COMMAND`` ``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE`` ``-d`` ``parser_debug++`` ``-E`` ``use_environment = 0`` ``-i`` ``inspect++`` and ``interactive++`` ``-I`` ``isolated = 1`` ``-m MODULE`` ``run_module = MODULE`` ``-O`` ``optimization_level++`` ``-q`` ``quiet++`` ``-R`` ``use_hash_seed = 0`` ``-s`` ``user_site_directory = 0`` ``-S`` ``site_import`` ``-t`` ignored (kept for backwards compatibility) ``-u`` ``buffered_stdio = 0`` ``-v`` ``verbose++`` ``-W WARNING`` add ``WARNING`` to ``warnoptions`` ``-x`` ``skip_source_first_line = 1`` ``-X OPTION`` add ``OPTION`` to ``xoptions`` ================================ ================================ ``-h``, ``-?`` and ``-V`` options are handled without ``PyConfig``. -X Options ---------- -X options mapped to pseudo-action on ``PyConfig`` fields: ================================ ================================ Option ``PyConfig`` field ================================ ================================ ``-X dev`` ``dev_mode = 1`` ``-X faulthandler`` ``faulthandler = 1`` ``-X importtime`` ``import_time = 1`` ``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX`` ``-X showalloccount`` ``show_alloc_count = 1`` ``-X showrefcount`` ``show_ref_count = 1`` ``-X tracemalloc=N`` ``tracemalloc = N`` ================================ ================================ Environment Variables --------------------- Environment variables mapped to ``PyPreConfig`` fields: ================================= ============================================= Variable ``PyPreConfig`` field ================================= ============================================= ``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding`` ``PYTHONMALLOC`` ``allocator`` ``PYTHONUTF8`` ``utf8_mode`` ================================= ============================================= Environment variables mapped to ``PyConfig`` fields: ================================= ==================================== Variable ``PyConfig`` field ================================= ==================================== ``PYTHONDEBUG`` ``parser_debug`` ``PYTHONDEVMODE`` ``dev_mode`` ``PYTHONDONTWRITEBYTECODE`` ``write_bytecode`` ``PYTHONDUMPREFS`` ``dump_refs`` ``PYTHONEXECUTABLE`` ``program_name`` ``PYTHONFAULTHANDLER`` ``faulthandler`` ``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed`` ``PYTHONHOME`` ``home`` ``PYTHONINSPECT`` ``inspect`` ``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors`` ``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio`` ``PYTHONMALLOCSTATS`` ``malloc_stats`` ``PYTHONNOUSERSITE`` ``user_site_directory`` ``PYTHONOPTIMIZE`` ``optimization_level`` ``PYTHONPATH`` ``pythonpath_env`` ``PYTHONPROFILEIMPORTTIME`` ``import_time`` ``PYTHONPYCACHEPREFIX,`` ``pycache_prefix`` ``PYTHONTRACEMALLOC`` ``tracemalloc`` ``PYTHONUNBUFFERED`` ``buffered_stdio`` ``PYTHONVERBOSE`` ``verbose`` ``PYTHONWARNINGS`` ``warnoptions`` ================================= ==================================== ``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are specific to Windows. Default Python Configugration ----------------------------- ``PyPreConfig_InitPythonConfig()``: * ``allocator`` = ``PYMEM_ALLOCATOR_NOT_SET`` * ``coerce_c_locale_warn`` = -1 * ``coerce_c_locale`` = -1 * ``configure_locale`` = 1 * ``dev_mode`` = -1 * ``isolated`` = -1 * ``legacy_windows_fs_encoding`` = -1 * ``use_environment`` = -1 * ``utf8_mode`` = -1 ``PyConfig_InitPythonConfig()``: * ``argv`` = [] * ``base_exec_prefix`` = ``NULL`` * ``base_prefix`` = ``NULL`` * ``buffered_stdio`` = 1 * ``bytes_warning`` = 0 * ``check_hash_pycs_mode`` = ``NULL`` * ``configure_c_stdio`` = 1 * ``dev_mode`` = 0 * ``dump_refs`` = 0 * ``exec_prefix`` = ``NULL`` * ``executable`` = ``NULL`` * ``faulthandler`` = 0 * ``filesystem_encoding`` = ``NULL`` * ``filesystem_errors`` = ``NULL`` * ``hash_seed`` = 0 * ``home`` = ``NULL`` * ``import_time`` = 0 * ``inspect`` = 0 * ``install_signal_handlers`` = 1 * ``interactive`` = 0 * ``isolated`` = 0 * ``malloc_stats`` = 0 * ``module_search_path_env`` = ``NULL`` * ``module_search_paths`` = [] * ``optimization_level`` = 0 * ``parse_argv`` = 1 * ``parser_debug`` = 0 * ``pathconfig_warnings`` = 1 * ``prefix`` = ``NULL`` * ``program_name`` = ``NULL`` * ``pycache_prefix`` = ``NULL`` * ``quiet`` = 0 * ``run_command`` = ``NULL`` * ``run_filename`` = ``NULL`` * ``run_module`` = ``NULL`` * ``show_alloc_count`` = 0 * ``show_ref_count`` = 0 * ``site_import`` = 1 * ``skip_source_first_line`` = 0 * ``stdio_encoding`` = ``NULL`` * ``stdio_errors`` = ``NULL`` * ``tracemalloc`` = 0 * ``use_environment`` = 1 * ``use_hash_seed`` = 0 * ``user_site_directory`` = 1 * ``verbose`` = 0 * ``warnoptions`` = [] * ``write_bytecode`` = 1 * ``xoptions`` = [] * ``_init_main`` = 1 * ``_install_importlib`` = 1 Default Isolated Configugration ------------------------------- ``PyPreConfig_InitIsolatedConfig()``: * ``allocator`` = ``PYMEM_ALLOCATOR_NOT_SET`` * ``coerce_c_locale_warn`` = 0 * ``coerce_c_locale`` = 0 * ``configure_locale`` = 0 * ``dev_mode`` = 0 * ``isolated`` = 1 * ``legacy_windows_fs_encoding`` = 0 * ``use_environment`` = 0 * ``utf8_mode`` = 0 ``PyConfig_InitIsolatedConfig()``: * ``argv`` = [] * ``base_exec_prefix`` = ``NULL`` * ``base_prefix`` = ``NULL`` * ``buffered_stdio`` = 1 * ``bytes_warning`` = 0 * ``check_hash_pycs_mode`` = ``NULL`` * ``configure_c_stdio`` = 0 * ``dev_mode`` = 0 * ``dump_refs`` = 0 * ``exec_prefix`` = ``NULL`` * ``executable`` = ``NULL`` * ``faulthandler`` = 0 * ``filesystem_encoding`` = ``NULL`` * ``filesystem_errors`` = ``NULL`` * ``hash_seed`` = 0 * ``home`` = ``NULL`` * ``import_time`` = 0 * ``inspect`` = 0 * ``install_signal_handlers`` = 0 * ``interactive`` = 0 * ``isolated`` = 1 * ``malloc_stats`` = 0 * ``module_search_path_env`` = ``NULL`` * ``module_search_paths`` = [] * ``optimization_level`` = 0 * ``parse_argv`` = 0 * ``parser_debug`` = 0 * ``pathconfig_warnings`` = 0 * ``prefix`` = ``NULL`` * ``program_name`` = ``NULL`` * ``pycache_prefix`` = ``NULL`` * ``quiet`` = 0 * ``run_command`` = ``NULL`` * ``run_filename`` = ``NULL`` * ``run_module`` = ``NULL`` * ``show_alloc_count`` = 0 * ``show_ref_count`` = 0 * ``site_import`` = 1 * ``skip_source_first_line`` = 0 * ``stdio_encoding`` = ``NULL`` * ``stdio_errors`` = ``NULL`` * ``tracemalloc`` = 0 * ``use_environment`` = 0 * ``use_hash_seed`` = 0 * ``user_site_directory`` = 0 * ``verbose`` = 0 * ``warnoptions`` = [] * ``write_bytecode`` = 1 * ``xoptions`` = [] * ``_init_main`` = 1 * ``_install_importlib`` = 1 Python 3.7 API -------------- Python 3.7 has 4 functions in its C API to initialize and finalize Python: * ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python * ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python Python 3.7 can be configured using `Global Configuration Variables`_, `Environment Variables`_, and the following functions: * ``PyImport_AppendInittab()`` * ``PyImport_ExtendInittab()`` * ``PyMem_SetAllocator()`` * ``PyMem_SetupDebugHooks()`` * ``PyObject_SetArenaAllocator()`` * ``Py_SetPath()`` * ``Py_SetProgramName()`` * ``Py_SetPythonHome()`` * ``Py_SetStandardStreamEncoding()`` * ``PySys_AddWarnOption()`` * ``PySys_AddXOption()`` * ``PySys_ResetWarnOptions()`` There is also a high-level ``Py_Main()`` function and ``PyImport_FrozenModules`` variable which can be overridden. See `Initialization, Finalization, and Threads `_ documentation. Python Issues ============= Issues that will be fixed by this PEP, directly or indirectly: * `bpo-1195571 `_: "simple callback system for Py_FatalError" * `bpo-11320 `_: "Usage of API method Py_SetPath causes errors in Py_Initialize() (Posix ony)" * `bpo-13533 `_: "Would like Py_Initialize to play friendly with host app" * `bpo-14956 `_: "custom PYTHONPATH may break apps embedding Python" * `bpo-19983 `_: "When interrupted during startup, Python should not call abort() but exit()" * `bpo-22213 `_: "Make pyvenv style virtual environments easier to configure when embedding Python". This PEP more or * `bpo-22257 `_: "PEP 432: Redesign the interpreter startup sequence" * `bpo-29778 `_: "_Py_CheckPython3 uses uninitialized dllpath when embedder sets module path with Py_SetPath" * `bpo-30560 `_: "Add Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal errors". * `bpo-31745 `_: "Overloading "Py_GetPath" does not work" * `bpo-32573 `_: "All sys attributes (.argv, ...) should exist in embedded environments". * `bpo-34725 `_: "Py_GetProgramFullPath() odd behaviour in Windows" * `bpo-36204 `_: "Deprecate calling Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?" * `bpo-33135 `_: "Define field prefixes for the various config structs". The PEP now defines well how warnings options are handled. Issues of the PEP implementation: * `bpo-16961 `_: "No regression tests for -E and individual environment vars" * `bpo-20361 `_: "-W command line options and PYTHONWARNINGS environmental variable should not override -b / -bb command line options" * `bpo-26122 `_: "Isolated mode doesn't ignore PYTHONHASHSEED" * `bpo-29818 `_: "Py_SetStandardStreamEncoding leads to a memory error in debug mode" * `bpo-31845 `_: "PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect" * `bpo-32030 `_: "PEP 432: Rewrite Py_Main()" * `bpo-32124 `_: "Document functions safe to be called before Py_Initialize()" * `bpo-33042 `_: "New 3.7 startup sequence crashes PyInstaller" * `bpo-33932 `_: "Calling Py_Initialize() twice now triggers a fatal error (Python 3.7)" * `bpo-34008 `_: "Do we support calling Py_Main() after Py_Initialize()?" * `bpo-34170 `_: "Py_Initialize(): computing path configuration must not have side effect (PEP 432)" * `bpo-34589 `_: "Py_Initialize() and Py_Main() should not enable C locale coercion" * `bpo-34639 `_: "PYTHONCOERCECLOCALE is ignored when using -E or -I option" * `bpo-36142 `_: "Add a new _PyPreConfig step to Python initialization to setup memory allocator and encodings" * `bpo-36202 `_: "Calling Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake" * `bpo-36301 `_: "Add _Py_PreInitialize() function" * `bpo-36443 `_: "Disable coerce_c_locale and utf8_mode by default in _PyPreConfig?" * `bpo-36444 `_: "Python initialization: remove _PyMainInterpreterConfig" * `bpo-36471 `_: "PEP 432, PEP 587: Add _Py_RunMain()" * `bpo-36763 `_: "PEP 587: Rework initialization API to prepare second version of the PEP" * `bpo-36775 `_: "Rework filesystem codec implementation" * `bpo-36900 `_: "Use _PyCoreConfig rather than global configuration variables" Issues related to this PEP: * `bpo-12598 `_: "Move sys variable initialization from import.c to sysmodule.c" * `bpo-15577 `_: "Real argc and argv in embedded interpreter" * `bpo-16202 `_: "sys.path[0] security issues" * `bpo-18309 `_: "Make python slightly more relocatable" * `bpo-25631 `_: "Segmentation fault with invalid Unicode command-line arguments in embedded Python" * `bpo-26007 `_: "Support embedding the standard library in an executable" * `bpo-31210 `_: "Can not import modules if sys.prefix contains DELIM". * `bpo-31349 `_: "Embedded initialization ignores Py_SetProgramName()" * `bpo-33919 `_: "Expose _PyCoreConfig structure to Python" * `bpo-35173 `_: "Re-use already existing functionality to allow Python 2.7.x (both embedded and standalone) to locate the module path according to the shared library" Version History =============== * Version 4: * Introduce "Python Configuration" and "Isolated Configuration" which are well better defined. Replace all macros with functions. * Replace ``PyPreConfig_INIT`` and ``PyConfig_INIT`` macros with functions: * ``PyPreConfig_InitIsolatedConfig()``, ``PyConfig_InitIsolatedConfig()`` * ``PyPreConfig_InitPythonConfig()``, ``PyConfig_InitPythonConfig()`` * ``PyPreConfig`` no longer uses dynamic memory, the ``allocator`` field type becomes an int, add ``configure_locale`` and ``parse_argv`` field. * ``PyConfig``: rename ``module_search_path_env`` to ``pythonpath_env``, rename ``use_module_search_paths`` to ``module_search_paths_set``, remove ``program`` and ``dll_path``. * Replace ``Py_INIT_xxx()`` macros with ``PyInitError_xxx()`` functions. * Remove the "Constant PyConfig" section. Remove ``Py_InitializeFromArgs()`` and ``Py_InitializeFromBytesArgs()`` functions. * Version 3: * ``PyConfig``: Add ``configure_c_stdio`` and ``parse_argv``; rename ``_frozen`` to ``pathconfig_warnings``. * Rename functions using bytes strings and wide character strings. For example, ``Py_PreInitializeFromWideArgs()`` becomes ``Py_PreInitializeFromArgs()``, and ``PyConfig_SetArgv()`` becomes ``PyConfig_SetBytesArgv()``. * Add ``PyWideStringList_Insert()`` function. * New "Path configuration", "Isolate Python", "Python Issues" and "Version History" sections. * ``PyConfig_SetString()`` and ``PyConfig_SetBytesString()`` now requires the configuration as the first argument. * Rename ``Py_UnixMain()`` to ``Py_BytesMain()`` * Version 2: Add ``PyConfig`` methods (ex: ``PyConfig_Read()``), add ``PyWideStringList_Append()``, rename ``PyWideCharList`` to ``PyWideStringList``. * Version 1: Initial version. Copyright ========= This document has been placed in the public domain. From eric at trueblade.com Mon May 20 09:25:57 2019 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 20 May 2019 09:25:57 -0400 Subject: [Python-Dev] ast changes for "debug" f-strings Message-ID: I added an optional "expr_text" field to the FormattedValue node, which represents the text of the expression in a "debug" f-string expression. So in f'{x=}', expr_text would be "x=". This strictly speaking isn't necessary. I could have added another Constant node for "x=" and left FormattedValue alone. I didn't for three reasons: it was expedient; it didn't require a lot of surgery to f-string parsing, which the extra Constant node would require; and it allowed the Python/ast_unparse.c code to produce a string that was more consistent with input string. Now that I have more time, I'm rethinking this decision. I'd rather not have the extra churn that modifying the ast node will cause to downstream code. Someone (Serhiy?) already pointed this out somewhere, but now I can't find the original message. But the issue of the fidelity of ast_unparse is still there. I think the only place this code is used is "from __future__ import annotations", but for all I know there's a plan to use it elsewhere. Does anyone care that f'{x=}' would become f'x={x!r}' if I removed expr_text from the FormattedValue node? This isn't the only place where ast_unparse isn't exact with what it produces. It's not designed to give the exact source text back, just something equivalent. But it's not clear to me how equivalent it needs to be. Even if I did add the extra Constant node, then you'd have the following discrepancy: f'value: {x=}' would produce: [Expr(value=JoinedStr(values=[Constant(value='value: ', kind=None), Constant(value='x=', kind=None), FormattedValue(value=Name(id='x', ctx=Load()), conversion=114, format_spec=None, expr_text='x=')]))] Note that there are two consecutive Constant nodes here. So this: f'value: x={x!r}' would produce the slightly different: [Expr(value=JoinedStr(values=[Constant(value='value: x=', kind=None), FormattedValue(value=Name(id='x', ctx=Load()), conversion=114, format_spec=None)]))] Note that there is a single Constant node, with the combined string in it. I'm not sure how much we care about all of this, but let me know if you have a strong feeling about it. My suggestion for the 3.8b1 release is to add the extra Constant node and remove expr_text from the FormattedValue node. This would at least mean that there's no change to the ast definitions due to this feature. If we decide that the two consecutive Constant nodes generated by f'value: {x=}' need to be combined, then we can do that post 3.8b1. But it's a lot more work to combine the nodes while the f-string is being compiled. Let me know your thoughts, since time is of the essence. Eric From guido at python.org Mon May 20 10:35:53 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 20 May 2019 07:35:53 -0700 Subject: [Python-Dev] Plan to accept PEP 544, PEP 586, PEP 589, PEP 591 -- last call for discussion Message-ID: Please send all discussions about these PEPs to the typing-sig mailing list, so we have the archives all in one place. Read the full message here: https://mail.python.org/archives/list/typing-sig at python.org/thread/GCI2TF43SESP2XVSETWOW33SRRWQTHM4/ -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From isidentical at gmail.com Mon May 20 10:32:03 2019 From: isidentical at gmail.com (Batuhan Taskaya) Date: Mon, 20 May 2019 17:32:03 +0300 Subject: [Python-Dev] ast changes for "debug" f-strings In-Reply-To: References: Message-ID: > This strictly speaking isn't necessary. I could have added another Constant node for "x=" and left FormattedValue alone. I didn't for three reasons: it was expedient; it didn't require a lot of surgery to f-string parsing, which the extra Constant node would require; and it allowed the Python/ast_unparse.c code to produce a string that was more consistent with input string. Agreed. > Does anyone care that f'{x=}' would become f'x={x!r}' if I removed expr_text from the FormattedValue node? Yes, when i was implementing f-string debugging support to Berker's astor project the roundtrip tests i wrote is failing because of it adds an extra `!r` to end. Then i realized you added a new field (expr_text) for that. > I'm not sure how much we care about all of this, but let me know if you have a strong feeling about it. I don't think we should complicate this. The current version is very simple and understandable. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Mon May 20 11:09:32 2019 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 20 May 2019 17:09:32 +0200 Subject: [Python-Dev] Plan to accept PEP 544, PEP 586, PEP 589, PEP 591 -- last call for discussion In-Reply-To: References: Message-ID: FYI, copy of Guido's message: --- I'm planning to accept the following PEPs related to the type system later this week: PEP 544: Protocols PEP 586: Literal Types PEP 591: Final qualifier and @final decorator PEP 589: TypedDict All of these have been proposed for and discussed before. All of these have been implemented in mypy and (AFAIK) in some other type checkers (with the exception of Protocols for module objects, which is in the design stage for mypy but not controversial in any way). No substantial opposition came up during the discussions so far. Some changes were made to some of the PEPs and it seems everyone is happy with the results. I expect that these PEPs will be accepted in time to get their implementations into the stdlib typing.py module in 3.8 beta 1, which is due May 31st (i.e., in less than two weeks!). --- But please follow Guido's request: Le lun. 20 mai 2019 ? 16:39, Guido van Rossum a ?crit : > Please send all discussions about these PEPs to the typing-sig mailing list, so we have the archives all in one place. Read the full message here: Victor -- Night gathers, and now my watch begins. It shall not end until my death. From guido at python.org Mon May 20 11:55:59 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 20 May 2019 08:55:59 -0700 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: On Thu, May 16, 2019 at 3:57 PM Steve Dower wrote: > [...] > We still have the policy of not removing modules that exist in the > Python 2 standard library. But 3.9 won't be covered by that :) > I didn't even remember that. Where's that written down? And by the time 3.8 .0(final) comes out, 2.7 has only about two months of life left... FWIW I am strongly in favor of getting rid of the `parser` module, in 3.8 is we can, otherwise in 3.9 (after strong deprecation in 3.8). I am interested in switching CPython's parsing strategy to something else (what exactly remains to be seen) and any new approach is unlikely to reuse the current CST technology. (OTOH I think it would be wise to keep the current AST.) > But I'm in favor of having a proper CST module that matches the version > of Python it's in. It doesn't help people on earlier versions (yet), but > given how closely tied it is to the Python version you're on I think it > makes sense in the stdlib. > I presume this is in reference to ?ukasz's https://bugs.python.org/issue33337. I think we should act on that issue, but I don't think there's a good reason to tie deletion (or deprecation) of the `parser` module to whatever we do there. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon May 20 12:28:24 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 20 May 2019 09:28:24 -0700 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: On Thu, May 16, 2019 at 3:51 PM Pablo Galindo Salgado wrote: > [Nathaniel Smith] > >Will the folks using forks be happy to switch to the stdlib version? > >For example I can imagine that if black wants to process 3.7 input > >code while running on 3.6, it might prefer a parser on PyPI even if > >he stdlib version were public, since the PyPI version can be updated > >independently of the host Python. > > The tool can parse arbitrary grammars, the one that is packed into is just > one of them. > > I think it would be useful, among other things because the standard library > lacks currently a proper CST solution. The ast module is heavily leveraged > for > things like formatters, > Actually, I think the `ast` module doesn't work very well for formatters, because it loses comments. (Retaining comments and all details of whitespace is the specific use case for which I created pgen2.) > static code analyzers...etc but CST can be very useful as > ?ukasz describes here: > > https://bugs.python.org/issue33337 > > I think is missing an important gap in the stdlib and the closest thing we > have > (the current parser module) is not useful for any of that. Also, the core > to generating > the hypothetical new package (with some new API over it may be) is already > undocumented > as an implementation detail of lib2to3 (and some people are already using > it directly). > I wonder if lib2to3 is actually something that would benefit from moving out of the stdlib. (Wasn't it on Amber's list?) As ?ukasz points out in that issue, it is outdated. Maybe if it was out of the stdlib it would attract more contributors. Then again, I have recently started exploring the idea of a PEG parser for Python. Maybe a revamped version of the core of lib2to3 based on PEG instead of pgen would be interesting to some folks. I do agree that the two versions of tokenize.py should be unified (and the result kept in the stdlib). However there are some issues here, because tokenize.py is closely tied to the names and numbers of the tokens, and the latter information is currently generated based on the contents of the Grammar file. This may get in the way of using it to tokenize old Python versions. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Mon May 20 13:13:31 2019 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 20 May 2019 20:13:31 +0300 Subject: [Python-Dev] ast changes for "debug" f-strings In-Reply-To: References: Message-ID: 20.05.19 16:25, Eric V. Smith ????: > I added an optional "expr_text" field to the FormattedValue node, which > represents the text of the expression in a "debug" f-string expression. > So in f'{x=}', expr_text would be "x=". > > This strictly speaking isn't necessary. I could have added another > Constant node for "x=" and left FormattedValue alone. I didn't for three > reasons: it was expedient; it didn't require a lot of surgery to > f-string parsing, which the extra Constant node would require; and it > allowed the Python/ast_unparse.c code to produce a string that was more > consistent with input string. > > Now that I have more time, I'm rethinking this decision. I'd rather not > have the extra churn that modifying the ast node will cause to > downstream code. Someone (Serhiy?) already pointed this out somewhere, > but now I can't find the original message. Of course I am for removing "expr_text". This makes the AST simpler. I do not care about ast_unparse because it does nor roundtrip in general. For example it can add and remove parenthesis, commas, backslashes. Currently it even has a bug in handling some corner cases in f-strings -- nobody complains. In theory it is possible to detect when !d was used (parse the previous Constant value and compare the result with the FormattedValue expression). We can implement this if there is a need after beta1. From eric at trueblade.com Mon May 20 13:20:12 2019 From: eric at trueblade.com (Eric V. Smith) Date: Mon, 20 May 2019 13:20:12 -0400 Subject: [Python-Dev] ast changes for "debug" f-strings In-Reply-To: References: Message-ID: <9b29ae0c-74e3-3a65-f396-1f6683c8a1d6@trueblade.com> On 5/20/2019 10:32 AM, Batuhan Taskaya wrote: > > This strictly speaking isn't necessary. I could have added another > Constant node for "x=" and left FormattedValue alone. I didn't for three > reasons: it was expedient; it didn't require a lot of surgery to > f-string parsing, which the extra Constant node would require; and it > allowed the Python/ast_unparse.c code to produce a string that was more > consistent with input string. > > Agreed. > > > Does anyone care that f'{x=}' would become f'x={x!r}' if I removed > expr_text from the FormattedValue node? > > Yes, when i was implementing f-string debugging support to Berker's > astor project > the roundtrip tests i wrote is failing because of it adds an extra `!r` > to end. Then > i realized you added a new field (expr_text) for that. > > > I'm not sure how much we care about all of this, but let me know if you > have a strong feeling about it. > > I don't think we should complicate this. The current version is very > simple and understandable. I think the salient question is: does the lack of expr_text make anything more difficult for anyone? As Serhiy said in the other email, lots of things are lost while round-tripping, this would just be another one. Anyone really interested in high-fidelity round trips already can't use the ast module to unparse from. Eric From pablogsal at gmail.com Mon May 20 14:05:54 2019 From: pablogsal at gmail.com (Pablo Galindo Salgado) Date: Mon, 20 May 2019 19:05:54 +0100 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: > Actually, I think the `ast` module doesn't work very well for formatters, because it loses comments. (Retaining comments and all details of whitespace is the specific use case for which I created pgen2.) Some uses I have seen include using it to check that the code before and after the formatting has no functional changes (both have the same ast) or to augment the information obtained with other sources. But yeah, I agree that static code analyzers and linters are a much bigger target. >I wonder if lib2to3 is actually something that would benefit from moving out of the stdlib. (Wasn't it on Amber's list?) As ?ukasz points out in that issue, it is outdated. Maybe if it was out of the stdlib it would attract more contributors. Then again, I have recently started exploring the idea of a PEG parser for Python. Maybe a revamped version of the core of lib2to3 based on PEG instead of pgen would be interesting to some folks. I was thinking more on the line of leveraging some parts lib2to3 having some CST-related solution similar to the ast module, not exposing the whole functionality of lib2to3. Basically, it would be a more high-level abstraction to substitute the current parser module. Technically you should be able to reconstruct some primitives that lib2to3 uses on top of the output that the parser module generates (modulo some extra information from the grammar), but the raw output that the parser module generates is not super useful by itself, especially when you consider the maintenance costs. On the other side, as you mention here: >I am interested in switching CPython's parsing strategy to something else (what exactly remains to be seen) and any new approach is unlikely to reuse the current CST technology. (OTOH I think it would be wise to keep the current AST.) it is true that changing the parser can influence greatly the hypothetical CST module so it may complicate the conversion to a new parser solution if the API does not abstract enough (or it may be close to impractical depending on the new parser solution). My original suggestion was based on the fact that the parser module is not super useful and it has a great maintenance cost, but the "realm" of what it solves (providing access to the parse trees) could be useful to some use cases so that is why I was talking about "parser" and lib2to3 in the same email. Perhaps we can be more productive if we focus on just deprecating the "parser" module, but I thought it was an opportunity to solve two (related) problems at once. On Mon, 20 May 2019 at 17:28, Guido van Rossum wrote: > On Thu, May 16, 2019 at 3:51 PM Pablo Galindo Salgado > wrote: > >> [Nathaniel Smith] >> > >Will the folks using forks be happy to switch to the stdlib version? >> >For example I can imagine that if black wants to process 3.7 input >> >code while running on 3.6, it might prefer a parser on PyPI even if >> >he stdlib version were public, since the PyPI version can be updated >> >independently of the host Python. >> >> The tool can parse arbitrary grammars, the one that is packed into is >> just one of them. >> >> I think it would be useful, among other things because the standard >> library >> lacks currently a proper CST solution. The ast module is heavily >> leveraged for >> things like formatters, >> > > Actually, I think the `ast` module doesn't work very well for formatters, > because it loses comments. (Retaining comments and all details of > whitespace is the specific use case for which I created pgen2.) > > >> static code analyzers...etc but CST can be very useful as >> ?ukasz describes here: >> >> https://bugs.python.org/issue33337 >> >> I think is missing an important gap in the stdlib and the closest thing >> we have >> (the current parser module) is not useful for any of that. Also, the core >> to generating >> the hypothetical new package (with some new API over it may be) is >> already undocumented >> as an implementation detail of lib2to3 (and some people are already using >> it directly). >> > > I wonder if lib2to3 is actually something that would benefit from moving > out of the stdlib. (Wasn't it on Amber's list?) As ?ukasz points out in > that issue, it is outdated. Maybe if it was out of the stdlib it would > attract more contributors. Then again, I have recently started exploring > the idea of a PEG parser for Python. Maybe a revamped version of the core > of lib2to3 based on PEG instead of pgen would be interesting to some folks. > > I do agree that the two versions of tokenize.py should be unified (and the > result kept in the stdlib). However there are some issues here, because > tokenize.py is closely tied to the names and numbers of the tokens, and the > latter information is currently generated based on the contents of the > Grammar file. This may get in the way of using it to tokenize old Python > versions. > > -- > --Guido van Rossum (python.org/~guido) > *Pronouns: he/him/his **(why is my pronoun here?)* > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 20 14:27:24 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 May 2019 04:27:24 +1000 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: <20190520182722.GH4221@ando.pearwood.info> On Mon, May 20, 2019 at 08:55:59AM -0700, Guido van Rossum wrote: > I am interested in switching CPython's parsing strategy to something else > (what exactly remains to be seen) Are you re-thinking the restriction to LL(1) grammars? -- Steven From guido at python.org Mon May 20 14:49:01 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 20 May 2019 11:49:01 -0700 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: <20190520182722.GH4221@ando.pearwood.info> References: <20190520182722.GH4221@ando.pearwood.info> Message-ID: On Mon, May 20, 2019 at 11:29 AM Steven D'Aprano wrote: > On Mon, May 20, 2019 at 08:55:59AM -0700, Guido van Rossum wrote: > > > I am interested in switching CPython's parsing strategy to something else > > (what exactly remains to be seen) > > Are you re-thinking the restriction to LL(1) grammars? > Indeed. I think it served us well for the first 10-15 years, but now it is just a burden. See a thread I started at Discourse: https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 And some followup there: https://discuss.python.org/t/preparing-for-new-python-parsing/1550 Note that this is very much speculative. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From pavel.koneski at gmail.com Mon May 20 15:26:31 2019 From: pavel.koneski at gmail.com (Pavel Koneski) Date: Mon, 20 May 2019 12:26:31 -0700 Subject: [Python-Dev] bpo-36919: Exception from 'compile' reports a newline char not present in input Message-ID: Hi, The following issue arose when implementing IronPython 3. Normally the behavior of CPython is authoritative, but in this case I started having doubts. Since Python 3.2, input in 'exec' mode of 'compile' does not have to end in a newline anymore. However, it creates a surprising behavior when a 'SyntaxError' is reported: >>> try: compile('try', '', 'exec') ... except SyntaxError as ex: print(repr(ex)) ... SyntaxError('invalid syntax', ('', 1, 4, 'try\n')) The 'text' field of the exception thrown contains an additional newline character that was not present in the input. Is it: a. Proper Python language behavior? b. CPython implementation artifact? c. A bug? The comments on bpo (https://bugs.python.org/issue36919) suggest Case B, but inconclusive. If case B it is, since IronPython does not exhibit this behavior, I will submit a patch to StdLib test(s) to make them implementation independent. Thanks for comments, Pavel From solipsis at pitrou.net Mon May 20 15:31:35 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 20 May 2019 21:31:35 +0200 Subject: [Python-Dev] PEP 587 "Python Initialization Configuration" version 4 References: Message-ID: <20190520213135.201de6f2@fsol> Hi, - Since PyInitError can be "ok", I'd rather call it "PyInitStatus". - The PyConfig example sets "module_search_paths" but not "module_search_paths_set". Is it an error? - "PyConfig_InitPythonConfig" vs "PyConfig_InitIsolatedConfig": why not "PyConfig_InitConfig" and "PyConfig_InitIsolatedConfig"? - Why is there an "isolated" field in addition to the "IsolatedConfig" functions? - PyConfig.isolated is not documented. - "Configuration files are still used with this configuration": why is this? Shouldn't they be ignored in an isolated configuration? - In "Comparison of Python and Isolated Configurations": what does -1 mean here? Overall, this looks like a great improvement. My only worry is that interactions between the various options seem complicated and difficult to understand. Perhaps we will need some detailed documentation and examples. Regards Antoine. On Mon, 20 May 2019 14:05:42 +0200 Victor Stinner wrote: > Hi, > > I expected the version 3 of my PEP to be complete, but Gregory Szorc > and Steve Dower spotted a few more issues ;-) > > The main change of the version 4 is the introduction of "Python > Configuration" and "Isolated Configuration" default configuration > which are well better defined. > > The new "Isolated Configuration" provides sane default values to > isolate Python from the system. For example, to embed Python into an > application. Using the environment are now opt-in options, rather than > an opt-out options. For example, environment variables, command line > arguments and global configuration variables are ignored by default. > > Building a customized Python which behaves as the regular Python > becomes easier using the new Py_RunMain() function. Moreover, using > the "Python Configuration", PyConfig.argv arguments are now parsed the > same way the regular Python parses command line arguments, and > PyConfig.xoptions are handled as -X opt command line options. > > I replaced all macros with functions. Macros can cause issues when > used from different programming languages, whereas functions are > always well supported. > > PyPreConfig structure doesn't allocate memory anymore (the allocator > field becomes an integer, instead of a string). I removed the > "Constant PyConfig" special case which introduced too many exceptions > for little benefit. > > See the "Version History" section for the full changes. > > HTML version: > https://www.python.org/dev/peps/pep-0587/ > > Full text below. > > Victor > > > PEP: 587 > Title: Python Initialization Configuration > Author: Victor Stinner , Nick Coghlan > BDFL-Delegate: Thomas Wouters > Discussions-To: python-dev at python.org > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 27-Mar-2019 > Python-Version: 3.8 > > Abstract > ======== > > Add a new C API to configure the Python Initialization providing finer > control on the whole configuration and better error reporting. > > It becomes possible to read the configuration and then override some > computed parameters before it is applied. It also becomes possible to > completely override how Python computes the module search paths > (``sys.path``). > > The new `Isolated Configuration`_ provides sane default values to > isolate Python from the system. For example, to embed Python into an > application. Using the environment are now opt-in options, rather than > an opt-out options. For example, environment variables, command line > arguments and global configuration variables are ignored by default. > > Building a customized Python which behaves as the regular Python becomes > easier using the new ``Py_RunMain()`` function. Moreover, using the > `Python Configuration`_, ``PyConfig.argv`` arguments are now parsed the > same way the regular Python parses command line arguments, and > ``PyConfig.xoptions`` are handled as ``-X opt`` command line options. > > This extracts a subset of the API design from the PEP 432 development > and refactoring work that is now considered sufficiently stable to make > public (allowing 3rd party embedding applications access to the same > configuration APIs that the native CPython CLI is now using). > > > Rationale > ========= > > Python is highly configurable but its configuration evolved organically. > The initialization configuration is scattered all around the code using > different ways to set them: global configuration variables (ex: > ``Py_IsolatedFlag``), environment variables (ex: ``PYTHONPATH``), > command line arguments (ex: ``-b``), configuration files (ex: > ``pyvenv.cfg``), function calls (ex: ``Py_SetProgramName()``). A > straightforward and reliable way to configure Python is needed. > > Some configuration parameters are not accessible from the C API, or not > easily. For example, there is no API to override the default values of > ``sys.executable``. > > Some options like ``PYTHONPATH`` can only be set using an environment > variable which has a side effect on Python child processes if not unset > properly. > > Some options also depends on other options: see `Priority and Rules`_. > Python 3.7 API does not provide a consistent view of the overall > configuration. > > The C API of Python 3.7 Initialization takes ``wchar_t*`` strings as > input whereas the Python filesystem encoding is set during the > initialization which can lead to mojibake. > > Python 3.7 APIs like ``Py_Initialize()`` aborts the process on memory > allocation failure which is not convenient when Python is embedded. > Moreover, ``Py_Main()`` could exit directly the process rather than > returning an exit code. Proposed new API reports the error or exit code > to the caller which can decide how to handle it. > > Implementing the PEP 540 (UTF-8 Mode) and the new ``-X dev`` correctly > was almost impossible in Python 3.6. The code base has been deeply > reworked in Python 3.7 and then in Python 3.8 to read the configuration > into a structure with no side effect. It becomes possible to clear the > configuration (release memory) and read again the configuration if the > encoding changed . It is required to implement properly the UTF-8 which > changes the encoding using ``-X utf8`` command line option. Internally, > bytes ``argv`` strings are decoded from the filesystem encoding. The > ``-X dev`` changes the memory allocator (behaves as > ``PYTHONMALLOC=debug``), whereas it was not possible to change the > memory allocation *while* parsing the command line arguments. The new > design of the internal implementation not only allowed to implement > properly ``-X utf8`` and ``-X dev``, it also allows to change the Python > behavior way more easily, especially for corner cases like that, and > ensure that the configuration remains consistent: see `Priority and > Rules`_. > > This PEP is a partial implementation of PEP 432 which is the overall > design. New fields can be added later to ``PyConfig`` structure to > finish the implementation of the PEP 432 (e.g. by adding a new partial > initialization API which allows to configure Python using Python objects to > finish the full initialization). However, those features are omitted from this > PEP as even the native CPython CLI doesn't work that way - the public API > proposal in this PEP is limited to features which have already been implemented > and adopted as private APIs for us in the native CPython CLI. > > > Python Initialization C API > =========================== > > This PEP proposes to add the following new structures, functions and > macros. > > New structures: > > * ``PyConfig`` > * ``PyInitError`` > * ``PyPreConfig`` > * ``PyWideStringList`` > > New functions: > > * ``PyConfig_Clear(config)`` > * ``PyConfig_InitIsolatedConfig()`` > * ``PyConfig_InitPythonConfig()`` > * ``PyConfig_Read(config)`` > * ``PyConfig_SetArgv(config, argc, argv)`` > * ``PyConfig_SetBytesArgv(config, argc, argv)`` > * ``PyConfig_SetBytesString(config, config_str, str)`` > * ``PyConfig_SetString(config, config_str, str)`` > * ``PyInitError_Error(err_msg)`` > * ``PyInitError_Exit(exitcode)`` > * ``PyInitError_Failed(err)`` > * ``PyInitError_IsError(err)`` > * ``PyInitError_IsExit(err)`` > * ``PyInitError_NoMemory()`` > * ``PyInitError_Ok()`` > * ``PyPreConfig_InitIsolatedConfig(preconfig)`` > * ``PyPreConfig_InitPythonConfig(preconfig)`` > * ``PyWideStringList_Append(list, item)`` > * ``PyWideStringList_Insert(list, index, item)`` > * ``Py_BytesMain(argc, argv)`` > * ``Py_ExitInitError(err)`` > * ``Py_InitializeFromConfig(config)`` > * ``Py_PreInitialize(preconfig)`` > * ``Py_PreInitializeFromArgs(preconfig, argc, argv)`` > * ``Py_PreInitializeFromBytesArgs(preconfig, argc, argv)`` > * ``Py_RunMain()`` > > This PEP also adds ``_PyRuntimeState.preconfig`` (``PyPreConfig`` type) > and ``PyInterpreterState.config`` (``PyConfig`` type) fields to these > internal structures. ``PyInterpreterState.config`` becomes the new > reference configuration, replacing global configuration variables and > other private variables. > > > PyWideStringList > ---------------- > > ``PyWideStringList`` is a list of ``wchar_t*`` strings. > > ``PyWideStringList`` structure fields: > > * ``length`` (``Py_ssize_t``) > * ``items`` (``wchar_t**``) > > Methods: > > * ``PyInitError PyWideStringList_Append(PyWideStringList *list, const > wchar_t *item)``: > Append *item* to *list*. > * ``PyInitError PyWideStringList_Insert(PyWideStringList *list, > Py_ssize_t index, const wchar_t *item)``: > Insert *item* into *list* at *index*. If *index* is greater than > *list* length, just append *item* to *list*. > > If *length* is non-zero, *items* must be non-NULL and all strings must > be non-NULL. > > PyInitError > ----------- > > ``PyInitError`` is a structure to store an error message or an exit code > for the Python Initialization. For an error, it stores the C function > name which created the error. > > Example:: > > PyInitError alloc(void **ptr, size_t size) > { > *ptr = PyMem_RawMalloc(size); > if (*ptr == NULL) { > return PyInitError_NoMemory(); > } > return PyInitError_Ok(); > } > > int main(int argc, char **argv) > { > void *ptr; > PyInitError err = alloc(&ptr, 16); > if (PyInitError_Failed(err)) { > Py_ExitInitError(err); > } > PyMem_Free(ptr); > return 0; > } > > ``PyInitError`` fields: > > * ``exitcode`` (``int``): > Argument passed to ``exit()``. > * ``err_msg`` (``const char*``): > Error message. > * ``func`` (``const char *``): > Name of the function which created an error, can be ``NULL``. > * private ``_type`` field: for internal usage only. > > Functions to create an error: > > * ``PyInitError_Ok()``: Success. > * ``PyInitError_Error(err_msg)``: Initialization error with a message. > * ``PyInitError_NoMemory()``: Memory allocation failure (out of memory). > * ``PyInitError_Exit(exitcode)``: Exit Python with the specified exit code. > > Functions to handle an error: > > * ``PyInitError_Failed(err)``: Is the result an error or an exit? > * ``PyInitError_IsError(err)``: Is the result an error? > * ``PyInitError_IsExit(err)``: Is the result an exit? > * ``Py_ExitInitError(err)``: Call ``exit(exitcode)`` if *err* is an > exit, print the error and exit if *err* is an error. Must only be > called with an error and an exit: if ``PyInitError_Failed(err)`` is > true. > > Preinitialization with PyPreConfig > ---------------------------------- > > The ``PyPreConfig`` structure is used to preinitialize Python: > > * Set the Python memory allocator > * Configure the LC_CTYPE locale > * Set the UTF-8 mode > > Example using the preinitialization to enable the UTF-8 Mode:: > > PyPreConfig preconfig; > PyPreConfig_InitPythonConfig(&preconfig); > > preconfig.utf8_mode = 1; > > PyInitError err = Py_PreInitialize(&preconfig); > if (PyInitError_Failed(err)) { > Py_ExitInitError(err); > } > > /* at this point, Python will speak UTF-8 */ > > Py_Initialize(); > /* ... use Python API here ... */ > Py_Finalize(); > > Function to initialize a pre-configuration: > > * ``void PyPreConfig_InitIsolatedConfig(PyPreConfig *preconfig)`` > * ``void PyPreConfig_InitPythonConfig(PyPreConfig *preconfig)`` > > Functions to preinitialization Python: > > * ``PyInitError Py_PreInitialize(const PyPreConfig *preconfig)`` > * ``PyInitError Py_PreInitializeFromBytesArgs(const PyPreConfig > *preconfig, int argc, char * const *argv)`` > * ``PyInitError Py_PreInitializeFromArgs(const PyPreConfig *preconfig, > int argc, wchar_t * const * argv)`` > > The caller is responsible to handle error or exit using > ``PyInitError_Failed()`` and ``Py_ExitInitError()``. > > If Python is initialized with command line arguments, the command line > arguments must also be passed to preinitialize Python, since they have > an effect on the pre-configuration like encodings. For example, the > ``-X utf8`` command line option enables the UTF-8 Mode. > > ``PyPreConfig`` fields: > > * ``allocator`` (``int``): > Name of the memory allocator (ex: ``PYMEM_ALLOCATOR_MALLOC``). Valid > values: > > * ``PYMEM_ALLOCATOR_NOT_SET`` (``0``): don't change memory allocators > (use defaults) > * ``PYMEM_ALLOCATOR_DEFAULT`` (``1``): default memory allocators > * ``PYMEM_ALLOCATOR_DEBUG`` (``2``): enable debug hooks > * ``PYMEM_ALLOCATOR_MALLOC`` (``3``): force usage of ``malloc()`` > * ``PYMEM_ALLOCATOR_MALLOC_DEBUG`` (``4``): ``malloc()`` with debug hooks > * ``PYMEM_ALLOCATOR_PYMALLOC`` (``5``): Python "pymalloc" allocator > * ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` (``6``): pymalloc with debug hooks > * Note: ``PYMEM_ALLOCATOR_PYMALLOC`` and > ``PYMEM_ALLOCATOR_PYMALLOC_DEBUG`` are not supported if Python is > configured using ``--without-pymalloc`` > > * ``configure_locale`` (``int``): > Set the LC_CTYPE locale to the user preferred locale? If equals to 0, > set ``coerce_c_locale`` and ``coerce_c_locale_warn`` to 0. > * ``coerce_c_locale`` (``int``): > If equals to 2, coerce the C locale; if equals to 1, read the LC_CTYPE > locale to decide if it should be coerced. > * ``coerce_c_locale_warn`` (``int``): > If non-zero, emit a warning if the C locale is coerced. > * ``dev_mode`` (``int``): > See ``PyConfig.dev_mode``. > * ``isolated`` (``int``): > See ``PyConfig.isolated``. > * ``legacy_windows_fs_encoding`` (``int``): > If non-zero, disable UTF-8 Mode, set the Python filesystem encoding to > ``mbcs``, set the filesystem error handler to ``replace``. > * ``parse_argv`` (``int``): > If non-zero, ``Py_PreInitializeFromArgs()`` and > ``Py_PreInitializeFromBytesArgs()`` parse their ``argv`` argument the > same way the regular Python parses command line arguments: see > `Command Line Arguments`_. > * ``use_environment`` (``int``): > See ``PyConfig.use_environment``. > * ``utf8_mode`` (``int``): > If non-zero, enable the UTF-8 mode. > > The ``legacy_windows_fs_encoding`` is only available on Windows. > > There is also a private field, for internal use only, > ``_config_version`` (``int``): the configuration version, used > for ABI compatibility. > > ``PyMem_SetAllocator()`` can be called after ``Py_PreInitialize()`` and > before ``Py_InitializeFromConfig()`` to install a custom memory > allocator. It can be called before ``Py_PreInitialize()`` if > ``allocator`` is set to ``PYMEM_ALLOCATOR_NOT_SET`` (default value). > > Python memory allocation functions like ``PyMem_RawMalloc()`` must not > be used before Python preinitialization, whereas calling directly > ``malloc()`` and ``free()`` is always safe. ``Py_DecodeLocale()`` must > not be called before the preinitialization. > > > Initialization with PyConfig > ---------------------------- > > The ``PyConfig`` structure contains most parameters to configure Python. > > Example setting the program name:: > > void init_python(void) > { > PyInitError err; > PyConfig config; > > err = PyConfig_InitPythonConfig(&config); > if (PyInitError_Failed(err)) { > goto fail; > } > > /* Set the program name. Implicitly preinitialize Python. */ > err = PyConfig_SetString(&config, &config.program_name, > L"/path/to/my_program"); > if (PyInitError_Failed(err)) { > goto fail; > } > > err = Py_InitializeFromConfig(&config); > if (PyInitError_Failed(err)) { > goto fail; > } > PyConfig_Clear(&config); > return; > > fail: > PyConfig_Clear(&config); > Py_ExitInitError(err); > } > > ``PyConfig`` methods: > > * ``PyInitError PyConfig_InitPythonConfig(PyConfig *config)`` > Initialize configuration with `Python Configuration`_. > * ``PyInitError PyConfig_InitIsolatedConfig(PyConfig *config)``: > Initialize configuration with `Isolated Configuration`_. > * ``PyInitError PyConfig_SetString(PyConfig *config, wchar_t * const > *config_str, const wchar_t *str)``: > Copy the wide character string *str* into ``*config_str``. > Preinitialize Python if needed. > * ``PyInitError PyConfig_SetBytesString(PyConfig *config, wchar_t * > const *config_str, const char *str)``: > Decode *str* using ``Py_DecodeLocale()`` and set the result into > ``*config_str``. > Preinitialize Python if needed. > * ``PyInitError PyConfig_SetArgv(PyConfig *config, int argc, wchar_t * > const *argv)``: > Set command line arguments from wide character strings. > Preinitialize Python if needed. > * ``PyInitError PyConfig_SetBytesArgv(PyConfig *config, int argc, char > * const *argv)``: > Set command line arguments: decode bytes using ``Py_DecodeLocale()``. > Preinitialize Python if needed. > * ``PyInitError PyConfig_Read(PyConfig *config)``: > Read all Python configuration. Fields which are already initialized > are left unchanged. > Preinitialize Python if needed. > * ``void PyConfig_Clear(PyConfig *config)``: > Release configuration memory. > > Most ``PyConfig`` methods preinitialize Python if needed. In that case, > the Python preinitialization configuration in based on the ``PyConfig``. > If configuration fields which are in common with ``PyPreConfig`` are > tuned, they must be set before calling a ``PyConfig`` method: > > * ``dev_mode`` > * ``isolated`` > * ``parse_argv`` > * ``use_environment`` > > Moreover, if ``PyConfig_SetArgv()`` or ``PyConfig_SetBytesArgv()`` is > used, this method must be called first, before other methods, since the > preinitialization configuration depends on command line arguments (if > ``parse_argv`` is non-zero). > > Functions to initialize Python: > > * ``PyInitError Py_InitializeFromConfig(const PyConfig *config)``: > Initialize Python from *config* configuration. > > The caller of these methods and functions is responsible to handle error > or exit using ``PyInitError_Failed()`` and ``Py_ExitInitError()``. > > ``PyConfig`` fields: > > * ``argv`` (``PyWideStringList``): > Command line arguments, ``sys.argv``. See ``parse_argv`` to parse > ``argv`` the same way the regular Python parses Python command line > arguments. If ``argv`` is empty, an empty string is added to ensure > that ``sys.argv`` always exists and is never empty. > * ``base_exec_prefix`` (``wchar_t*``): > ``sys.base_exec_prefix``. > * ``base_prefix`` (``wchar_t*``): > ``sys.base_prefix``. > * ``buffered_stdio`` (``int``): > If equals to 0, enable unbuffered mode, making the stdout and stderr streams > unbuffered. > * ``bytes_warning`` (``int``): > If equals to 1, issue a warning when comparing ``bytes`` or > ``bytearray`` with ``str``, or comparing ``bytes`` with ``int``. If > equal or greater to 2, raise a ``BytesWarning`` exception. > * ``check_hash_pycs_mode`` (``wchar_t*``): > ``--check-hash-based-pycs`` command line option value (see PEP 552). > * ``configure_c_stdio`` (``int``): > If non-zero, configure C standard streams (``stdio``, ``stdout``, > ``stdout``). For example, set their mode to ``O_BINARY`` on Windows. > * ``dev_mode`` (``int``): > Development mode > * ``dump_refs`` (``int``): > If non-zero, dump all objects which are still alive at exit > * ``exec_prefix`` (``wchar_t*``): > ``sys.exec_prefix``. > * ``executable`` (``wchar_t*``): > ``sys.executable``. > * ``faulthandler`` (``int``): > If non-zero, call ``faulthandler.enable()``. > * ``filesystem_encoding`` (``wchar_t*``): > Filesystem encoding, ``sys.getfilesystemencoding()``. > * ``filesystem_errors`` (``wchar_t*``): > Filesystem encoding errors, ``sys.getfilesystemencodeerrors()``. > * ``use_hash_seed`` (``int``), > ``hash_seed`` (``unsigned long``): > Randomized hash function seed. > * ``home`` (``wchar_t*``): > Python home directory. > * ``import_time`` (``int``): > If non-zero, profile import time. > * ``inspect`` (``int``): > Enter interactive mode after executing a script or a command. > * ``install_signal_handlers`` (``int``): > Install signal handlers? > * ``interactive`` (``int``): > Interactive mode. > * ``legacy_windows_stdio`` (``int``, Windows only): > If non-zero, use ``io.FileIO`` instead of ``WindowsConsoleIO`` for > ``sys.stdin``, ``sys.stdout`` and ``sys.stderr``. > * ``malloc_stats`` (``int``): > If non-zero, dump memory allocation statistics at exit. > * ``pythonpath_env`` (``wchar_t*``): > Module search paths as a string separated by DELIM (usually ``:``). > Initialized from ``PYTHONPATH`` environment variable value by default. > * ``module_search_paths_set`` (``int``), > ``module_search_paths`` (``PyWideStringList``): > ``sys.path``. If ``module_search_paths_set`` is equal to 0, the > ``module_search_paths`` is replaced by the function computing the > `Path Configuration`. > * ``optimization_level`` (``int``): > Compilation optimization level. > * ``parse_argv`` (``int``): > If non-zero, parse ``argv`` the same way the regular Python command > line arguments, and strip Python arguments from ``argv``: see `Command > Line Arguments`_. > * ``parser_debug`` (``int``): > If non-zero, turn on parser debugging output (for expert only, > depending on compilation options). > * ``pathconfig_warnings`` (``int``): > If equal to 0, suppress warnings when computing the path > configuration (Unix only, Windows does not log any warning). Otherwise, > warnings are written into stderr. > * ``prefix`` (``wchar_t*``): > ``sys.prefix``. > * ``program_name`` (``wchar_t*``): > Program name. > * ``pycache_prefix`` (``wchar_t*``): > ``.pyc`` cache prefix. > * ``quiet`` (``int``): > Quiet mode. For example, don't display the copyright and version > messages even in interactive mode. > * ``run_command`` (``wchar_t*``): > ``-c COMMAND`` argument. > * ``run_filename`` (``wchar_t*``): > ``python3 SCRIPT`` argument. > * ``run_module`` (``wchar_t*``): > ``python3 -m MODULE`` argument. > * ``show_alloc_count`` (``int``): > Show allocation counts at exit? > * ``show_ref_count`` (``int``): > Show total reference count at exit? > * ``site_import`` (``int``): > Import the ``site`` module at startup? > * ``skip_source_first_line`` (``int``): > Skip the first line of the source? > * ``stdio_encoding`` (``wchar_t*``), > ``stdio_errors`` (``wchar_t*``): > Encoding and encoding errors of ``sys.stdin``, ``sys.stdout`` > and ``sys.stderr``. > * ``tracemalloc`` (``int``): > If non-zero, call ``tracemalloc.start(value)``. > * ``user_site_directory`` (``int``): > If non-zero, add user site directory to ``sys.path``. > * ``verbose`` (``int``): > If non-zero, enable verbose mode. > * ``warnoptions`` (``PyWideStringList``): > Options of the ``warnings`` module to build warnings filters. > * ``write_bytecode`` (``int``): > If non-zero, write ``.pyc`` files. > * ``xoptions`` (``PyWideStringList``): > ``sys._xoptions``. > > If ``parse_argv`` is non-zero, ``argv`` arguments are parsed the same > way the regular Python parses command line arguments, and Python > arguments are stripped from ``argv``: see `Command Line Arguments`_. > > The ``xoptions`` options are parsed to set other options: see `-X > Options`_. > > ``PyConfig`` private fields, for internal use only: > > * ``_config_version`` (``int``): > Configuration version, used for ABI compatibility. > * ``_config_init`` (``int``): > Function used to initalize ``PyConfig``, used for preinitialization. > * ``_install_importlib`` (``int``): > Install importlib? > * ``_init_main`` (``int``): > If equal to 0, stop Python initialization before the "main" phase > (see PEP 432). > > More complete example modifying the default configuration, read the > configuration, and then override some parameters:: > > PyInitError init_python(const char *program_name) > { > PyInitError err; > PyConfig config; > > err = PyConfig_InitPythonConfig(&config); > if (PyInitError_Failed(err)) { > goto done; > } > > /* Set the program name before reading the configuraton > (decode byte string from the locale encoding). > > Implicitly preinitialize Python. */ > err = PyConfig_SetBytesString(&config, &config.program_name, > program_name); > if (PyInitError_Failed(err)) { > goto done; > } > > /* Read all configuration at once */ > err = PyConfig_Read(&config); > if (PyInitError_Failed(err)) { > goto done; > } > > /* Append our custom search path to sys.path */ > err = PyWideStringList_Append(&config.module_search_paths, > L"/path/to/more/modules"); > if (PyInitError_Failed(err)) { > goto done; > } > > /* Override executable computed by PyConfig_Read() */ > err = PyConfig_SetString(&config, &config.executable, > L"/path/to/my_executable"); > if (PyInitError_Failed(err)) { > goto done; > } > > err = Py_InitializeFromConfig(&config); > > done: > PyConfig_Clear(&config); > return err; > } > > .. note:: > ``PyImport_FrozenModules``, ``PyImport_AppendInittab()`` and > ``PyImport_ExtendInittab()`` functions are still relevant and > continue to work as previously. They should be set or called > before the Python initialization. > > > Isolated Configuration > ---------------------- > > ``PyPreConfig_InitIsolatedConfig()`` and > ``PyConfig_InitIsolatedConfig()`` functions create a configuration to > isolate Python from the system. For example, to embed Python into an > application. > > This configuration ignores global configuration variables, environments > variables and command line arguments (``argv`` is not parsed). The C > standard streams (ex: ``stdout``) and the LC_CTYPE locale are left > unchanged by default. > > Configuration files are still used with this configuration. Set the > `Path Configuration`_ ("output fields") to ignore these configuration > files and avoid the function computing the default path configuration. > > > Python Configuration > -------------------- > > ``PyPreConfig_InitPythonConfig()`` and ``PyConfig_InitPythonConfig()`` > functions create a configuration to build a customized Python which > behaves as the regular Python. > > Environments variables and command line arguments are used to configure > Python, whereas global configuration variables are ignored. > > This function enables C locale coercion (PEP 538) and UTF-8 Mode (PEP > 540) depending on the LC_CTYPE locale, ``PYTHONUTF8`` and > ``PYTHONCOERCECLOCALE`` environment variables. > > Example of customized Python always running in isolated mode:: > > int main(int argc, char **argv) > { > PyConfig config; > PyInitError err; > > err = PyConfig_InitPythonConfig(&config); > if (PyInitError_Failed(err)) { > goto fail; > } > > config.isolated = 1; > > /* Decode command line arguments. > Implicitly preinitialize Python (in isolated mode). */ > err = PyConfig_SetBytesArgv(&config, argc, argv); > if (PyInitError_Failed(err)) { > goto fail; > } > > err = Py_InitializeFromConfig(&config); > if (PyInitError_Failed(err)) { > goto fail; > } > PyConfig_Clear(&config); > > return Py_RunMain(); > > fail: > PyConfig_Clear(&config); > if (!PyInitError_IsExit(err)) { > /* Display the error message and exit the process with > non-zero exit code */ > Py_ExitInitError(err); > } > return err.exitcode; > } > > This example is a basic implementation of the "System Python Executable" > discussed in PEP 432. > > > Path Configuration > ------------------ > > ``PyConfig`` contains multiple fields for the path configuration: > > * Path configuration input fields: > > * ``home`` > * ``pythonpath_env`` > * ``pathconfig_warnings`` > > * Path configuration output fields: > > * ``exec_prefix`` > * ``executable`` > * ``prefix`` > * ``module_search_paths_set``, ``module_search_paths`` > > It is possible to completely ignore the function computing the default > path configuration by setting explicitly all path configuration output > fields listed above. A string is considered as set even if it's an empty > string. ``module_search_paths`` is considered as set if > ``module_search_paths_set`` is set to 1. In this case, path > configuration input fields are ignored as well. > > Set ``pathconfig_warnings`` to 0 to suppress warnings when computing the > path configuration (Unix only, Windows does not log any warning). > > If ``base_prefix`` or ``base_exec_prefix`` fields are not set, they > inherit their value from ``prefix`` and ``exec_prefix`` respectively. > > If ``site_import`` is non-zero, ``sys.path`` can be modified by the > ``site`` module. For example, if ``user_site_directory`` is non-zero, > the user site directory is added to ``sys.path`` (if it exists). > > See also `Configuration Files`_ used by the path configuration. > > > Py_BytesMain() > -------------- > > Python 3.7 provides a high-level ``Py_Main()`` function which requires > to pass command line arguments as ``wchar_t*`` strings. It is > non-trivial to use the correct encoding to decode bytes. Python has its > own set of issues with C locale coercion and UTF-8 Mode. > > This PEP adds a new ``Py_BytesMain()`` function which takes command line > arguments as bytes:: > > int Py_BytesMain(int argc, char **argv) > > Py_RunMain() > ------------ > > The new ``Py_RunMain()`` function executes the command > (``PyConfig.run_command``), the script (``PyConfig.run_filename``) or > the module (``PyConfig.run_module``) specified on the command line or in > the configuration, and then finalizes Python. It returns an exit status > that can be passed to the ``exit()`` function. :: > > int Py_RunMain(void); > > See `Python Configuration`_ for an example of customized Python always > running in isolated mode using ``Py_RunMain()``. > > > Backwards Compatibility > ======================= > > This PEP only adds a new API: it leaves the existing API unchanged and > has no impact on the backwards compatibility. > > The Python 3.7 ``Py_Initialize()`` function now disable the C locale > coercion (PEP 538) and the UTF-8 Mode (PEP 540) by default to prevent > mojibake. The new API using the `Python Configuration`_ is needed to > enable them automatically. > > > Annexes > ======= > > Comparison of Python and Isolated Configurations > ------------------------------------------------ > > Differences between ``PyPreConfig_InitPythonConfig()`` > and ``PyPreConfig_InitIsolatedConfig()``: > > =============================== ======= ======== > PyPreConfig Python Isolated > =============================== ======= ======== > ``coerce_c_locale_warn`` -1 0 > ``coerce_c_locale`` -1 0 > ``configure_locale`` **1** 0 > ``dev_mode`` -1 0 > ``isolated`` -1 **1** > ``legacy_windows_fs_encoding`` -1 0 > ``use_environment`` -1 0 > ``parse_argv`` **1** 0 > ``utf8_mode`` -1 0 > =============================== ======= ======== > > Differences between ``PyConfig_InitPythonConfig()`` > and ``PyConfig_InitIsolatedConfig()``: > > =============================== ======= ======== > PyConfig Python Isolated > =============================== ======= ======== > ``configure_c_stdio`` **1** 0 > ``install_signal_handlers`` **1** 0 > ``isolated`` 0 **1** > ``parse_argv`` **1** 0 > ``pathconfig_warnings`` **1** 0 > ``use_environment`` **1** 0 > ``user_site_directory`` **1** 0 > =============================== ======= ======== > > Priority and Rules > ------------------ > > Priority of configuration parameters, highest to lowest: > > * ``PyConfig`` > * ``PyPreConfig`` > * Configuration files > * Command line options > * Environment variables > * Global configuration variables > > Priority of warning options, highest to lowest: > > * ``PyConfig.warnoptions`` > * ``PyConfig.dev_mode`` (add ``"default"``) > * ``PYTHONWARNINGS`` environment variables > * ``-W WARNOPTION`` command line argument > * ``PyConfig.bytes_warning`` (add ``"error::BytesWarning"`` if greater > than 1, or add ``"default::BytesWarning``) > > Rules on ``PyConfig`` parameters: > > * If ``isolated`` is non-zero, ``use_environment`` and > ``user_site_directory`` are set to 0. > * If ``dev_mode`` is non-zero, ``allocator`` is set to ``"debug"``, > ``faulthandler`` is set to 1, and ``"default"`` filter is added to > ``warnoptions``. But the ``PYTHONMALLOC`` environment variable has the > priority over ``dev_mode`` to set the memory allocator. > * If ``base_prefix`` is not set, it inherits ``prefix`` value. > * If ``base_exec_prefix`` is not set, it inherits ``exec_prefix`` value. > * If the ``python._pth`` configuration file is present, ``isolated`` is > set to 1 and ``site_import`` is set to 0; but ``site_import`` is set > to 1 if ``python._pth`` contains ``import site``. > > Rules on ``PyConfig`` and ``PyPreConfig`` parameters: > > * If ``PyPreConfig.legacy_windows_fs_encoding`` is non-zero, > set ``PyPreConfig.utf8_mode`` to 0, set > ``PyConfig.filesystem_encoding`` to ``mbcs``, and set > ``PyConfig.filesystem_errors`` to ``replace``. > > Configuration Files > ------------------- > > Python configuration files used by the `Path Configuration`_: > > * ``pyvenv.cfg`` > * ``python._pth`` (Windows only) > * ``pybuilddir.txt`` (Unix only) > > Global Configuration Variables > ------------------------------ > > Global configuration variables mapped to ``PyPreConfig`` fields: > > ======================================== ================================ > Variable Field > ======================================== ================================ > ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) > ``Py_IsolatedFlag`` ``isolated`` > ``Py_LegacyWindowsFSEncodingFlag`` ``legacy_windows_fs_encoding`` > ``Py_UTF8Mode`` ``utf8_mode`` > ======================================== ================================ > > (NOT) means that the ``PyPreConfig`` value is the oposite of the global > configuration variable value. ``Py_LegacyWindowsFSEncodingFlag`` is only > available on Windows. > > Global configuration variables mapped to ``PyConfig`` fields: > > ======================================== ================================ > Variable Field > ======================================== ================================ > ``Py_BytesWarningFlag`` ``bytes_warning`` > ``Py_DebugFlag`` ``parser_debug`` > ``Py_DontWriteBytecodeFlag`` ``write_bytecode`` (NOT) > ``Py_FileSystemDefaultEncodeErrors`` ``filesystem_errors`` > ``Py_FileSystemDefaultEncoding`` ``filesystem_encoding`` > ``Py_FrozenFlag`` ``pathconfig_warnings`` (NOT) > ``Py_HasFileSystemDefaultEncoding`` ``filesystem_encoding`` > ``Py_HashRandomizationFlag`` ``use_hash_seed``, ``hash_seed`` > ``Py_IgnoreEnvironmentFlag`` ``use_environment`` (NOT) > ``Py_InspectFlag`` ``inspect`` > ``Py_InteractiveFlag`` ``interactive`` > ``Py_IsolatedFlag`` ``isolated`` > ``Py_LegacyWindowsStdioFlag`` ``legacy_windows_stdio`` > ``Py_NoSiteFlag`` ``site_import`` (NOT) > ``Py_NoUserSiteDirectory`` ``user_site_directory`` (NOT) > ``Py_OptimizeFlag`` ``optimization_level`` > ``Py_QuietFlag`` ``quiet`` > ``Py_UnbufferedStdioFlag`` ``buffered_stdio`` (NOT) > ``Py_VerboseFlag`` ``verbose`` > ``_Py_HasFileSystemDefaultEncodeErrors`` ``filesystem_errors`` > ======================================== ================================ > > (NOT) means that the ``PyConfig`` value is the oposite of the global > configuration variable value. ``Py_LegacyWindowsStdioFlag`` is only > available on Windows. > > > Command Line Arguments > ---------------------- > > Usage:: > > python3 [options] > python3 [options] -c COMMAND > python3 [options] -m MODULE > python3 [options] SCRIPT > > > Command line options mapped to pseudo-action on ``PyPreConfig`` fields: > > ================================ ================================ > Option ``PyConfig`` field > ================================ ================================ > ``-E`` ``use_environment = 0`` > ``-I`` ``isolated = 1`` > ``-X dev`` ``dev_mode = 1`` > ``-X utf8`` ``utf8_mode = 1`` > ``-X utf8=VALUE`` ``utf8_mode = VALUE`` > ================================ ================================ > > Command line options mapped to pseudo-action on ``PyConfig`` fields: > > ================================ ================================ > Option ``PyConfig`` field > ================================ ================================ > ``-b`` ``bytes_warning++`` > ``-B`` ``write_bytecode = 0`` > ``-c COMMAND`` ``run_command = COMMAND`` > ``--check-hash-based-pycs=MODE`` ``_check_hash_pycs_mode = MODE`` > ``-d`` ``parser_debug++`` > ``-E`` ``use_environment = 0`` > ``-i`` ``inspect++`` and ``interactive++`` > ``-I`` ``isolated = 1`` > ``-m MODULE`` ``run_module = MODULE`` > ``-O`` ``optimization_level++`` > ``-q`` ``quiet++`` > ``-R`` ``use_hash_seed = 0`` > ``-s`` ``user_site_directory = 0`` > ``-S`` ``site_import`` > ``-t`` ignored (kept for backwards compatibility) > ``-u`` ``buffered_stdio = 0`` > ``-v`` ``verbose++`` > ``-W WARNING`` add ``WARNING`` to ``warnoptions`` > ``-x`` ``skip_source_first_line = 1`` > ``-X OPTION`` add ``OPTION`` to ``xoptions`` > ================================ ================================ > > ``-h``, ``-?`` and ``-V`` options are handled without ``PyConfig``. > > -X Options > ---------- > > -X options mapped to pseudo-action on ``PyConfig`` fields: > > ================================ ================================ > Option ``PyConfig`` field > ================================ ================================ > ``-X dev`` ``dev_mode = 1`` > ``-X faulthandler`` ``faulthandler = 1`` > ``-X importtime`` ``import_time = 1`` > ``-X pycache_prefix=PREFIX`` ``pycache_prefix = PREFIX`` > ``-X showalloccount`` ``show_alloc_count = 1`` > ``-X showrefcount`` ``show_ref_count = 1`` > ``-X tracemalloc=N`` ``tracemalloc = N`` > ================================ ================================ > > Environment Variables > --------------------- > > Environment variables mapped to ``PyPreConfig`` fields: > > ================================= ============================================= > Variable ``PyPreConfig`` field > ================================= ============================================= > ``PYTHONCOERCECLOCALE`` ``coerce_c_locale``, ``coerce_c_locale_warn`` > ``PYTHONDEVMODE`` ``dev_mode`` > ``PYTHONLEGACYWINDOWSFSENCODING`` ``legacy_windows_fs_encoding`` > ``PYTHONMALLOC`` ``allocator`` > ``PYTHONUTF8`` ``utf8_mode`` > ================================= ============================================= > > Environment variables mapped to ``PyConfig`` fields: > > ================================= ==================================== > Variable ``PyConfig`` field > ================================= ==================================== > ``PYTHONDEBUG`` ``parser_debug`` > ``PYTHONDEVMODE`` ``dev_mode`` > ``PYTHONDONTWRITEBYTECODE`` ``write_bytecode`` > ``PYTHONDUMPREFS`` ``dump_refs`` > ``PYTHONEXECUTABLE`` ``program_name`` > ``PYTHONFAULTHANDLER`` ``faulthandler`` > ``PYTHONHASHSEED`` ``use_hash_seed``, ``hash_seed`` > ``PYTHONHOME`` ``home`` > ``PYTHONINSPECT`` ``inspect`` > ``PYTHONIOENCODING`` ``stdio_encoding``, ``stdio_errors`` > ``PYTHONLEGACYWINDOWSSTDIO`` ``legacy_windows_stdio`` > ``PYTHONMALLOCSTATS`` ``malloc_stats`` > ``PYTHONNOUSERSITE`` ``user_site_directory`` > ``PYTHONOPTIMIZE`` ``optimization_level`` > ``PYTHONPATH`` ``pythonpath_env`` > ``PYTHONPROFILEIMPORTTIME`` ``import_time`` > ``PYTHONPYCACHEPREFIX,`` ``pycache_prefix`` > ``PYTHONTRACEMALLOC`` ``tracemalloc`` > ``PYTHONUNBUFFERED`` ``buffered_stdio`` > ``PYTHONVERBOSE`` ``verbose`` > ``PYTHONWARNINGS`` ``warnoptions`` > ================================= ==================================== > > ``PYTHONLEGACYWINDOWSFSENCODING`` and ``PYTHONLEGACYWINDOWSSTDIO`` are > specific to Windows. > > > Default Python Configugration > ----------------------------- > > ``PyPreConfig_InitPythonConfig()``: > > * ``allocator`` = ``PYMEM_ALLOCATOR_NOT_SET`` > * ``coerce_c_locale_warn`` = -1 > * ``coerce_c_locale`` = -1 > * ``configure_locale`` = 1 > * ``dev_mode`` = -1 > * ``isolated`` = -1 > * ``legacy_windows_fs_encoding`` = -1 > * ``use_environment`` = -1 > * ``utf8_mode`` = -1 > > ``PyConfig_InitPythonConfig()``: > > * ``argv`` = [] > * ``base_exec_prefix`` = ``NULL`` > * ``base_prefix`` = ``NULL`` > * ``buffered_stdio`` = 1 > * ``bytes_warning`` = 0 > * ``check_hash_pycs_mode`` = ``NULL`` > * ``configure_c_stdio`` = 1 > * ``dev_mode`` = 0 > * ``dump_refs`` = 0 > * ``exec_prefix`` = ``NULL`` > * ``executable`` = ``NULL`` > * ``faulthandler`` = 0 > * ``filesystem_encoding`` = ``NULL`` > * ``filesystem_errors`` = ``NULL`` > * ``hash_seed`` = 0 > * ``home`` = ``NULL`` > * ``import_time`` = 0 > * ``inspect`` = 0 > * ``install_signal_handlers`` = 1 > * ``interactive`` = 0 > * ``isolated`` = 0 > * ``malloc_stats`` = 0 > * ``module_search_path_env`` = ``NULL`` > * ``module_search_paths`` = [] > * ``optimization_level`` = 0 > * ``parse_argv`` = 1 > * ``parser_debug`` = 0 > * ``pathconfig_warnings`` = 1 > * ``prefix`` = ``NULL`` > * ``program_name`` = ``NULL`` > * ``pycache_prefix`` = ``NULL`` > * ``quiet`` = 0 > * ``run_command`` = ``NULL`` > * ``run_filename`` = ``NULL`` > * ``run_module`` = ``NULL`` > * ``show_alloc_count`` = 0 > * ``show_ref_count`` = 0 > * ``site_import`` = 1 > * ``skip_source_first_line`` = 0 > * ``stdio_encoding`` = ``NULL`` > * ``stdio_errors`` = ``NULL`` > * ``tracemalloc`` = 0 > * ``use_environment`` = 1 > * ``use_hash_seed`` = 0 > * ``user_site_directory`` = 1 > * ``verbose`` = 0 > * ``warnoptions`` = [] > * ``write_bytecode`` = 1 > * ``xoptions`` = [] > * ``_init_main`` = 1 > * ``_install_importlib`` = 1 > > > Default Isolated Configugration > ------------------------------- > > ``PyPreConfig_InitIsolatedConfig()``: > > * ``allocator`` = ``PYMEM_ALLOCATOR_NOT_SET`` > * ``coerce_c_locale_warn`` = 0 > * ``coerce_c_locale`` = 0 > * ``configure_locale`` = 0 > * ``dev_mode`` = 0 > * ``isolated`` = 1 > * ``legacy_windows_fs_encoding`` = 0 > * ``use_environment`` = 0 > * ``utf8_mode`` = 0 > > ``PyConfig_InitIsolatedConfig()``: > > * ``argv`` = [] > * ``base_exec_prefix`` = ``NULL`` > * ``base_prefix`` = ``NULL`` > * ``buffered_stdio`` = 1 > * ``bytes_warning`` = 0 > * ``check_hash_pycs_mode`` = ``NULL`` > * ``configure_c_stdio`` = 0 > * ``dev_mode`` = 0 > * ``dump_refs`` = 0 > * ``exec_prefix`` = ``NULL`` > * ``executable`` = ``NULL`` > * ``faulthandler`` = 0 > * ``filesystem_encoding`` = ``NULL`` > * ``filesystem_errors`` = ``NULL`` > * ``hash_seed`` = 0 > * ``home`` = ``NULL`` > * ``import_time`` = 0 > * ``inspect`` = 0 > * ``install_signal_handlers`` = 0 > * ``interactive`` = 0 > * ``isolated`` = 1 > * ``malloc_stats`` = 0 > * ``module_search_path_env`` = ``NULL`` > * ``module_search_paths`` = [] > * ``optimization_level`` = 0 > * ``parse_argv`` = 0 > * ``parser_debug`` = 0 > * ``pathconfig_warnings`` = 0 > * ``prefix`` = ``NULL`` > * ``program_name`` = ``NULL`` > * ``pycache_prefix`` = ``NULL`` > * ``quiet`` = 0 > * ``run_command`` = ``NULL`` > * ``run_filename`` = ``NULL`` > * ``run_module`` = ``NULL`` > * ``show_alloc_count`` = 0 > * ``show_ref_count`` = 0 > * ``site_import`` = 1 > * ``skip_source_first_line`` = 0 > * ``stdio_encoding`` = ``NULL`` > * ``stdio_errors`` = ``NULL`` > * ``tracemalloc`` = 0 > * ``use_environment`` = 0 > * ``use_hash_seed`` = 0 > * ``user_site_directory`` = 0 > * ``verbose`` = 0 > * ``warnoptions`` = [] > * ``write_bytecode`` = 1 > * ``xoptions`` = [] > * ``_init_main`` = 1 > * ``_install_importlib`` = 1 > > > Python 3.7 API > -------------- > > Python 3.7 has 4 functions in its C API to initialize and finalize > Python: > > * ``Py_Initialize()``, ``Py_InitializeEx()``: initialize Python > * ``Py_Finalize()``, ``Py_FinalizeEx()``: finalize Python > > Python 3.7 can be configured using `Global Configuration Variables`_, > `Environment Variables`_, and the following functions: > > * ``PyImport_AppendInittab()`` > * ``PyImport_ExtendInittab()`` > * ``PyMem_SetAllocator()`` > * ``PyMem_SetupDebugHooks()`` > * ``PyObject_SetArenaAllocator()`` > * ``Py_SetPath()`` > * ``Py_SetProgramName()`` > * ``Py_SetPythonHome()`` > * ``Py_SetStandardStreamEncoding()`` > * ``PySys_AddWarnOption()`` > * ``PySys_AddXOption()`` > * ``PySys_ResetWarnOptions()`` > > There is also a high-level ``Py_Main()`` function and > ``PyImport_FrozenModules`` variable which can be overridden. > > See `Initialization, Finalization, and Threads > `_ documentation. > > > Python Issues > ============= > > Issues that will be fixed by this PEP, directly or indirectly: > > * `bpo-1195571 `_: "simple > callback system for Py_FatalError" > * `bpo-11320 `_: > "Usage of API method Py_SetPath causes errors in Py_Initialize() > (Posix ony)" > * `bpo-13533 `_: "Would like > Py_Initialize to play friendly with host app" > * `bpo-14956 `_: "custom PYTHONPATH > may break apps embedding Python" > * `bpo-19983 `_: "When interrupted > during startup, Python should not call abort() but exit()" > * `bpo-22213 `_: "Make pyvenv style > virtual environments easier to configure when embedding Python". This > PEP more or > * `bpo-22257 `_: "PEP 432: Redesign > the interpreter startup sequence" > * `bpo-29778 `_: "_Py_CheckPython3 > uses uninitialized dllpath when embedder sets module path with > Py_SetPath" > * `bpo-30560 `_: "Add > Py_SetFatalErrorAbortFunc: Allow embedding program to handle fatal > errors". > * `bpo-31745 `_: "Overloading > "Py_GetPath" does not work" > * `bpo-32573 `_: "All sys attributes > (.argv, ...) should exist in embedded environments". > * `bpo-34725 `_: > "Py_GetProgramFullPath() odd behaviour in Windows" > * `bpo-36204 `_: "Deprecate calling > Py_Main() after Py_Initialize()? Add Py_InitializeFromArgv()?" > * `bpo-33135 `_: "Define field > prefixes for the various config structs". The PEP now defines well > how warnings options are handled. > > Issues of the PEP implementation: > > * `bpo-16961 `_: "No regression > tests for -E and individual environment vars" > * `bpo-20361 `_: "-W command line > options and PYTHONWARNINGS environmental variable should not override > -b / -bb command line options" > * `bpo-26122 `_: "Isolated mode > doesn't ignore PYTHONHASHSEED" > * `bpo-29818 `_: > "Py_SetStandardStreamEncoding leads to a memory error in debug mode" > * `bpo-31845 `_: > "PYTHONDONTWRITEBYTECODE and PYTHONOPTIMIZE have no effect" > * `bpo-32030 `_: "PEP 432: Rewrite > Py_Main()" > * `bpo-32124 `_: "Document functions > safe to be called before Py_Initialize()" > * `bpo-33042 `_: "New 3.7 startup > sequence crashes PyInstaller" > * `bpo-33932 `_: "Calling > Py_Initialize() twice now triggers a fatal error (Python 3.7)" > * `bpo-34008 `_: "Do we support > calling Py_Main() after Py_Initialize()?" > * `bpo-34170 `_: "Py_Initialize(): > computing path configuration must not have side effect (PEP 432)" > * `bpo-34589 `_: "Py_Initialize() > and Py_Main() should not enable C locale coercion" > * `bpo-34639 `_: > "PYTHONCOERCECLOCALE is ignored when using -E or -I option" > * `bpo-36142 `_: "Add a new > _PyPreConfig step to Python initialization to setup memory allocator > and encodings" > * `bpo-36202 `_: "Calling > Py_DecodeLocale() before _PyPreConfig_Write() can produce mojibake" > * `bpo-36301 `_: "Add > _Py_PreInitialize() function" > * `bpo-36443 `_: "Disable > coerce_c_locale and utf8_mode by default in _PyPreConfig?" > * `bpo-36444 `_: "Python > initialization: remove _PyMainInterpreterConfig" > * `bpo-36471 `_: "PEP 432, PEP 587: > Add _Py_RunMain()" > * `bpo-36763 `_: "PEP 587: Rework > initialization API to prepare second version of the PEP" > * `bpo-36775 `_: "Rework filesystem > codec implementation" > * `bpo-36900 `_: "Use _PyCoreConfig > rather than global configuration variables" > > Issues related to this PEP: > > * `bpo-12598 `_: "Move sys variable > initialization from import.c to sysmodule.c" > * `bpo-15577 `_: "Real argc and argv > in embedded interpreter" > * `bpo-16202 `_: "sys.path[0] > security issues" > * `bpo-18309 `_: "Make python > slightly more relocatable" > * `bpo-25631 `_: "Segmentation fault > with invalid Unicode command-line arguments in embedded Python" > * `bpo-26007 `_: "Support embedding > the standard library in an executable" > * `bpo-31210 `_: "Can not import > modules if sys.prefix contains DELIM". > * `bpo-31349 `_: "Embedded > initialization ignores Py_SetProgramName()" > * `bpo-33919 `_: "Expose > _PyCoreConfig structure to Python" > * `bpo-35173 `_: "Re-use already > existing functionality to allow Python 2.7.x (both embedded and > standalone) to locate the module path according to the shared library" > > > Version History > =============== > > * Version 4: > > * Introduce "Python Configuration" and "Isolated Configuration" which > are well better defined. Replace all macros with functions. > * Replace ``PyPreConfig_INIT`` and ``PyConfig_INIT`` macros with > functions: > > * ``PyPreConfig_InitIsolatedConfig()``, ``PyConfig_InitIsolatedConfig()`` > * ``PyPreConfig_InitPythonConfig()``, ``PyConfig_InitPythonConfig()`` > > * ``PyPreConfig`` no longer uses dynamic memory, the ``allocator`` > field type becomes an int, add ``configure_locale`` and > ``parse_argv`` field. > * ``PyConfig``: rename ``module_search_path_env`` to > ``pythonpath_env``, rename ``use_module_search_paths`` to > ``module_search_paths_set``, remove ``program`` and ``dll_path``. > * Replace ``Py_INIT_xxx()`` macros with ``PyInitError_xxx()`` > functions. > * Remove the "Constant PyConfig" section. Remove > ``Py_InitializeFromArgs()`` and ``Py_InitializeFromBytesArgs()`` > functions. > > * Version 3: > > * ``PyConfig``: Add ``configure_c_stdio`` and ``parse_argv``; > rename ``_frozen`` to ``pathconfig_warnings``. > * Rename functions using bytes strings and wide character strings. For > example, ``Py_PreInitializeFromWideArgs()`` becomes > ``Py_PreInitializeFromArgs()``, and ``PyConfig_SetArgv()`` becomes > ``PyConfig_SetBytesArgv()``. > * Add ``PyWideStringList_Insert()`` function. > * New "Path configuration", "Isolate Python", "Python Issues" > and "Version History" sections. > * ``PyConfig_SetString()`` and ``PyConfig_SetBytesString()`` now > requires the configuration as the first argument. > * Rename ``Py_UnixMain()`` to ``Py_BytesMain()`` > > * Version 2: Add ``PyConfig`` methods (ex: ``PyConfig_Read()``), add > ``PyWideStringList_Append()``, rename ``PyWideCharList`` to > ``PyWideStringList``. > * Version 1: Initial version. > > Copyright > ========= > > This document has been placed in the public domain. From guido at python.org Mon May 20 15:46:12 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 20 May 2019 12:46:12 -0700 Subject: [Python-Dev] bpo-36919: Exception from 'compile' reports a newline char not present in input In-Reply-To: References: Message-ID: I answered (B) in the tracker. On Mon, May 20, 2019 at 12:32 PM Pavel Koneski wrote: > Hi, > > The following issue arose when implementing IronPython 3. Normally the > behavior of CPython is authoritative, but in this case I started having > doubts. > > Since Python 3.2, input in 'exec' mode of 'compile' does not have to > end in a newline anymore. However, it creates a surprising behavior when > a 'SyntaxError' is reported: > > >>> try: compile('try', '', 'exec') > ... except SyntaxError as ex: print(repr(ex)) > ... > SyntaxError('invalid syntax', ('', 1, 4, 'try\n')) > > The 'text' field of the exception thrown contains an additional newline > character that was not present in the input. Is it: > > a. Proper Python language behavior? > b. CPython implementation artifact? > c. A bug? > > The comments on bpo (https://bugs.python.org/issue36919) suggest Case B, > but inconclusive. If case B it is, since IronPython does not exhibit > this behavior, I will submit a patch to StdLib test(s) to make them > implementation independent. > > Thanks for comments, > Pavel > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Mon May 20 16:15:22 2019 From: christian at python.org (Christian Heimes) Date: Mon, 20 May 2019 22:15:22 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library Message-ID: Hi, here is the first version of my PEP 594 to deprecate and eventually remove modules from the standard library. The PEP started last year with talk during the Python Language Summit 2018, https://lwn.net/Articles/755229/. The PEP can be confirmed in two stages. I'm not planning any code changes for 3.8. Instead I only like to document a bunch of modules as deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The long deprecation phase gives us 3 years to change our minds or handle edge cases, too. Regards, Christian ---------------------------------------------------------------------------- PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes Status: Active Type: Process Content-Type: text/x-rst Created: 20-May-2019 Post-History: Abstract ======== This PEP proposed a list of standard library modules to be removed from the standard library. The modules are mostly historic data formats and APIs that have been superseded a long time ago, e.g. Mac OS 9 and Commodore. Rationale ========= Back in the early days of Python, the interpreter came with a large set of useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email. Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions. On the other hand, Python's standard library is piling up cruft, unnecessary duplication of functionality, and dispensable features. This is undesirable for several reasons. * Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance cost frees development time for other improvements. * Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used. * A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size. The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better solved outside the standard library. This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the standard library, mostly for environments where installing a package from PyPI is not an option. This can be cooperate environments or class rooms where external code is not permitted without legal approval. * The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay. * The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead. * According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy sounds is powerful and highly motivating exercise for a 9yo aspiring developer. It's a fun battery to keep. Deprecation schedule ==================== 3.8 --- This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released a few months before Python 2.7 will reach its end of lifetime. We expect that Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and 2020. To reduce churn and to allow a smooth transition from Python 2, Python 3.8 will neither raise `DeprecationWarning` nor remove any modules that have been scheduled for removal. Instead deprecated modules will just be *documented* as deprecated. Optionally modules may emit a `PendingDeprecationWarning`. All deprecated modules will also undergo a feature freeze. No additional features should be added. Bug should still be fixed. 3.9 --- Starting with Python 3.9, deprecated modules will start issuing `DeprecationWarning`. 3.10 ---- In 3.10 all deprecated modules will be removed from the CPython repository together with tests, documentation, and autoconf rules. PEP acceptance process ====================== 3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process. The first *provisionally accepted* phase targets Python 3.8.0b1. In the first phase no code is changes or removed. Modules are only documented as deprecated. The final decision, which modules will be removed and how the removed code is preserved, can be delayed for another year. Deprecated modules ================== The modules are grouped as data encoding, multimedia, network, OS interface, and misc modules. The majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with audio processing. .. csv-table:: Table 1: Proposed modules deprecations :header: "Module", "Deprecated in", "To be removed", "Replacement" aifc,3.8,3.10,\- asynchat,3.8,3.10,asyncio asyncore,3.8,3.10,asyncio audioop,3.8,3.10,\- binhex,3.8,3.10,\- cgi,3.8,3.10,\- cgitb,3.8,3.10,\- chunk,3.8,3.10,\- colorsys,**3.8?**,**3.10?**,\- crypt,3.8,3.10,\- fileinput,3.8,3.10,argparse formatter,3.4,3.10,\- fpectl,**3.7**,**3.7**,\- getopt,**3.2**,**keep**,"argparse, optparse" imghdr,3.8,3.10,\- imp,**3.4**,3.10,importlib lib2to3,\-,**keep**, macpath,**3.7**,**3.8**,\- msilib,3.8,3.10,\- nntplib,3.8,3.10,\- nis,3.8,3.10,\- optparse,\-,**keep**,argparse ossaudiodev,3.8,3.10,\- pipes,3.8,3.10,subprocess smtpd,**3.7**,3.10,aiosmtpd sndhdr,3.8,3.10,\- spwd,3.8,3.10,\- sunau,3.8,3.10,\- uu,3.8,3.10,\- wave,\-,**keep**, xdrlib,3.8,3.10,\- Data encoding modules --------------------- binhex ~~~~~~ The `binhex `_ module encodes and decodes Apple Macintosh binhex4 data. It was originally developed for TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to encode binary email attachments. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** uu ~~ The `uu `_ module provides uuencode format, an old binary encoding format for email from 1980. The uu format has been replaced by MIME. The uu codec is provided by the binascii module. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** xdrlib ~~~~~~ The `xdrlib `_ module supports the Sun External Data Representation Standard. XDR is an old binary serialization format from 1987. These days it's rarely used outside specialized domains like NFS. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Multimedia modules ------------------ aifc ~~~~ The `aifc `_ module provides support for reading and writing AIFF and AIFF-C files. The Audio Interchange File Format is an old audio format from 1988 based on Amiga IFF. It was most commonly used on the Apple Macintosh. These days only few specialized application use AIFF. Module type pure Python (depends on `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute **none** audioop ~~~~~~~ The `audioop `_ module contains helper functions to manipulate raw audio data and adaptive differential pulse-code modulated audio data. The module is implemented in C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ module depend on `audioop`_ for some operations. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** colorsys ~~~~~~~~ The `colorsys `_ module defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate systems. The Pillow library provides much faster conversation between color systems. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `Pillow `_, `colorspacious `_ chunk ~~~~~ The `chunk `_ module provides support for reading and writing Electronic Arts' Interchange File Format. IFF is an old audio file format originally introduced for Commodore and Amiga. The format is no longer relevant. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** imghdr ~~~~~~ The `imghdr `_ module is a simple tool to guess the image file format from the first 32 bytes of a file or buffer. It supports only a limited amount of formats and neither returns resolution nor color depth. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute *n/a* ossaudiodev ~~~~~~~~~~~ The `ossaudiodev `_ module provides support for Open Sound System, an interface to sound playback and capture devices. OSS was initially free software, but later support for newer sound devices and improvements were proprietary. Linux community abandoned OSS in favor of ALSA [1]_. Some operation systems like OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** sndhdr ~~~~~~ The `sndhdr `_ module is similar to the `imghdr`_ module but for audio formats. It guesses file format, channels, frame rate, and sample widths from the first 512 bytes of a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a* sunau ~~~~~ The `sunau `_ module provides support for Sun AU sound format. It's yet another old, obsolete file format. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute **none** Networking modules ------------------ asynchat ~~~~~~~~ The `asynchat `_ module is build on top of `asyncore`_ and has been deprecated since Python 3.6. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio asyncore ~~~~~~~~ The `asyncore `_ module was the first module for asynchronous socket service clients and servers. It has been replaced by asyncio and is deprecated since Python 3.6. The ``asyncore`` module is also used in stdlib tests. The tests for ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly based on ``asyncore``. These tests must be updated to use asyncio or threading. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio cgi ~~~ The `cgi `_ module is a support module for Common Gateway Interface (CGI) scripts. CGI is deemed as inefficient because every incoming request is handled in a new process. PEP 206 considers the module as *designed poorly and are now near-impossible to fix*. Several people proposed to either keep the cgi module for features like `cgi.parse_qs()` or move `cgi.escape()` to a different module. The functions `cgi.parse_qs` and `cgi.parse_qsl` have been deprecated for a while and are actually aliases for `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The function `cgi.quote` has been deprecated in favor of `html.quote` with secure default values. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** cgitb ~~~~~ The `cgitb `_ module is a helper for the cgi module for configurable tracebacks. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** smtpd ~~~~~ The `smtpd `_ module provides a simple implementation of a SMTP mail server. The module documentation recommends ``aiosmtpd``. Module type pure Python Deprecated in **3.7** To be removed in 3.10 Substitute aiosmtpd nntplib ~~~~~~~ The `nntplib `_ module implements the client side of the Network News Transfer Protocol (nntp). News groups used to be a dominant platform for online discussions. Over the last two decades, news has been slowly but steadily replaced with mailing lists and web-based discussion platforms. The ``nntplib`` tests have been the cause of additional work in the recent past. Python only contains client side of NNTP. The test cases depend on external news server. These servers were unstable in the past. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Operating system interface -------------------------- crypt ~~~~~ The `crypt `_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. Module type C extension + Python module Deprecated in 3.8 To be removed in 3.10 Substitute `bcrypt `_, `passlib `_, `argon2cffi `_, hashlib module (PBKDF2, scrypt) macpath ~~~~~~~ The `macpath `_ module provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer supported Module type pure Python Deprecated in 3.7 Removed in 3.8 Substitute **none** nis ~~~ The `nis `_ module provides NIS/YP support. Network Information Service / Yellow Pages is an old and deprecated directory service protocol developed by Sun Microsystems. It's designed successor NIS+ from 1992 never took off. For a long time, libc's Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful and more secure replacement of NIS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** spwd ~~~~ The `spwd `_ module provides direct access to Unix shadow password database using non-standard APIs. In general it's a bad idea to use the spwd. The spwd circumvents system security policies, it does not use the PAM stack, and is only compatible with local user accounts. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** Misc modules ------------ fileinput ~~~~~~~~~ The `fileinput `_ module implements a helpers to iterate over a list of files from ``sys.argv``. The module predates the optparser and argparser module. The same functionality can be implemented with the argparser module. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute argparse formatter ~~~~~~~~~ The `formatter `_ module is an old text formatting module which has been deprecated since Python 3.4. Module type pure Python Deprecated in 3.4 To be removed in 3.10 Substitute *n/a* imp ~~~ The `imp `_ module is the predecessor of the `importlib `_ module. Most functions have been deprecated since Python 3.3 and the module since Python 3.4. Module type C extension Deprecated in 3.4 To be removed in 3.10 Substitute importlib msilib ~~~~~~ The `msilib `_ package is a Windows-only package. It supports the creation of Microsoft Installers (MSI). The package also exposes additional APIs to create cabinet files (CAB). The module is used to facilitate distutils to create MSI installers with ``bdist_msi`` command. In the past it was used to create CPython's official Windows installer, too. Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as new deployment model [3]_. Module type C extension + Python code Deprecated in 3.8 To be removed in 3.10 Substitute **none** pipes ~~~~~ The `pipes `_ module provides helpers to pipe the input of one command into the output of another command. The module is built on top of ``os.popen``. Users are encouraged to use the subprocess module instead. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute subprocess module Removed modules =============== fpectl ------ The `fpectl `_ module was never built by default, its usage was discouraged and considered dangerous. It also required a configure flag that caused an ABI incompatibility. The module was removed in 3.7 by Nathaniel J. Smith in `bpo-29137 `_. Module type C extension + CAPI Deprecated in 3.7 Removed in 3.7 Substitute **none** Modules to keep =============== Some modules were originally proposed for deprecation. lib2to3 ------- The `lib2to3 `_ package provides the ``2to3`` command to transpile Python 2 code to Python 3 code. The package is useful for other tasks besides porting code from Python 2 to 3. For example `black`_ uses it for code reformatting. Module type pure Python getopt ------ The `getopt `_ module mimics C's getopt() option parser. Although users are encouraged to use argparse instead, the getopt module is still widely used. Module type pure Python optparse -------- The `optparse `_ module is the predecessor of the argparse module. Although it has been deprecated for many years, it's still widely used. Module type pure Python Deprecated in 3.2 Substitute argparse wave ~~~~ The `wave `_ module provides support for the WAV sound format. The module uses one simple function from the `audioop`_ module to perform byte swapping between little and big endian formats. Before 24 bit WAV support was added, byte swap used to be implemented with the ``array`` module. To remove ``wave``'s dependency on the ``audioop``, the byte swap function could be either be moved to another module (e.g. ``operator``) or the ``array`` module could gain support for 24 bit (3 byte) arrays. Module type pure Python (depends on *byteswap* from `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a* Future maintenance of removed modules ===================================== The main goal of the PEP is to reduce the burden and workload on the Python core developer team. Therefore removed modules will not be maintained by the core team as separate PyPI packages. However the removed code, tests and documentation may be moved into a new git repository, so community members have a place from which they can pick up and fork code. A first draft of a `legacylib `_ repository is available on my private Github account. It's my hope that some of the deprecated modules will be picked up and adopted by users that actually care about them. For example ``colorsys`` and ``imghdr`` are useful modules, but have limited feature set. A fork of ``imghdr`` can add new features and support for more image formats, without being constrained by Python's release cycle. Most of the modules are in pure Python and can be easily packaged. Some depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ does not depend on any external libraries, it can be shipped in as binary wheels with some effort. Other C modules can be replaced with ctypes or cffi. For example I created `legacycrypt `_ with ``_crypt`` extension reimplemented with a few lines of ctypes code. Discussions =========== * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. * Berker Peksag proposed to deprecate and removed *msilib*. * Brett Cannon recommended to delay active deprecation warnings and removal of modules like *imp* until Python 3.10. Version 3.8 will be released shortly before Python 2 reaches end of lifetime. A delay reduced churn for users that migrate from Python 2 to 3.8. * Brett also came up with the idea to keep lib2to3. The package is useful for other purposes, e.g. `black `_ uses it to reformat Python code. * At one point, distutils was mentioned in the same sentence as this PEP. To avoid lengthy discussion and delay of the PEP, I decided against dealing with distutils. Deprecation of the distutils package will be handled by another PEP. * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) convinced me to keep the `wave`_ module. [4]_ * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ References ========== .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free .. [2] https://man.openbsd.org/ossaudio .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 .. [5] https://twitter.com/dabeaz/status/1130278844479545351 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From andrew.svetlov at gmail.com Mon May 20 17:12:34 2019 From: andrew.svetlov at gmail.com (Andrew Svetlov) Date: Tue, 21 May 2019 00:12:34 +0300 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: socketserver.py is also questionable On Mon, May 20, 2019 at 11:15 PM Christian Heimes wrote: > > Hi, > > here is the first version of my PEP 594 to deprecate and eventually remove modules from the standard library. The PEP started last year with talk during the Python Language Summit 2018, https://lwn.net/Articles/755229/. > > The PEP can be confirmed in two stages. I'm not planning any code changes for 3.8. Instead I only like to document a bunch of modules as deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The long deprecation phase gives us 3 years to change our minds or handle edge cases, too. > > Regards, > Christian > > > ---------------------------------------------------------------------------- > PEP: 594 > Title: Removing dead batteries from the standard library > Author: Christian Heimes > Status: Active > Type: Process > Content-Type: text/x-rst > Created: 20-May-2019 > Post-History: > > > Abstract > ======== > > This PEP proposed a list of standard library modules to be removed from the > standard library. The modules are mostly historic data formats and APIs that > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > Rationale > ========= > > Back in the early days of Python, the interpreter came with a large set of > useful modules. This was often refrained to as "batteries included" > philosophy and was one of the corner stones to Python's success story. > Users didn't have to figure out how to download and install separate > packages in order to write a simple web server or parse email. > > Times have changed. The introduction of the cheese shop (PyPI), setuptools, > and later pip, it became simple and straight forward to download and install > packages. Nowadays Python has a rich and vibrant ecosystem of third party > packages. It's pretty much standard to either install packages from PyPI or > use one of the many Python or Linux distributions. > > On the other hand, Python's standard library is piling up cruft, unnecessary > duplication of functionality, and dispensable features. This is undesirable > for several reasons. > > * Any additional module increases the maintenance cost for the Python core > development team. The team has limited resources, reduced maintenance cost > frees development time for other improvements. > * Modules in the standard library are generally favored and seen as the > de-facto solution for a problem. A majority of users only pick 3rd party > modules to replace a stdlib module, when they have a compelling reason, e.g. > lxml instead of `xml`. The removal of an unmaintained stdlib module > increases the chances of a community contributed module to become widely > used. > * A lean and mean standard library benefits platforms with limited resources > like devices with just a few hundred kilobyte of storage (e.g. BBC > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > (e.g. pyodide) also benefit from reduced download size. > > The modules in the PEP have been selected for deprecation because their > removal is either least controversial or most beneficial. For example > least controversial are 30 years old multimedia formats like ``sunau`` > audio format, which was used on SPARC and NeXT workstations in the late > 1980ties. The ``crypt`` module has fundamental flaws that are better solved > outside the standard library. > > This PEP also designates some modules as not scheduled for removal. Some > modules have been deprecated for several releases or seem unnecessary at > first glance. However it is beneficial to keep the modules in the standard > library, mostly for environments where installing a package from PyPI is not > an option. This can be cooperate environments or class rooms where external > code is not permitted without legal approval. > > * The usage of FTP is declining, but some files are still provided over > the FTP protocol or hosters offer FTP to upload content. Therefore > ``ftplib`` is going to stay. > * The ``optparse`` and ``getopt`` module are widely used. They are mature > modules with very low maintenance overhead. > * According to David Beazley [5]_ the ``wave`` module is easy to teach to > kids and can make crazy sounds. Making a computer generate crazy sounds is > powerful and highly motivating exercise for a 9yo aspiring developer. It's > a fun battery to keep. > > > Deprecation schedule > ==================== > > 3.8 > --- > > This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released > a few months before Python 2.7 will reach its end of lifetime. We expect that > Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and > 2020. To reduce churn and to allow a smooth transition from Python 2, > Python 3.8 will neither raise `DeprecationWarning` nor remove any > modules that have been scheduled for removal. Instead deprecated modules will > just be *documented* as deprecated. Optionally modules may emit a > `PendingDeprecationWarning`. > > All deprecated modules will also undergo a feature freeze. No additional > features should be added. Bug should still be fixed. > > 3.9 > --- > > Starting with Python 3.9, deprecated modules will start issuing > `DeprecationWarning`. > > > 3.10 > ---- > > In 3.10 all deprecated modules will be removed from the CPython repository > together with tests, documentation, and autoconf rules. > > > PEP acceptance process > ====================== > > 3.8.0b1 is scheduled to be release shortly after the PEP is officially > submitted. Since it's improbable that the PEP will pass all stages of the > PEP process in time, I propose a two step acceptance process that is > analogous Python's two release deprecation process. > > The first *provisionally accepted* phase targets Python 3.8.0b1. In the first > phase no code is changes or removed. Modules are only documented as > deprecated. > > The final decision, which modules will be removed and how the removed code > is preserved, can be delayed for another year. > > > Deprecated modules > ================== > > The modules are grouped as data encoding, multimedia, network, OS interface, > and misc modules. The majority of modules are for old data formats or > old APIs. Some others are rarely useful and have better replacements on > PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with > audio processing. > > .. csv-table:: Table 1: Proposed modules deprecations > :header: "Module", "Deprecated in", "To be removed", "Replacement" > > aifc,3.8,3.10,\- > asynchat,3.8,3.10,asyncio > asyncore,3.8,3.10,asyncio > audioop,3.8,3.10,\- > binhex,3.8,3.10,\- > cgi,3.8,3.10,\- > cgitb,3.8,3.10,\- > chunk,3.8,3.10,\- > colorsys,**3.8?**,**3.10?**,\- > crypt,3.8,3.10,\- > fileinput,3.8,3.10,argparse > formatter,3.4,3.10,\- > fpectl,**3.7**,**3.7**,\- > getopt,**3.2**,**keep**,"argparse, optparse" > imghdr,3.8,3.10,\- > imp,**3.4**,3.10,importlib > lib2to3,\-,**keep**, > macpath,**3.7**,**3.8**,\- > msilib,3.8,3.10,\- > nntplib,3.8,3.10,\- > nis,3.8,3.10,\- > optparse,\-,**keep**,argparse > ossaudiodev,3.8,3.10,\- > pipes,3.8,3.10,subprocess > smtpd,**3.7**,3.10,aiosmtpd > sndhdr,3.8,3.10,\- > spwd,3.8,3.10,\- > sunau,3.8,3.10,\- > uu,3.8,3.10,\- > wave,\-,**keep**, > xdrlib,3.8,3.10,\- > > > Data encoding modules > --------------------- > > binhex > ~~~~~~ > > The `binhex `_ module encodes > and decodes Apple Macintosh binhex4 data. It was originally developed for > TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to > encode binary email attachments. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > uu > ~~ > > The `uu `_ module provides > uuencode format, an old binary encoding format for email from 1980. The uu > format has been replaced by MIME. The uu codec is provided by the binascii > module. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > xdrlib > ~~~~~~ > > The `xdrlib `_ module supports > the Sun External Data Representation Standard. XDR is an old binary > serialization format from 1987. These days it's rarely used outside > specialized domains like NFS. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Multimedia modules > ------------------ > > aifc > ~~~~ > > The `aifc `_ module provides > support for reading and writing AIFF and AIFF-C files. The Audio Interchange > File Format is an old audio format from 1988 based on Amiga IFF. It was most > commonly used on the Apple Macintosh. These days only few specialized > application use AIFF. > > Module type > pure Python (depends on `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > audioop > ~~~~~~~ > > The `audioop `_ module > contains helper functions to manipulate raw audio data and adaptive > differential pulse-code modulated audio data. The module is implemented in > C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ > module depend on `audioop`_ for some operations. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > colorsys > ~~~~~~~~ > > The `colorsys `_ module > defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate > systems. The Pillow library provides much faster conversation between > color systems. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `Pillow `_, > `colorspacious `_ > > chunk > ~~~~~ > > The `chunk `_ module provides > support for reading and writing Electronic Arts' Interchange File Format. > IFF is an old audio file format originally introduced for Commodore and > Amiga. The format is no longer relevant. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > imghdr > ~~~~~~ > > The `imghdr `_ module is a > simple tool to guess the image file format from the first 32 bytes > of a file or buffer. It supports only a limited amount of formats and > neither returns resolution nor color depth. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > ossaudiodev > ~~~~~~~~~~~ > > The `ossaudiodev `_ > module provides support for Open Sound System, an interface to sound > playback and capture devices. OSS was initially free software, but later > support for newer sound devices and improvements were proprietary. Linux > community abandoned OSS in favor of ALSA [1]_. Some operation systems like > OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > sndhdr > ~~~~~~ > > The `sndhdr `_ module is > similar to the `imghdr`_ module but for audio formats. It guesses file > format, channels, frame rate, and sample widths from the first 512 bytes of > a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and > other ancient formats. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > sunau > ~~~~~ > > The `sunau `_ module provides > support for Sun AU sound format. It's yet another old, obsolete file format. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Networking modules > ------------------ > > asynchat > ~~~~~~~~ > > The `asynchat `_ module > is build on top of `asyncore`_ and has been deprecated since Python 3.6. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > asyncore > ~~~~~~~~ > > The `asyncore `_ module was > the first module for asynchronous socket service clients and servers. It > has been replaced by asyncio and is deprecated since Python 3.6. > > The ``asyncore`` module is also used in stdlib tests. The tests for > ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly > based on ``asyncore``. These tests must be updated to use asyncio or > threading. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > > cgi > ~~~ > > The `cgi `_ module is a support > module for Common Gateway Interface (CGI) scripts. CGI is deemed as > inefficient because every incoming request is handled in a new process. PEP > 206 considers the module as *designed poorly and are now near-impossible > to fix*. > > Several people proposed to either keep the cgi module for features like > `cgi.parse_qs()` or move `cgi.escape()` to a different module. The > functions `cgi.parse_qs` and `cgi.parse_qsl` have been > deprecated for a while and are actually aliases for > `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The > function `cgi.quote` has been deprecated in favor of `html.quote` > with secure default values. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > cgitb > ~~~~~ > > The `cgitb `_ module is a > helper for the cgi module for configurable tracebacks. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > smtpd > ~~~~~ > > The `smtpd `_ module provides > a simple implementation of a SMTP mail server. The module documentation > recommends ``aiosmtpd``. > > Module type > pure Python > Deprecated in > **3.7** > To be removed in > 3.10 > Substitute > aiosmtpd > > nntplib > ~~~~~~~ > > The `nntplib `_ module > implements the client side of the Network News Transfer Protocol (nntp). News > groups used to be a dominant platform for online discussions. Over the last > two decades, news has been slowly but steadily replaced with mailing lists > and web-based discussion platforms. > > The ``nntplib`` tests have been the cause of additional work in the recent > past. Python only contains client side of NNTP. The test cases depend on > external news server. These servers were unstable in the past. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Operating system interface > -------------------------- > > crypt > ~~~~~ > > The `crypt `_ module implements > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor > quality and insecure. Users are discouraged to use them. > > * The module is not available on Windows. Cross-platform application need > an alternative implementation any way. > * Only DES encryption is guarenteed to be available. DES has an extremely > limited key space of 2**56. > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > algorithm that is still secure. However it's in glibc and therefore not > commonly available on Linux. > * Depending on the platform, the ``crypt`` module is not thread safe. Only > implementations with ``crypt_r(3)`` are thread safe. > > Module type > C extension + Python module > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `bcrypt `_, > `passlib `_, > `argon2cffi `_, > hashlib module (PBKDF2, scrypt) > > macpath > ~~~~~~~ > > The `macpath `_ module > provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer > supported > > Module type > pure Python > Deprecated in > 3.7 > Removed in > 3.8 > Substitute > **none** > > nis > ~~~ > > The `nis `_ module provides > NIS/YP support. Network Information Service / Yellow Pages is an old and > deprecated directory service protocol developed by Sun Microsystems. It's > designed successor NIS+ from 1992 never took off. For a long time, libc's > Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful > and more secure replacement of NIS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > spwd > ~~~~ > > The `spwd `_ module provides > direct access to Unix shadow password database using non-standard APIs. > In general it's a bad idea to use the spwd. The spwd circumvents system > security policies, it does not use the PAM stack, and is > only compatible with local user accounts. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > Misc modules > ------------ > > fileinput > ~~~~~~~~~ > > The `fileinput `_ module > implements a helpers to iterate over a list of files from ``sys.argv``. The > module predates the optparser and argparser module. The same functionality > can be implemented with the argparser module. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > argparse > > formatter > ~~~~~~~~~ > > The `formatter `_ module > is an old text formatting module which has been deprecated since Python 3.4. > > Module type > pure Python > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > *n/a* > > imp > ~~~ > > The `imp `_ module is the > predecessor of the > `importlib `_ module. Most > functions have been deprecated since Python 3.3 and the module since > Python 3.4. > > Module type > C extension > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > importlib > > msilib > ~~~~~~ > > The `msilib `_ package is a > Windows-only package. It supports the creation of Microsoft Installers (MSI). > The package also exposes additional APIs to create cabinet files (CAB). The > module is used to facilitate distutils to create MSI installers with > ``bdist_msi`` command. In the past it was used to create CPython's official > Windows installer, too. > > Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) > as new deployment model [3]_. > > Module type > C extension + Python code > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > pipes > ~~~~~ > > The `pipes `_ module provides > helpers to pipe the input of one command into the output of another command. > The module is built on top of ``os.popen``. Users are encouraged to use > the subprocess module instead. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > subprocess module > > Removed modules > =============== > > fpectl > ------ > > The `fpectl `_ module was > never built by default, its usage was discouraged and considered dangerous. > It also required a configure flag that caused an ABI incompatibility. The > module was removed in 3.7 by Nathaniel J. Smith in > `bpo-29137 `_. > > Module type > C extension + CAPI > Deprecated in > 3.7 > Removed in > 3.7 > Substitute > **none** > > > Modules to keep > =============== > > Some modules were originally proposed for deprecation. > > lib2to3 > ------- > > The `lib2to3 `_ package provides > the ``2to3`` command to transpile Python 2 code to Python 3 code. > > The package is useful for other tasks besides porting code from Python 2 to > 3. For example `black`_ uses it for code reformatting. > > Module type > pure Python > > getopt > ------ > > The `getopt `_ module mimics > C's getopt() option parser. Although users are encouraged to use argparse > instead, the getopt module is still widely used. > > Module type > pure Python > > optparse > -------- > > The `optparse `_ module is > the predecessor of the argparse module. Although it has been deprecated for > many years, it's still widely used. > > Module type > pure Python > Deprecated in > 3.2 > Substitute > argparse > > wave > ~~~~ > > The `wave `_ module provides > support for the WAV sound format. The module uses one simple function > from the `audioop`_ module to perform byte swapping between little and big > endian formats. Before 24 bit WAV support was added, byte swap used to be > implemented with the ``array`` module. To remove ``wave``'s dependency on the > ``audioop``, the byte swap function could be either be moved to another > module (e.g. ``operator``) or the ``array`` module could gain support for > 24 bit (3 byte) arrays. > > Module type > pure Python (depends on *byteswap* from `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > > Future maintenance of removed modules > ===================================== > > The main goal of the PEP is to reduce the burden and workload on the Python > core developer team. Therefore removed modules will not be maintained by > the core team as separate PyPI packages. However the removed code, tests and > documentation may be moved into a new git repository, so community members > have a place from which they can pick up and fork code. > > A first draft of a `legacylib `_ > repository is available on my private Github account. > > It's my hope that some of the deprecated modules will be picked up and > adopted by users that actually care about them. For example ``colorsys`` and > ``imghdr`` are useful modules, but have limited feature set. A fork of > ``imghdr`` can add new features and support for more image formats, without > being constrained by Python's release cycle. > > Most of the modules are in pure Python and can be easily packaged. Some > depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ > does not depend on any external libraries, it can be shipped in as binary > wheels with some effort. Other C modules can be replaced with ctypes or cffi. > For example I created `legacycrypt `_ > with ``_crypt`` extension reimplemented with a few lines of ctypes code. > > > Discussions > =========== > > * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. > * Berker Peksag proposed to deprecate and removed *msilib*. > * Brett Cannon recommended to delay active deprecation warnings and removal > of modules like *imp* until Python 3.10. Version 3.8 will be released > shortly before Python 2 reaches end of lifetime. A delay reduced churn for > users that migrate from Python 2 to 3.8. > * Brett also came up with the idea to keep lib2to3. The package is useful > for other purposes, e.g. `black `_ uses > it to reformat Python code. > * At one point, distutils was mentioned in the same sentence as this PEP. > To avoid lengthy discussion and delay of the PEP, I decided against dealing > with distutils. Deprecation of the distutils package will be handled by > another PEP. > * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) > convinced me to keep the `wave`_ module. [4]_ > * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ > > > References > ========== > > .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free > .. [2] https://man.openbsd.org/ossaudio > .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ > .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 > .. [5] https://twitter.com/dabeaz/status/1130278844479545351 > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/andrew.svetlov%40gmail.com -- Thanks, Andrew Svetlov From christian at python.org Mon May 20 17:20:58 2019 From: christian at python.org (Christian Heimes) Date: Mon, 20 May 2019 23:20:58 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> On 20/05/2019 23.12, Andrew Svetlov wrote: > socketserver.py is also questionable I briefly though about the module, but didn't consider it for removal. The http.server, xmlrpc.server, and logging configuration server are implemented on top of the socketserver. I don't want to remove the socketserver module without a suitable replacement for http.server in the standard library. Christian From solipsis at pitrou.net Mon May 20 17:27:49 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 20 May 2019 23:27:49 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library References: Message-ID: <20190520232749.442c45c7@fsol> NNTP is still quite used (often through GMane, but probably not only) so I'd question the removal of nntplib. cgitb used to be used by some Web frameworks in order to format exceptions. Perhaps one should check if that's still the case. If the wave module depends on the audioop module, and if the wave module is kept in the stdlib, then the audioop module can't be removed. Removing the crypt module would remove support for system-standard password files. I don't understand the rationale. Regards Antoine. On Mon, 20 May 2019 22:15:22 +0200 Christian Heimes wrote: > Hi, > > here is the first version of my PEP 594 to deprecate and eventually remove modules from the standard library. The PEP started last year with talk during the Python Language Summit 2018, https://lwn.net/Articles/755229/. > > The PEP can be confirmed in two stages. I'm not planning any code changes for 3.8. Instead I only like to document a bunch of modules as deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The long deprecation phase gives us 3 years to change our minds or handle edge cases, too. > > Regards, > Christian > > > ---------------------------------------------------------------------------- > PEP: 594 > Title: Removing dead batteries from the standard library > Author: Christian Heimes > Status: Active > Type: Process > Content-Type: text/x-rst > Created: 20-May-2019 > Post-History: > > > Abstract > ======== > > This PEP proposed a list of standard library modules to be removed from the > standard library. The modules are mostly historic data formats and APIs that > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > Rationale > ========= > > Back in the early days of Python, the interpreter came with a large set of > useful modules. This was often refrained to as "batteries included" > philosophy and was one of the corner stones to Python's success story. > Users didn't have to figure out how to download and install separate > packages in order to write a simple web server or parse email. > > Times have changed. The introduction of the cheese shop (PyPI), setuptools, > and later pip, it became simple and straight forward to download and install > packages. Nowadays Python has a rich and vibrant ecosystem of third party > packages. It's pretty much standard to either install packages from PyPI or > use one of the many Python or Linux distributions. > > On the other hand, Python's standard library is piling up cruft, unnecessary > duplication of functionality, and dispensable features. This is undesirable > for several reasons. > > * Any additional module increases the maintenance cost for the Python core > development team. The team has limited resources, reduced maintenance cost > frees development time for other improvements. > * Modules in the standard library are generally favored and seen as the > de-facto solution for a problem. A majority of users only pick 3rd party > modules to replace a stdlib module, when they have a compelling reason, e.g. > lxml instead of `xml`. The removal of an unmaintained stdlib module > increases the chances of a community contributed module to become widely > used. > * A lean and mean standard library benefits platforms with limited resources > like devices with just a few hundred kilobyte of storage (e.g. BBC > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > (e.g. pyodide) also benefit from reduced download size. > > The modules in the PEP have been selected for deprecation because their > removal is either least controversial or most beneficial. For example > least controversial are 30 years old multimedia formats like ``sunau`` > audio format, which was used on SPARC and NeXT workstations in the late > 1980ties. The ``crypt`` module has fundamental flaws that are better solved > outside the standard library. > > This PEP also designates some modules as not scheduled for removal. Some > modules have been deprecated for several releases or seem unnecessary at > first glance. However it is beneficial to keep the modules in the standard > library, mostly for environments where installing a package from PyPI is not > an option. This can be cooperate environments or class rooms where external > code is not permitted without legal approval. > > * The usage of FTP is declining, but some files are still provided over > the FTP protocol or hosters offer FTP to upload content. Therefore > ``ftplib`` is going to stay. > * The ``optparse`` and ``getopt`` module are widely used. They are mature > modules with very low maintenance overhead. > * According to David Beazley [5]_ the ``wave`` module is easy to teach to > kids and can make crazy sounds. Making a computer generate crazy sounds is > powerful and highly motivating exercise for a 9yo aspiring developer. It's > a fun battery to keep. > > > Deprecation schedule > ==================== > > 3.8 > --- > > This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released > a few months before Python 2.7 will reach its end of lifetime. We expect that > Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and > 2020. To reduce churn and to allow a smooth transition from Python 2, > Python 3.8 will neither raise `DeprecationWarning` nor remove any > modules that have been scheduled for removal. Instead deprecated modules will > just be *documented* as deprecated. Optionally modules may emit a > `PendingDeprecationWarning`. > > All deprecated modules will also undergo a feature freeze. No additional > features should be added. Bug should still be fixed. > > 3.9 > --- > > Starting with Python 3.9, deprecated modules will start issuing > `DeprecationWarning`. > > > 3.10 > ---- > > In 3.10 all deprecated modules will be removed from the CPython repository > together with tests, documentation, and autoconf rules. > > > PEP acceptance process > ====================== > > 3.8.0b1 is scheduled to be release shortly after the PEP is officially > submitted. Since it's improbable that the PEP will pass all stages of the > PEP process in time, I propose a two step acceptance process that is > analogous Python's two release deprecation process. > > The first *provisionally accepted* phase targets Python 3.8.0b1. In the first > phase no code is changes or removed. Modules are only documented as > deprecated. > > The final decision, which modules will be removed and how the removed code > is preserved, can be delayed for another year. > > > Deprecated modules > ================== > > The modules are grouped as data encoding, multimedia, network, OS interface, > and misc modules. The majority of modules are for old data formats or > old APIs. Some others are rarely useful and have better replacements on > PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with > audio processing. > > .. csv-table:: Table 1: Proposed modules deprecations > :header: "Module", "Deprecated in", "To be removed", "Replacement" > > aifc,3.8,3.10,\- > asynchat,3.8,3.10,asyncio > asyncore,3.8,3.10,asyncio > audioop,3.8,3.10,\- > binhex,3.8,3.10,\- > cgi,3.8,3.10,\- > cgitb,3.8,3.10,\- > chunk,3.8,3.10,\- > colorsys,**3.8?**,**3.10?**,\- > crypt,3.8,3.10,\- > fileinput,3.8,3.10,argparse > formatter,3.4,3.10,\- > fpectl,**3.7**,**3.7**,\- > getopt,**3.2**,**keep**,"argparse, optparse" > imghdr,3.8,3.10,\- > imp,**3.4**,3.10,importlib > lib2to3,\-,**keep**, > macpath,**3.7**,**3.8**,\- > msilib,3.8,3.10,\- > nntplib,3.8,3.10,\- > nis,3.8,3.10,\- > optparse,\-,**keep**,argparse > ossaudiodev,3.8,3.10,\- > pipes,3.8,3.10,subprocess > smtpd,**3.7**,3.10,aiosmtpd > sndhdr,3.8,3.10,\- > spwd,3.8,3.10,\- > sunau,3.8,3.10,\- > uu,3.8,3.10,\- > wave,\-,**keep**, > xdrlib,3.8,3.10,\- > > > Data encoding modules > --------------------- > > binhex > ~~~~~~ > > The `binhex `_ module encodes > and decodes Apple Macintosh binhex4 data. It was originally developed for > TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to > encode binary email attachments. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > uu > ~~ > > The `uu `_ module provides > uuencode format, an old binary encoding format for email from 1980. The uu > format has been replaced by MIME. The uu codec is provided by the binascii > module. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > xdrlib > ~~~~~~ > > The `xdrlib `_ module supports > the Sun External Data Representation Standard. XDR is an old binary > serialization format from 1987. These days it's rarely used outside > specialized domains like NFS. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Multimedia modules > ------------------ > > aifc > ~~~~ > > The `aifc `_ module provides > support for reading and writing AIFF and AIFF-C files. The Audio Interchange > File Format is an old audio format from 1988 based on Amiga IFF. It was most > commonly used on the Apple Macintosh. These days only few specialized > application use AIFF. > > Module type > pure Python (depends on `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > audioop > ~~~~~~~ > > The `audioop `_ module > contains helper functions to manipulate raw audio data and adaptive > differential pulse-code modulated audio data. The module is implemented in > C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ > module depend on `audioop`_ for some operations. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > colorsys > ~~~~~~~~ > > The `colorsys `_ module > defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate > systems. The Pillow library provides much faster conversation between > color systems. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `Pillow `_, > `colorspacious `_ > > chunk > ~~~~~ > > The `chunk `_ module provides > support for reading and writing Electronic Arts' Interchange File Format. > IFF is an old audio file format originally introduced for Commodore and > Amiga. The format is no longer relevant. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > imghdr > ~~~~~~ > > The `imghdr `_ module is a > simple tool to guess the image file format from the first 32 bytes > of a file or buffer. It supports only a limited amount of formats and > neither returns resolution nor color depth. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > ossaudiodev > ~~~~~~~~~~~ > > The `ossaudiodev `_ > module provides support for Open Sound System, an interface to sound > playback and capture devices. OSS was initially free software, but later > support for newer sound devices and improvements were proprietary. Linux > community abandoned OSS in favor of ALSA [1]_. Some operation systems like > OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > sndhdr > ~~~~~~ > > The `sndhdr `_ module is > similar to the `imghdr`_ module but for audio formats. It guesses file > format, channels, frame rate, and sample widths from the first 512 bytes of > a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and > other ancient formats. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > sunau > ~~~~~ > > The `sunau `_ module provides > support for Sun AU sound format. It's yet another old, obsolete file format. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Networking modules > ------------------ > > asynchat > ~~~~~~~~ > > The `asynchat `_ module > is build on top of `asyncore`_ and has been deprecated since Python 3.6. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > asyncore > ~~~~~~~~ > > The `asyncore `_ module was > the first module for asynchronous socket service clients and servers. It > has been replaced by asyncio and is deprecated since Python 3.6. > > The ``asyncore`` module is also used in stdlib tests. The tests for > ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly > based on ``asyncore``. These tests must be updated to use asyncio or > threading. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > > cgi > ~~~ > > The `cgi `_ module is a support > module for Common Gateway Interface (CGI) scripts. CGI is deemed as > inefficient because every incoming request is handled in a new process. PEP > 206 considers the module as *designed poorly and are now near-impossible > to fix*. > > Several people proposed to either keep the cgi module for features like > `cgi.parse_qs()` or move `cgi.escape()` to a different module. The > functions `cgi.parse_qs` and `cgi.parse_qsl` have been > deprecated for a while and are actually aliases for > `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The > function `cgi.quote` has been deprecated in favor of `html.quote` > with secure default values. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > cgitb > ~~~~~ > > The `cgitb `_ module is a > helper for the cgi module for configurable tracebacks. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > smtpd > ~~~~~ > > The `smtpd `_ module provides > a simple implementation of a SMTP mail server. The module documentation > recommends ``aiosmtpd``. > > Module type > pure Python > Deprecated in > **3.7** > To be removed in > 3.10 > Substitute > aiosmtpd > > nntplib > ~~~~~~~ > > The `nntplib `_ module > implements the client side of the Network News Transfer Protocol (nntp). News > groups used to be a dominant platform for online discussions. Over the last > two decades, news has been slowly but steadily replaced with mailing lists > and web-based discussion platforms. > > The ``nntplib`` tests have been the cause of additional work in the recent > past. Python only contains client side of NNTP. The test cases depend on > external news server. These servers were unstable in the past. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Operating system interface > -------------------------- > > crypt > ~~~~~ > > The `crypt `_ module implements > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor > quality and insecure. Users are discouraged to use them. > > * The module is not available on Windows. Cross-platform application need > an alternative implementation any way. > * Only DES encryption is guarenteed to be available. DES has an extremely > limited key space of 2**56. > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > algorithm that is still secure. However it's in glibc and therefore not > commonly available on Linux. > * Depending on the platform, the ``crypt`` module is not thread safe. Only > implementations with ``crypt_r(3)`` are thread safe. > > Module type > C extension + Python module > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `bcrypt `_, > `passlib `_, > `argon2cffi `_, > hashlib module (PBKDF2, scrypt) > > macpath > ~~~~~~~ > > The `macpath `_ module > provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer > supported > > Module type > pure Python > Deprecated in > 3.7 > Removed in > 3.8 > Substitute > **none** > > nis > ~~~ > > The `nis `_ module provides > NIS/YP support. Network Information Service / Yellow Pages is an old and > deprecated directory service protocol developed by Sun Microsystems. It's > designed successor NIS+ from 1992 never took off. For a long time, libc's > Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful > and more secure replacement of NIS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > spwd > ~~~~ > > The `spwd `_ module provides > direct access to Unix shadow password database using non-standard APIs. > In general it's a bad idea to use the spwd. The spwd circumvents system > security policies, it does not use the PAM stack, and is > only compatible with local user accounts. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > Misc modules > ------------ > > fileinput > ~~~~~~~~~ > > The `fileinput `_ module > implements a helpers to iterate over a list of files from ``sys.argv``. The > module predates the optparser and argparser module. The same functionality > can be implemented with the argparser module. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > argparse > > formatter > ~~~~~~~~~ > > The `formatter `_ module > is an old text formatting module which has been deprecated since Python 3.4. > > Module type > pure Python > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > *n/a* > > imp > ~~~ > > The `imp `_ module is the > predecessor of the > `importlib `_ module. Most > functions have been deprecated since Python 3.3 and the module since > Python 3.4. > > Module type > C extension > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > importlib > > msilib > ~~~~~~ > > The `msilib `_ package is a > Windows-only package. It supports the creation of Microsoft Installers (MSI). > The package also exposes additional APIs to create cabinet files (CAB). The > module is used to facilitate distutils to create MSI installers with > ``bdist_msi`` command. In the past it was used to create CPython's official > Windows installer, too. > > Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) > as new deployment model [3]_. > > Module type > C extension + Python code > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > pipes > ~~~~~ > > The `pipes `_ module provides > helpers to pipe the input of one command into the output of another command. > The module is built on top of ``os.popen``. Users are encouraged to use > the subprocess module instead. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > subprocess module > > Removed modules > =============== > > fpectl > ------ > > The `fpectl `_ module was > never built by default, its usage was discouraged and considered dangerous. > It also required a configure flag that caused an ABI incompatibility. The > module was removed in 3.7 by Nathaniel J. Smith in > `bpo-29137 `_. > > Module type > C extension + CAPI > Deprecated in > 3.7 > Removed in > 3.7 > Substitute > **none** > > > Modules to keep > =============== > > Some modules were originally proposed for deprecation. > > lib2to3 > ------- > > The `lib2to3 `_ package provides > the ``2to3`` command to transpile Python 2 code to Python 3 code. > > The package is useful for other tasks besides porting code from Python 2 to > 3. For example `black`_ uses it for code reformatting. > > Module type > pure Python > > getopt > ------ > > The `getopt `_ module mimics > C's getopt() option parser. Although users are encouraged to use argparse > instead, the getopt module is still widely used. > > Module type > pure Python > > optparse > -------- > > The `optparse `_ module is > the predecessor of the argparse module. Although it has been deprecated for > many years, it's still widely used. > > Module type > pure Python > Deprecated in > 3.2 > Substitute > argparse > > wave > ~~~~ > > The `wave `_ module provides > support for the WAV sound format. The module uses one simple function > from the `audioop`_ module to perform byte swapping between little and big > endian formats. Before 24 bit WAV support was added, byte swap used to be > implemented with the ``array`` module. To remove ``wave``'s dependency on the > ``audioop``, the byte swap function could be either be moved to another > module (e.g. ``operator``) or the ``array`` module could gain support for > 24 bit (3 byte) arrays. > > Module type > pure Python (depends on *byteswap* from `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > > Future maintenance of removed modules > ===================================== > > The main goal of the PEP is to reduce the burden and workload on the Python > core developer team. Therefore removed modules will not be maintained by > the core team as separate PyPI packages. However the removed code, tests and > documentation may be moved into a new git repository, so community members > have a place from which they can pick up and fork code. > > A first draft of a `legacylib `_ > repository is available on my private Github account. > > It's my hope that some of the deprecated modules will be picked up and > adopted by users that actually care about them. For example ``colorsys`` and > ``imghdr`` are useful modules, but have limited feature set. A fork of > ``imghdr`` can add new features and support for more image formats, without > being constrained by Python's release cycle. > > Most of the modules are in pure Python and can be easily packaged. Some > depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ > does not depend on any external libraries, it can be shipped in as binary > wheels with some effort. Other C modules can be replaced with ctypes or cffi. > For example I created `legacycrypt `_ > with ``_crypt`` extension reimplemented with a few lines of ctypes code. > > > Discussions > =========== > > * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. > * Berker Peksag proposed to deprecate and removed *msilib*. > * Brett Cannon recommended to delay active deprecation warnings and removal > of modules like *imp* until Python 3.10. Version 3.8 will be released > shortly before Python 2 reaches end of lifetime. A delay reduced churn for > users that migrate from Python 2 to 3.8. > * Brett also came up with the idea to keep lib2to3. The package is useful > for other purposes, e.g. `black `_ uses > it to reformat Python code. > * At one point, distutils was mentioned in the same sentence as this PEP. > To avoid lengthy discussion and delay of the PEP, I decided against dealing > with distutils. Deprecation of the distutils package will be handled by > another PEP. > * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) > convinced me to keep the `wave`_ module. [4]_ > * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ > > > References > ========== > > .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free > .. [2] https://man.openbsd.org/ossaudio > .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ > .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 > .. [5] https://twitter.com/dabeaz/status/1130278844479545351 > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > From christian at python.org Mon May 20 18:06:35 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 00:06:35 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190520232749.442c45c7@fsol> References: <20190520232749.442c45c7@fsol> Message-ID: <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> On 20/05/2019 23.27, Antoine Pitrou wrote: > NNTP is still quite used (often through GMane, but probably not only) so > I'd question the removal of nntplib. Is NNTP support important enough to keep the module in the standard library? > cgitb used to be used by some Web frameworks in order to format > exceptions. Perhaps one should check if that's still the case. A search on github did not reveal any relevant use of cgitb besides tons of copies of test_cgitb and an optional debugging middleware in Paste. I checked Django, Plone, CherryPy, flask, and bottle. None uses cgitb. > If the wave module depends on the audioop module, and if the wave > module is kept in the stdlib, then the audioop module can't be removed. No, it can be removed. I explained the situation in the "wave" section of the PEP. > Removing the crypt module would remove support for system-standard > password files. I don't understand the rationale. Applications *must* not access system-standard password files directly. On any sanely and securely configured systems, application cannot even access system password files like /etc/shadow. Access restrictions and system security policies will prevent read access. Also applications cannot assume that users are present in any user file. They may come from LDAP, SSSD, ActiveDirectory, or other sources. The correct way to interact with system users is to use the proper APIs, that are NSS (name service switch) and PAM (pluggable authentication modules). NSS looks up and enumerate users and groups. PAM performs password validation and much, much, much more. The pwd and grp modules use the correct APIs to interact with NSS. If you need to check or change passwords, you must go through PAM. Christian From jeff at kintscher.cc Mon May 20 18:23:05 2019 From: jeff at kintscher.cc (Jeff Kintscher) Date: Mon, 20 May 2019 15:23:05 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190520232749.442c45c7@fsol> References: <20190520232749.442c45c7@fsol> Message-ID: <1361d2de-fef8-d4da-a8b1-46cae323f48b@kintscher.cc> What replacements are available for NNTP? All I could find was pynntp, which had a single release 6 years ago. https://github.com/greenbender/pynntp //Jeff On 5/20/19 2:27 PM, Antoine Pitrou wrote: > NNTP is still quite used (often through GMane, but probably not only) so > I'd question the removal of nntplib. > > cgitb used to be used by some Web frameworks in order to format > exceptions. Perhaps one should check if that's still the case. > > If the wave module depends on the audioop module, and if the wave > module is kept in the stdlib, then the audioop module can't be removed. > > Removing the crypt module would remove support for system-standard > password files. I don't understand the rationale. > > Regards > > Antoine. > > > On Mon, 20 May 2019 22:15:22 +0200 > Christian Heimes wrote: >> Hi, >> >> here is the first version of my PEP 594 to deprecate and eventually remove modules from the standard library. The PEP started last year with talk during the Python Language Summit 2018, https://lwn.net/Articles/755229/. >> >> The PEP can be confirmed in two stages. I'm not planning any code changes for 3.8. Instead I only like to document a bunch of modules as deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The long deprecation phase gives us 3 years to change our minds or handle edge cases, too. >> >> Regards, >> Christian >> >> >> ---------------------------------------------------------------------------- >> PEP: 594 >> Title: Removing dead batteries from the standard library >> Author: Christian Heimes >> Status: Active >> Type: Process >> Content-Type: text/x-rst >> Created: 20-May-2019 >> Post-History: >> >> >> Abstract >> ======== >> >> This PEP proposed a list of standard library modules to be removed from the >> standard library. The modules are mostly historic data formats and APIs that >> have been superseded a long time ago, e.g. Mac OS 9 and Commodore. >> >> >> Rationale >> ========= >> >> Back in the early days of Python, the interpreter came with a large set of >> useful modules. This was often refrained to as "batteries included" >> philosophy and was one of the corner stones to Python's success story. >> Users didn't have to figure out how to download and install separate >> packages in order to write a simple web server or parse email. >> >> Times have changed. The introduction of the cheese shop (PyPI), setuptools, >> and later pip, it became simple and straight forward to download and install >> packages. Nowadays Python has a rich and vibrant ecosystem of third party >> packages. It's pretty much standard to either install packages from PyPI or >> use one of the many Python or Linux distributions. >> >> On the other hand, Python's standard library is piling up cruft, unnecessary >> duplication of functionality, and dispensable features. This is undesirable >> for several reasons. >> >> * Any additional module increases the maintenance cost for the Python core >> development team. The team has limited resources, reduced maintenance cost >> frees development time for other improvements. >> * Modules in the standard library are generally favored and seen as the >> de-facto solution for a problem. A majority of users only pick 3rd party >> modules to replace a stdlib module, when they have a compelling reason, e.g. >> lxml instead of `xml`. The removal of an unmaintained stdlib module >> increases the chances of a community contributed module to become widely >> used. >> * A lean and mean standard library benefits platforms with limited resources >> like devices with just a few hundred kilobyte of storage (e.g. BBC >> Micro:bit). Python on mobile platforms like BeeWare or WebAssembly >> (e.g. pyodide) also benefit from reduced download size. >> >> The modules in the PEP have been selected for deprecation because their >> removal is either least controversial or most beneficial. For example >> least controversial are 30 years old multimedia formats like ``sunau`` >> audio format, which was used on SPARC and NeXT workstations in the late >> 1980ties. The ``crypt`` module has fundamental flaws that are better solved >> outside the standard library. >> >> This PEP also designates some modules as not scheduled for removal. Some >> modules have been deprecated for several releases or seem unnecessary at >> first glance. However it is beneficial to keep the modules in the standard >> library, mostly for environments where installing a package from PyPI is not >> an option. This can be cooperate environments or class rooms where external >> code is not permitted without legal approval. >> >> * The usage of FTP is declining, but some files are still provided over >> the FTP protocol or hosters offer FTP to upload content. Therefore >> ``ftplib`` is going to stay. >> * The ``optparse`` and ``getopt`` module are widely used. They are mature >> modules with very low maintenance overhead. >> * According to David Beazley [5]_ the ``wave`` module is easy to teach to >> kids and can make crazy sounds. Making a computer generate crazy sounds is >> powerful and highly motivating exercise for a 9yo aspiring developer. It's >> a fun battery to keep. >> >> >> Deprecation schedule >> ==================== >> >> 3.8 >> --- >> >> This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released >> a few months before Python 2.7 will reach its end of lifetime. We expect that >> Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and >> 2020. To reduce churn and to allow a smooth transition from Python 2, >> Python 3.8 will neither raise `DeprecationWarning` nor remove any >> modules that have been scheduled for removal. Instead deprecated modules will >> just be *documented* as deprecated. Optionally modules may emit a >> `PendingDeprecationWarning`. >> >> All deprecated modules will also undergo a feature freeze. No additional >> features should be added. Bug should still be fixed. >> >> 3.9 >> --- >> >> Starting with Python 3.9, deprecated modules will start issuing >> `DeprecationWarning`. >> >> >> 3.10 >> ---- >> >> In 3.10 all deprecated modules will be removed from the CPython repository >> together with tests, documentation, and autoconf rules. >> >> >> PEP acceptance process >> ====================== >> >> 3.8.0b1 is scheduled to be release shortly after the PEP is officially >> submitted. Since it's improbable that the PEP will pass all stages of the >> PEP process in time, I propose a two step acceptance process that is >> analogous Python's two release deprecation process. >> >> The first *provisionally accepted* phase targets Python 3.8.0b1. In the first >> phase no code is changes or removed. Modules are only documented as >> deprecated. >> >> The final decision, which modules will be removed and how the removed code >> is preserved, can be delayed for another year. >> >> >> Deprecated modules >> ================== >> >> The modules are grouped as data encoding, multimedia, network, OS interface, >> and misc modules. The majority of modules are for old data formats or >> old APIs. Some others are rarely useful and have better replacements on >> PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with >> audio processing. >> >> .. csv-table:: Table 1: Proposed modules deprecations >> :header: "Module", "Deprecated in", "To be removed", "Replacement" >> >> aifc,3.8,3.10,\- >> asynchat,3.8,3.10,asyncio >> asyncore,3.8,3.10,asyncio >> audioop,3.8,3.10,\- >> binhex,3.8,3.10,\- >> cgi,3.8,3.10,\- >> cgitb,3.8,3.10,\- >> chunk,3.8,3.10,\- >> colorsys,**3.8?**,**3.10?**,\- >> crypt,3.8,3.10,\- >> fileinput,3.8,3.10,argparse >> formatter,3.4,3.10,\- >> fpectl,**3.7**,**3.7**,\- >> getopt,**3.2**,**keep**,"argparse, optparse" >> imghdr,3.8,3.10,\- >> imp,**3.4**,3.10,importlib >> lib2to3,\-,**keep**, >> macpath,**3.7**,**3.8**,\- >> msilib,3.8,3.10,\- >> nntplib,3.8,3.10,\- >> nis,3.8,3.10,\- >> optparse,\-,**keep**,argparse >> ossaudiodev,3.8,3.10,\- >> pipes,3.8,3.10,subprocess >> smtpd,**3.7**,3.10,aiosmtpd >> sndhdr,3.8,3.10,\- >> spwd,3.8,3.10,\- >> sunau,3.8,3.10,\- >> uu,3.8,3.10,\- >> wave,\-,**keep**, >> xdrlib,3.8,3.10,\- >> >> >> Data encoding modules >> --------------------- >> >> binhex >> ~~~~~~ >> >> The `binhex `_ module encodes >> and decodes Apple Macintosh binhex4 data. It was originally developed for >> TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to >> encode binary email attachments. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> uu >> ~~ >> >> The `uu `_ module provides >> uuencode format, an old binary encoding format for email from 1980. The uu >> format has been replaced by MIME. The uu codec is provided by the binascii >> module. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> xdrlib >> ~~~~~~ >> >> The `xdrlib `_ module supports >> the Sun External Data Representation Standard. XDR is an old binary >> serialization format from 1987. These days it's rarely used outside >> specialized domains like NFS. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Multimedia modules >> ------------------ >> >> aifc >> ~~~~ >> >> The `aifc `_ module provides >> support for reading and writing AIFF and AIFF-C files. The Audio Interchange >> File Format is an old audio format from 1988 based on Amiga IFF. It was most >> commonly used on the Apple Macintosh. These days only few specialized >> application use AIFF. >> >> Module type >> pure Python (depends on `audioop`_ C extension) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> audioop >> ~~~~~~~ >> >> The `audioop `_ module >> contains helper functions to manipulate raw audio data and adaptive >> differential pulse-code modulated audio data. The module is implemented in >> C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ >> module depend on `audioop`_ for some operations. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> colorsys >> ~~~~~~~~ >> >> The `colorsys `_ module >> defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate >> systems. The Pillow library provides much faster conversation between >> color systems. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `Pillow `_, >> `colorspacious `_ >> >> chunk >> ~~~~~ >> >> The `chunk `_ module provides >> support for reading and writing Electronic Arts' Interchange File Format. >> IFF is an old audio file format originally introduced for Commodore and >> Amiga. The format is no longer relevant. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> imghdr >> ~~~~~~ >> >> The `imghdr `_ module is a >> simple tool to guess the image file format from the first 32 bytes >> of a file or buffer. It supports only a limited amount of formats and >> neither returns resolution nor color depth. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> ossaudiodev >> ~~~~~~~~~~~ >> >> The `ossaudiodev `_ >> module provides support for Open Sound System, an interface to sound >> playback and capture devices. OSS was initially free software, but later >> support for newer sound devices and improvements were proprietary. Linux >> community abandoned OSS in favor of ALSA [1]_. Some operation systems like >> OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> sndhdr >> ~~~~~~ >> >> The `sndhdr `_ module is >> similar to the `imghdr`_ module but for audio formats. It guesses file >> format, channels, frame rate, and sample widths from the first 512 bytes of >> a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and >> other ancient formats. >> >> Module type >> pure Python (depends on `audioop`_ C extension for some operations) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> sunau >> ~~~~~ >> >> The `sunau `_ module provides >> support for Sun AU sound format. It's yet another old, obsolete file format. >> >> Module type >> pure Python (depends on `audioop`_ C extension for some operations) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Networking modules >> ------------------ >> >> asynchat >> ~~~~~~~~ >> >> The `asynchat `_ module >> is build on top of `asyncore`_ and has been deprecated since Python 3.6. >> >> Module type >> pure Python >> Deprecated in >> 3.6 >> Removed in >> 3.10 >> Substitute >> asyncio >> >> asyncore >> ~~~~~~~~ >> >> The `asyncore `_ module was >> the first module for asynchronous socket service clients and servers. It >> has been replaced by asyncio and is deprecated since Python 3.6. >> >> The ``asyncore`` module is also used in stdlib tests. The tests for >> ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly >> based on ``asyncore``. These tests must be updated to use asyncio or >> threading. >> >> Module type >> pure Python >> Deprecated in >> 3.6 >> Removed in >> 3.10 >> Substitute >> asyncio >> >> >> cgi >> ~~~ >> >> The `cgi `_ module is a support >> module for Common Gateway Interface (CGI) scripts. CGI is deemed as >> inefficient because every incoming request is handled in a new process. PEP >> 206 considers the module as *designed poorly and are now near-impossible >> to fix*. >> >> Several people proposed to either keep the cgi module for features like >> `cgi.parse_qs()` or move `cgi.escape()` to a different module. The >> functions `cgi.parse_qs` and `cgi.parse_qsl` have been >> deprecated for a while and are actually aliases for >> `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The >> function `cgi.quote` has been deprecated in favor of `html.quote` >> with secure default values. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> cgitb >> ~~~~~ >> >> The `cgitb `_ module is a >> helper for the cgi module for configurable tracebacks. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> smtpd >> ~~~~~ >> >> The `smtpd `_ module provides >> a simple implementation of a SMTP mail server. The module documentation >> recommends ``aiosmtpd``. >> >> Module type >> pure Python >> Deprecated in >> **3.7** >> To be removed in >> 3.10 >> Substitute >> aiosmtpd >> >> nntplib >> ~~~~~~~ >> >> The `nntplib `_ module >> implements the client side of the Network News Transfer Protocol (nntp). News >> groups used to be a dominant platform for online discussions. Over the last >> two decades, news has been slowly but steadily replaced with mailing lists >> and web-based discussion platforms. >> >> The ``nntplib`` tests have been the cause of additional work in the recent >> past. Python only contains client side of NNTP. The test cases depend on >> external news server. These servers were unstable in the past. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Operating system interface >> -------------------------- >> >> crypt >> ~~~~~ >> >> The `crypt `_ module implements >> password hashing based on ``crypt(3)`` function from ``libcrypt`` or >> ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor >> quality and insecure. Users are discouraged to use them. >> >> * The module is not available on Windows. Cross-platform application need >> an alternative implementation any way. >> * Only DES encryption is guarenteed to be available. DES has an extremely >> limited key space of 2**56. >> * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. >> SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only >> algorithm that is still secure. However it's in glibc and therefore not >> commonly available on Linux. >> * Depending on the platform, the ``crypt`` module is not thread safe. Only >> implementations with ``crypt_r(3)`` are thread safe. >> >> Module type >> C extension + Python module >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `bcrypt `_, >> `passlib `_, >> `argon2cffi `_, >> hashlib module (PBKDF2, scrypt) >> >> macpath >> ~~~~~~~ >> >> The `macpath `_ module >> provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer >> supported >> >> Module type >> pure Python >> Deprecated in >> 3.7 >> Removed in >> 3.8 >> Substitute >> **none** >> >> nis >> ~~~ >> >> The `nis `_ module provides >> NIS/YP support. Network Information Service / Yellow Pages is an old and >> deprecated directory service protocol developed by Sun Microsystems. It's >> designed successor NIS+ from 1992 never took off. For a long time, libc's >> Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful >> and more secure replacement of NIS. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> spwd >> ~~~~ >> >> The `spwd `_ module provides >> direct access to Unix shadow password database using non-standard APIs. >> In general it's a bad idea to use the spwd. The spwd circumvents system >> security policies, it does not use the PAM stack, and is >> only compatible with local user accounts. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> Misc modules >> ------------ >> >> fileinput >> ~~~~~~~~~ >> >> The `fileinput `_ module >> implements a helpers to iterate over a list of files from ``sys.argv``. The >> module predates the optparser and argparser module. The same functionality >> can be implemented with the argparser module. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> argparse >> >> formatter >> ~~~~~~~~~ >> >> The `formatter `_ module >> is an old text formatting module which has been deprecated since Python 3.4. >> >> Module type >> pure Python >> Deprecated in >> 3.4 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> imp >> ~~~ >> >> The `imp `_ module is the >> predecessor of the >> `importlib `_ module. Most >> functions have been deprecated since Python 3.3 and the module since >> Python 3.4. >> >> Module type >> C extension >> Deprecated in >> 3.4 >> To be removed in >> 3.10 >> Substitute >> importlib >> >> msilib >> ~~~~~~ >> >> The `msilib `_ package is a >> Windows-only package. It supports the creation of Microsoft Installers (MSI). >> The package also exposes additional APIs to create cabinet files (CAB). The >> module is used to facilitate distutils to create MSI installers with >> ``bdist_msi`` command. In the past it was used to create CPython's official >> Windows installer, too. >> >> Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) >> as new deployment model [3]_. >> >> Module type >> C extension + Python code >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> pipes >> ~~~~~ >> >> The `pipes `_ module provides >> helpers to pipe the input of one command into the output of another command. >> The module is built on top of ``os.popen``. Users are encouraged to use >> the subprocess module instead. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> subprocess module >> >> Removed modules >> =============== >> >> fpectl >> ------ >> >> The `fpectl `_ module was >> never built by default, its usage was discouraged and considered dangerous. >> It also required a configure flag that caused an ABI incompatibility. The >> module was removed in 3.7 by Nathaniel J. Smith in >> `bpo-29137 `_. >> >> Module type >> C extension + CAPI >> Deprecated in >> 3.7 >> Removed in >> 3.7 >> Substitute >> **none** >> >> >> Modules to keep >> =============== >> >> Some modules were originally proposed for deprecation. >> >> lib2to3 >> ------- >> >> The `lib2to3 `_ package provides >> the ``2to3`` command to transpile Python 2 code to Python 3 code. >> >> The package is useful for other tasks besides porting code from Python 2 to >> 3. For example `black`_ uses it for code reformatting. >> >> Module type >> pure Python >> >> getopt >> ------ >> >> The `getopt `_ module mimics >> C's getopt() option parser. Although users are encouraged to use argparse >> instead, the getopt module is still widely used. >> >> Module type >> pure Python >> >> optparse >> -------- >> >> The `optparse `_ module is >> the predecessor of the argparse module. Although it has been deprecated for >> many years, it's still widely used. >> >> Module type >> pure Python >> Deprecated in >> 3.2 >> Substitute >> argparse >> >> wave >> ~~~~ >> >> The `wave `_ module provides >> support for the WAV sound format. The module uses one simple function >> from the `audioop`_ module to perform byte swapping between little and big >> endian formats. Before 24 bit WAV support was added, byte swap used to be >> implemented with the ``array`` module. To remove ``wave``'s dependency on the >> ``audioop``, the byte swap function could be either be moved to another >> module (e.g. ``operator``) or the ``array`` module could gain support for >> 24 bit (3 byte) arrays. >> >> Module type >> pure Python (depends on *byteswap* from `audioop`_ C extension) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> >> Future maintenance of removed modules >> ===================================== >> >> The main goal of the PEP is to reduce the burden and workload on the Python >> core developer team. Therefore removed modules will not be maintained by >> the core team as separate PyPI packages. However the removed code, tests and >> documentation may be moved into a new git repository, so community members >> have a place from which they can pick up and fork code. >> >> A first draft of a `legacylib `_ >> repository is available on my private Github account. >> >> It's my hope that some of the deprecated modules will be picked up and >> adopted by users that actually care about them. For example ``colorsys`` and >> ``imghdr`` are useful modules, but have limited feature set. A fork of >> ``imghdr`` can add new features and support for more image formats, without >> being constrained by Python's release cycle. >> >> Most of the modules are in pure Python and can be easily packaged. Some >> depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ >> does not depend on any external libraries, it can be shipped in as binary >> wheels with some effort. Other C modules can be replaced with ctypes or cffi. >> For example I created `legacycrypt `_ >> with ``_crypt`` extension reimplemented with a few lines of ctypes code. >> >> >> Discussions >> =========== >> >> * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. >> * Berker Peksag proposed to deprecate and removed *msilib*. >> * Brett Cannon recommended to delay active deprecation warnings and removal >> of modules like *imp* until Python 3.10. Version 3.8 will be released >> shortly before Python 2 reaches end of lifetime. A delay reduced churn for >> users that migrate from Python 2 to 3.8. >> * Brett also came up with the idea to keep lib2to3. The package is useful >> for other purposes, e.g. `black `_ uses >> it to reformat Python code. >> * At one point, distutils was mentioned in the same sentence as this PEP. >> To avoid lengthy discussion and delay of the PEP, I decided against dealing >> with distutils. Deprecation of the distutils package will be handled by >> another PEP. >> * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) >> convinced me to keep the `wave`_ module. [4]_ >> * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ >> >> >> References >> ========== >> >> .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free >> .. [2] https://man.openbsd.org/ossaudio >> .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ >> .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 >> .. [5] https://twitter.com/dabeaz/status/1130278844479545351 >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> >> > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/websurfer%40surf2c.net -- ----------------------------------------- From there to here, from here to there, funny things are everywhere. -- Theodore Geisel -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon May 20 18:13:45 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 May 2019 00:13:45 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> Message-ID: <20190521001345.419d5948@fsol> On Tue, 21 May 2019 00:06:35 +0200 Christian Heimes wrote: > On 20/05/2019 23.27, Antoine Pitrou wrote: > > NNTP is still quite used (often through GMane, but probably not only) so > > I'd question the removal of nntplib. > > Is NNTP support important enough to keep the module in the standard library? I'd phrase the question differently: is NNTP dead enough, or nntplib painful enough to maintain, that it's worth removing it from the stdlib? If the stdlib didn't have NNTP support, obviously nobody would suggest adding it nowadays. But it has that support, and there are certainly uses of it in the wild, so we must take that into account. > > If the wave module depends on the audioop module, and if the wave > > module is kept in the stdlib, then the audioop module can't be removed. > > No, it can be removed. I explained the situation in the "wave" section of the PEP. My bad. I had skipped that. Regards Antoine. From hodgestar+pythondev at gmail.com Mon May 20 18:41:19 2019 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Tue, 21 May 2019 00:41:19 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: Woot. +100 on this PEP. > If the stdlib didn't have NNTP support, obviously nobody would suggest > adding it nowadays. Perhaps this is a good reason to keep nntplib in the deprecation list? Another question is "are there any places using nntplib where `pip install nntplib`" is not an reasonable option? There's quite a lot of time before 3.10 to move nntplib into an outside package. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon May 20 18:53:06 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 May 2019 00:53:06 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library References: Message-ID: <20190521005306.149eb2ae@fsol> On Tue, 21 May 2019 00:41:19 +0200 Simon Cross wrote: > Woot. +100 on this PEP. > > > If the stdlib didn't have NNTP support, obviously nobody would suggest > > adding it nowadays. > > Perhaps this is a good reason to keep nntplib in the deprecation list? No, because the same applies to getopt, optparse and others, which are not in the deprecation list. > Another question is "are there any places using nntplib where `pip install > nntplib`" is not an reasonable option? You could ask pretty much the same question about most stdlib packages. Isn't "pip install ftplib" a reasonable option? How about "pip install email" or "pip install xmlrpc"? The PEP is about "dead batteries". Not "batteries that some people think would be reasonable to install from PyPI". Regards Antoine. From christian at python.org Mon May 20 18:55:20 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 00:55:20 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190521001345.419d5948@fsol> References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> <20190521001345.419d5948@fsol> Message-ID: On 21/05/2019 00.13, Antoine Pitrou wrote: > On Tue, 21 May 2019 00:06:35 +0200 > Christian Heimes wrote: >> On 20/05/2019 23.27, Antoine Pitrou wrote: >>> NNTP is still quite used (often through GMane, but probably not only) so >>> I'd question the removal of nntplib. >> >> Is NNTP support important enough to keep the module in the standard library? > > I'd phrase the question differently: is NNTP dead enough, or nntplib > painful enough to maintain, that it's worth removing it from the stdlib? The module itself does not create much work. But its tests are a regular source of pain and instabilities. The tests for nntplib depend on external NNTP servers. These servers are sometimes down, very slow, or don't work over IPv6. I'm sure that Pablo and Victor can tell you some war stories. I briefly mentioned the issues in the PEP, too. > If the stdlib didn't have NNTP support, obviously nobody would suggest > adding it nowadays. But it has that support, and there are certainly > uses of it in the wild, so we must take that into account. > >>> If the wave module depends on the audioop module, and if the wave >>> module is kept in the stdlib, then the audioop module can't be removed. >> >> No, it can be removed. I explained the situation in the "wave" section of the PEP. > > My bad. I had skipped that. The section in audioop was confusing. I have updated the audioop paragraph and the crypt paragraph with your feedback. I'll keep the PR https://github.com/python/peps/pull/1063 option for a couple of days to collect more feedback. Christian From tjreedy at udel.edu Mon May 20 18:59:44 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 20 May 2019 18:59:44 -0400 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: On 5/20/2019 11:55 AM, Guido van Rossum wrote: > On Thu, May 16, 2019 at 3:57 PM Steve Dower > wrote: > > [...] > We still have the policy of not removing modules that exist in the > Python 2 standard library. But 3.9 won't be covered by that :) > > I didn't even remember that. Where's that written down? AFAIK, there has been no BDFL pronouncement or coredev vote. But it has become somewhat of a consensus as a result of discussions in multliple threads about removing modules going back to the 3.4 era. PEP 594, just posted, proposes to codify a 'soft' (or perhaps 'slow') version of the policy: doc deprecation, code deprecation, and code removal in 3.8, 3.9, 3.10. > FWIW I am strongly in favor of getting rid of the `parser` module, in > 3.8 if we can, otherwise in 3.9 (after strong deprecation in 3.8). You could request that 'parser' be specifically excluded from the PEP, along with 'distutils' (and effectively anything else not specifically named). Or request that it be included, but with an accelerated schedule. I have a vague idea of why you think it harmful to keep it around, but a reminder would not hurt ;-). -- Terry Jan Reedy From tjreedy at udel.edu Mon May 20 19:06:26 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 20 May 2019 19:06:26 -0400 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> Message-ID: On 5/20/2019 6:06 PM, Christian Heimes wrote: >> Removing the crypt module would remove support for system-standard >> password files. I don't understand the rationale. > > Applications *must* not access system-standard password files directly. On any sanely and securely configured systems, application cannot even access system password files like /etc/shadow. Access restrictions and system security policies will prevent read access. Also applications cannot assume that users are present in any user file. They may come from LDAP, SSSD, ActiveDirectory, or other sources. > > The correct way to interact with system users is to use the proper APIs, that are NSS (name service switch) and PAM (pluggable authentication modules). NSS looks up and enumerate users and groups. PAM performs password validation and much, much, much more. The pwd and grp modules use the correct APIs to interact with NSS. If you need to check or change passwords, you must go through PAM. Add this to the PEP? It might suggest that crypt should go away sooner. -- Terry Jan Reedy From christian at python.org Mon May 20 19:30:30 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 01:30:30 +0200 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: <27a1d22f-299b-a2b1-6093-046aaf1e136b@python.org> On 16/05/2019 23.12, Pablo Galindo Salgado wrote: > Hi everyone, > > TLDR > ===== > > I propose to remove the current parser module and expose pgen2 as a standard library module. I like to add this to PEP 594, see https://github.com/python/peps/pull/1063 Terry, thanks for connecting my PEP with Pablo's proposal. From christian at python.org Mon May 20 19:32:06 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 01:32:06 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> Message-ID: On 21/05/2019 01.06, Terry Reedy wrote: > On 5/20/2019 6:06 PM, Christian Heimes wrote: > >>> Removing the crypt module would remove support for system-standard >>> password files.? I don't understand the rationale. >> >> Applications *must* not access system-standard password files directly. On any sanely and securely configured systems, application cannot even access system password files like /etc/shadow. Access restrictions and system security policies will prevent read access. Also applications cannot assume that users are present in any user file. They may come from LDAP, SSSD, ActiveDirectory, or other sources. >> >> The correct way to interact with system users is to use the proper APIs, that are NSS (name service switch) and PAM (pluggable authentication modules). NSS looks up and enumerate users and groups. PAM performs password validation and much, much, much more. The pwd and grp modules use the correct APIs to interact with NSS. If you need to check or change passwords, you must go through PAM. > > Add this to the PEP?? It might suggest that crypt should go away sooner. Yes, I'll do that. I'm currently collecting updates from feedback in PR https://github.com/python/peps/pull/1063 Christian From songofacandy at gmail.com Mon May 20 20:16:52 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Tue, 21 May 2019 09:16:52 +0900 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: I use fileinput for several times per year. fileinput is handy tool to write single script file to analyze log files. * In such tools, I don't need real argument parser. * Some log files are compressed and some are not. It seems argparse doesn't support transparent decompression. * I don't want to use 3rd party library for such single script files. I'd like to add transparent decompression to argparse if fileinput is deprecated. Regards, -- Inada Naoki From guido at python.org Mon May 20 21:56:49 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 20 May 2019 18:56:49 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: Yeah, I was surprised to see fileinput on the list. It's indeed a handy little tool. On Mon, May 20, 2019 at 5:19 PM Inada Naoki wrote: > I use fileinput for several times per year. > > fileinput is handy tool to write single script file to analyze log files. > > * In such tools, I don't need real argument parser. > * Some log files are compressed and some are not. > It seems argparse doesn't support transparent decompression. > * I don't want to use 3rd party library for such single script files. > > I'd like to add transparent decompression to argparse if fileinput > is deprecated. > > Regards, > > -- > Inada Naoki > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 21 04:15:05 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 10:15:05 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: On 21/05/2019 02.16, Inada Naoki wrote: > I use fileinput for several times per year. > > fileinput is handy tool to write single script file to analyze log files. > > * In such tools, I don't need real argument parser. > * Some log files are compressed and some are not. > It seems argparse doesn't support transparent decompression. > * I don't want to use 3rd party library for such single script files. OK, let's keep it. I was under the impression that it's not used. Christian From walter at livinglogic.de Tue May 21 05:06:55 2019 From: walter at livinglogic.de (Walter =?utf-8?q?D=C3=B6rwald?=) Date: Tue, 21 May 2019 11:06:55 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: On 20 May 2019, at 22:15, Christian Heimes wrote: > Hi, > > here is the first version of my PEP 594 to deprecate and eventually > remove modules from the standard library. The PEP started last year > with talk during the Python Language Summit 2018, > https://lwn.net/Articles/755229/. > > [...] > > colorsys > ~~~~~~~~ > > The `colorsys `_ > module > defines color conversion functions between RGB, YIQ, HSL, and HSV > coordinate > systems. The Pillow library provides much faster conversation between > color systems. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `Pillow `_, > `colorspacious `_ I'm using colorsys constantly as the basis for a tool that converts CSS colors between different coordinate systems. I don't see how that could be done via Pillow (which AFAICT only converts complete images). RGB<->HSV<->HLS conversion seems to be not available (or not obvious) in colorspacious. colorsys is a module where we can be pretty sure that it has zero bugs, and doesn't require any maintenance or security updates, so I don't see any reason to deprecate it. > [...] Servus, Walter From njs at pobox.com Tue May 21 05:49:18 2019 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 21 May 2019 02:49:18 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: On Tue, May 21, 2019 at 2:40 AM Walter D?rwald wrote: > > On 20 May 2019, at 22:15, Christian Heimes wrote: > > > Hi, > > > > here is the first version of my PEP 594 to deprecate and eventually > > remove modules from the standard library. The PEP started last year > > with talk during the Python Language Summit 2018, > > https://lwn.net/Articles/755229/. > > > > [...] > > > > colorsys > > ~~~~~~~~ > > > > The `colorsys `_ > > module > > defines color conversion functions between RGB, YIQ, HSL, and HSV > > coordinate > > systems. The Pillow library provides much faster conversation between > > color systems. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > `Pillow `_, > > `colorspacious `_ > > I'm using colorsys constantly as the basis for a tool that converts CSS > colors between different coordinate systems. I don't see how that could > be done via Pillow (which AFAICT only converts complete images). > RGB<->HSV<->HLS conversion seems to be not available (or not obvious) in > colorspacious. Correct, colorspacious doesn't support HSV or HLS. I suppose it would be trivial enough to add... The 'colour' package has them (along with everything else you can dream of): https://colour.readthedocs.io/en/latest/colour.models.html -n -- Nathaniel J. Smith -- https://vorpus.org From solipsis at pitrou.net Tue May 21 06:01:14 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 May 2019 12:01:14 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> <20190521001345.419d5948@fsol> Message-ID: <20190521120114.6c14be3c@fsol> On Tue, 21 May 2019 00:55:20 +0200 Christian Heimes wrote: > On 21/05/2019 00.13, Antoine Pitrou wrote: > > On Tue, 21 May 2019 00:06:35 +0200 > > Christian Heimes wrote: > >> On 20/05/2019 23.27, Antoine Pitrou wrote: > >>> NNTP is still quite used (often through GMane, but probably not only) so > >>> I'd question the removal of nntplib. > >> > >> Is NNTP support important enough to keep the module in the standard library? > > > > I'd phrase the question differently: is NNTP dead enough, or nntplib > > painful enough to maintain, that it's worth removing it from the stdlib? > > The module itself does not create much work. But its tests are a regular source of pain and instabilities. The tests for nntplib depend on external NNTP servers. These servers are sometimes down, very slow, or don't work over IPv6. I'm sure that Pablo and Victor can tell you some war stories. I briefly mentioned the issues in the PEP, too. I plead guilty for adding those tests :-/ Back then, it seemed reasonable to test the nntplib module against real-world servers, since like all old Internet text protocols NNTP is not always implemented rigorously. If the NNTP networked tests are an annoyance, they should probably be skipped on CI. Regards Antoine. From steve at holdenweb.com Tue May 21 06:02:05 2019 From: steve at holdenweb.com (Steve Holden) Date: Tue, 21 May 2019 11:02:05 +0100 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: It's covered in "Python in a Nutshell," Alex Martelli having been a promoter of its ability simplify many utility programs for a long time. Not that that's any guide as to what should be in 3.10, by which time we'll be four minor releases out of date anyway. On Tue, May 21, 2019 at 9:16 AM Christian Heimes wrote: > On 21/05/2019 02.16, Inada Naoki wrote: > > I use fileinput for several times per year. > > > > fileinput is handy tool to write single script file to analyze log files. > > > > * In such tools, I don't need real argument parser. > > * Some log files are compressed and some are not. > > It seems argparse doesn't support transparent decompression. > > * I don't want to use 3rd party library for such single script files. > > OK, let's keep it. I was under the impression that it's not used. > > Christian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 21 06:06:03 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 12:06:03 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: <8ade5b62-75d1-589e-3b7b-0678f5e239ec@python.org> On 21/05/2019 11.49, Nathaniel Smith wrote: > On Tue, May 21, 2019 at 2:40 AM Walter D?rwald wrote: >> >> On 20 May 2019, at 22:15, Christian Heimes wrote: >> >>> Hi, >>> >>> here is the first version of my PEP 594 to deprecate and eventually >>> remove modules from the standard library. The PEP started last year >>> with talk during the Python Language Summit 2018, >>> https://lwn.net/Articles/755229/. >>> >>> [...] >>> >>> colorsys >>> ~~~~~~~~ >>> >>> The `colorsys `_ >>> module >>> defines color conversion functions between RGB, YIQ, HSL, and HSV >>> coordinate >>> systems. The Pillow library provides much faster conversation between >>> color systems. >>> >>> Module type >>> pure Python >>> Deprecated in >>> 3.8 >>> To be removed in >>> 3.10 >>> Substitute >>> `Pillow `_, >>> `colorspacious `_ >> >> I'm using colorsys constantly as the basis for a tool that converts CSS >> colors between different coordinate systems. I don't see how that could >> be done via Pillow (which AFAICT only converts complete images). >> RGB<->HSV<->HLS conversion seems to be not available (or not obvious) in >> colorspacious. > > Correct, colorspacious doesn't support HSV or HLS. I suppose it would > be trivial enough to add... > > The 'colour' package has them (along with everything else you can > dream of): https://colour.readthedocs.io/en/latest/colour.models.html Nice catch, I just added https://python-colormath.readthedocs.io/en/latest/ to my update PR. I'll add colour to the list, too. (It didn't pop up on my radar because I wasn't checking for British spelling) Christian From g.rodola at gmail.com Tue May 21 06:19:08 2019 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 21 May 2019 17:19:08 +0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: On Tue, 21 May 2019 at 03:17, Christian Heimes wrote: spwd > ~~~~ > > The `spwd `_ module provides > direct access to Unix shadow password database using non-standard APIs. > In general it's a bad idea to use the spwd. The spwd circumvents system > security policies, it does not use the PAM stack, and is > only compatible with local user accounts. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** I find this one useful and would be a bit sad to see it go. FWIW I use it in pyftpdlib and I suppose there are other apps out there relying on UNIX password db for authentication. The fact that it?s a C module is also an incentive to leave it in the stdlib IMO (pure python modules can easily be copied in the project instead of retrieving them from PYPI as a third party dep - e.g. this is how I am likely going to replace asyncore/asynchat). A big +1 to this PEP by the way. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 21 06:27:58 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 12:27:58 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: <04e9a5f2-3ce7-df61-9482-5fed26577ad1@python.org> On 21/05/2019 12.19, Giampaolo Rodola' wrote: > I find this one useful and would be a bit sad to see it go. FWIW I use it in pyftpdlib and I suppose there are other apps out there relying on UNIX password db for authentication. The fact that it?s a C module is also an incentive to leave it in the stdlib IMO (pure python modules can easily be copied in the project instead of retrieving them from PYPI as a third party dep - e.g. this is how I am likely going to replace asyncore/asynchat). If you use the spwd module for authentication, then you have a major security problem in your application. You must use the PAM stack to authenticate access to a service. Christian From nd at perlig.de Tue May 21 07:08:49 2019 From: nd at perlig.de (=?ISO-8859-1?Q?Andr=E9?= Malo) Date: Tue, 21 May 2019 13:08:49 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190520232749.442c45c7@fsol> References: <20190520232749.442c45c7@fsol> Message-ID: <2215566.ODhDVceCri@finnegan> On Montag, 20. Mai 2019 23:27:49 CEST Antoine Pitrou wrote: > NNTP is still quite used (often through GMane, but probably not only) so > I'd question the removal of nntplib. > > cgitb used to be used by some Web frameworks in order to format > exceptions. Perhaps one should check if that's still the case. I concur with both of those. There's software in production using both. (It doesn't mean it's on pypi or even free software). What would be the maintenance burden of those modules anyway? (at least for nntp, I guess it's not gonna change). nd -- package Hacker::Perl::Another::Just;print qq~@{[reverse split/::/ =>__PACKAGE__]}~; # Andr? Malo # http://pub.perlig.de # From vstinner at redhat.com Tue May 21 07:24:34 2019 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 21 May 2019 13:24:34 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <2215566.ODhDVceCri@finnegan> References: <20190520232749.442c45c7@fsol> <2215566.ODhDVceCri@finnegan> Message-ID: Le mar. 21 mai 2019 ? 13:18, Andr? Malo a ?crit : > There's software in production using both. (It doesn't mean it's on pypi or > even free software). > > What would be the maintenance burden of those modules anyway? (at least for > nntp, I guess it's not gonna change). The maintenance burden is real even if it's not visible. For example, test_nntplib is causing frequently issues on our CI: https://bugs.python.org/issue19756 https://bugs.python.org/issue19613 https://bugs.python.org/issue31850 It's failing frequently since 2013, and nobody managed to come with a fix.. in 6 years. There are 11 open issues with "nntp" in their title (8 with exactly "nntplib" in their title). test_nntplib uses the public server news.trigofacile.com which is operated by Julien ?LIE. Two years ago, Julien asked me if there is any plan to support the NNTP "COMPRESS" command. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From nd at perlig.de Tue May 21 07:39:44 2019 From: nd at perlig.de (=?ISO-8859-1?Q?Andr=E9?= Malo) Date: Tue, 21 May 2019 13:39:44 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <2215566.ODhDVceCri@finnegan> Message-ID: <2019037.SGpajkmyWO@finnegan> On Dienstag, 21. Mai 2019 13:24:34 CEST Victor Stinner wrote: > Le mar. 21 mai 2019 ? 13:18, Andr? Malo a ?crit : > > There's software in production using both. (It doesn't mean it's on pypi > > or even free software). > > > > What would be the maintenance burden of those modules anyway? (at least > > for nntp, I guess it's not gonna change). > > The maintenance burden is real even if it's not visible. For example, > test_nntplib is causing frequently issues on our CI: > > https://bugs.python.org/issue19756 > https://bugs.python.org/issue19613 > https://bugs.python.org/issue31850 > > It's failing frequently since 2013, and nobody managed to come with a > fix.. in 6 years. > > There are 11 open issues with "nntp" in their title (8 with exactly > "nntplib" in their title). > > test_nntplib uses the public server news.trigofacile.com which is > operated by Julien ?LIE. Two years ago, Julien asked me if there is > any plan to support the NNTP "COMPRESS" command. So what I hear is, this battery is definitely not dead, which is what the PEP is all about. it's just half charged (or discharged, depending on your POV), so to speak. Substitute: "none" should read pypi then? nd -- "Solides und umfangreiches Buch" -- aus einer Rezension From christian at python.org Tue May 21 07:46:34 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 13:46:34 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <2215566.ODhDVceCri@finnegan> References: <20190520232749.442c45c7@fsol> <2215566.ODhDVceCri@finnegan> Message-ID: On 21/05/2019 13.08, Andr? Malo wrote: > On Montag, 20. Mai 2019 23:27:49 CEST Antoine Pitrou wrote: >> NNTP is still quite used (often through GMane, but probably not only) so >> I'd question the removal of nntplib. >> >> cgitb used to be used by some Web frameworks in order to format >> exceptions. Perhaps one should check if that's still the case. > > I concur with both of those. > There's software in production using both. (It doesn't mean it's on pypi or > even free software). There is always somebody who uses a feature. This argument blocks any innovation or cleanup. Victor just reminded me of https://xkcd.com/1172/ . * The removed modules will be available through PyPI. * You don't have to start to worry until Python 3.10 is released in over 3 years from now. * The modules are fully supported in Python 3.8 and 3.9. Python 3.9 will reach EOL late 2026 or early 2027. That's plenty of time. Christian From srittau at rittau.biz Tue May 21 07:47:36 2019 From: srittau at rittau.biz (Sebastian Rittau) Date: Tue, 21 May 2019 13:47:36 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <2019037.SGpajkmyWO@finnegan> References: <2215566.ODhDVceCri@finnegan> <2019037.SGpajkmyWO@finnegan> Message-ID: <42a1a7db-b2c8-3fbc-b9be-c844f2aeeda8@rittau.biz> Am 21.05.19 um 13:39 schrieb Andr? Malo: > > So what I hear is, this battery is definitely not dead, which is what the > PEP is all about. > it's just half charged (or discharged, depending on your POV), so to speak. > > Substitute: "none" should read pypi then? Every single module on this list will have some users. To me the question is: Are there enough users to justify the extra burden on the maintainers, but also users not using that package? (The latter will have to download the package, have it installed on their systems, it makes it harder to find other things in the documentation etc.) If there not many users left, I believe these should either vendor the last official package or to band together to maintain the package themselves on pypi. ?- Sebastian From vstinner at redhat.com Tue May 21 07:50:30 2019 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 21 May 2019 13:50:30 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: Hi Christian, I dislike the PEP 594 title: "Removing dead batteries from the standard library". A module is never "dead", there are always users, even if there are less than 5 of them. Extract of the Rationale: "The modules are mostly historic data formats and APIs that have been superseded a long time ago" If we take the example of NNTP: even if it's old and "superseded", people still uses NTTP. Python doesn't have to be opinionated about formats. Wait, I like the overall PEP. I'm just talking about how it's explained. IMHO the question here is if the core developers want to continue to pay the price of the maintenance burden for modules which have "few" users (define few...). The other question is if it would be acceptable to move these modules to PyPI. "import wave" would continue to work as previously, but the maintenance would be moved *outside* Python. Who will maintain these modules on PyPI? Do we have to define this right now? If a module is pure Python, well, the easy solution is to embed it into your code: cp /path/to/cpython/Lib/.py /path/to/yourapp/.py. If it's a C extension, it's more complicated. For me the risk of the PEP 594 is to create a new Python 3 mess. Pedantic core devs who explain to users that their code is wrong and they must fix it... But the code was running fine for 20 years! The PEP is full of "Substitute: none". I'm not comfortable with that. I would prefer to require that there is always a solution, like putting the code on PyPI and let it die there. The old mojo "the stdlib is where modules die" would become "PyPI is where old stdlib modules die" :-) Python itself is bad at fixing DeprecationWarning: imp and asyncore are deprecated for years, but they are still widely used inside Python... "Stop using deprecated imp module; imp should now emit a real DeprecationWarning" is open since 2015: https://bugs.python.org/issue25160 "Replace asyncore" open since 2016 https://bugs.python.org/issue28533 "test_poplib replace asyncore" open since 2017 https://bugs.python.org/issue30514 I dislike DeprecationWarning *but* ... well, does it really hurt so much to keep these modules around? asyncore has know bugs, *but* many tests of our test suite are based on that and we managed to make these tests very reliable only with minor work. --- Even if it's not said this way, in fact this PEP reopens the question of splitting the stdlib. It would be helpful to have some references into the PEP about previous discussions on this topic. Recent discussion: "Amber Brown: Batteries Included, But They're Leaking " http://pyfound.blogspot.com/2019/05/amber-brown-batteries-included-but.html By the way, even if it's not directly related, the PEP doesn't address one of the common question: be able to override a stdlib module with a more recent version. For example, if the Python provided by your OS contains a bug in asyncio, you are stuck and cannot easily upgrade it to get a new version without the fix. --- For me, "getopt" is a dead battery. The PEP says "Although users are encouraged to use argparse instead, the getopt module is still widely used." Well, that makes sense. But so, what is the metric to decide if a module is "widely used" or not? Should we keep a module at soon as at least one user says "I'm using it!". In other replies, users asked to keep spwd, fileinput, nntplib, etc. --- Sorry, I mostly opened questions. I don't have answers. I just wanted to share that this PEP scares me a little bit. At least, the current version of the PEP. Victor Victor From ajm at flonidan.dk Tue May 21 08:06:59 2019 From: ajm at flonidan.dk (Anders Munch) Date: Tue, 21 May 2019 12:06:59 +0000 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library Message-ID: Fra: Python-Dev [mailto:python-dev-bounces+ajm=flonidan.dk at python.org] P? vegne af Christian Heimes > * The removed modules will be available through PyPI. Will they? That's not the impression I got from the PEP. regards, Anders From nd at perlig.de Tue May 21 08:25:14 2019 From: nd at perlig.de (=?ISO-8859-1?Q?Andr=E9?= Malo) Date: Tue, 21 May 2019 14:25:14 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: <1894882.UAujLPE7mA@finnegan> On Dienstag, 21. Mai 2019 13:50:30 CEST Victor Stinner wrote: > Well, that makes sense. But so, what is the metric to decide if a > module is "widely used" or not? Yes. Exactly the question that pops up right now. I think, that's the main issue when including batteries in general (that's not news though :-). And the problem I see there is: There *is* no valid answer. (Sorry if I seem to be just annoying. That's not intended, I'm just not carrying the good news.) nd -- package Hacker::Perl::Another::Just;print qq~@{[reverse split/::/ =>__PACKAGE__]}~; # Andr? Malo # http://www.perlig.de # From christian at python.org Tue May 21 08:30:46 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 14:30:46 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: On 21/05/2019 13.50, Victor Stinner wrote: > Hi Christian, > > I dislike the PEP 594 title: "Removing dead batteries from the > standard library". A module is never "dead", there are always users, > even if there are less than 5 of them. I'm open for suggestions for a better title. > Extract of the Rationale: "The modules are mostly historic data > formats and APIs that have been superseded a long time ago" > > If we take the example of NNTP: even if it's old and "superseded", > people still uses NTTP. Python doesn't have to be opinionated about > formats. NNTP was added late in the writing of the PEP. I can rephrase the rational > Wait, I like the overall PEP. I'm just talking about how it's explained. > > IMHO the question here is if the core developers want to continue to > pay the price of the maintenance burden for modules which have "few" > users (define few...). The other question is if it would be acceptable > to move these modules to PyPI. "import wave" would continue to work as > previously, but the maintenance would be moved *outside* Python. > > Who will maintain these modules on PyPI? Do we have to define this right now? Whoever steps up and is interested to maintain the module. But we don't have to define this now. We have more than a year to come up with a plan. Except for the parser module, all modules will be maintained by Python core until 2026. > If a module is pure Python, well, the easy solution is to embed it > into your code: cp /path/to/cpython/Lib/.py > /path/to/yourapp/.py. If it's a C extension, it's more > complicated. This only affects nis, ossaudiodev, spwd, and msi. For crypt I already have a ctypes wrapper. ossaudiodev is dead and has been replaced by ALSA about two decades ago. spwd is bad and users should use PAM instead. > The PEP is full of "Substitute: none". I'm not comfortable with that. > I would prefer to require that there is always a solution, like > putting the code on PyPI and let it die there. The old mojo "the > stdlib is where modules die" would become "PyPI is where old stdlib > modules die" :-) The latest version of the PEP lists many additional substitutes. I'll post a new version soonish. > Python itself is bad at fixing DeprecationWarning: imp and asyncore > are deprecated for years, but they are still widely used inside > Python... We are keeping the old packages to make transition from 2.7 to 3.8 easier. This is going to change with 3.9. > I dislike DeprecationWarning *but* ... well, does it really hurt so > much to keep these modules around? asyncore has know bugs, *but* many > tests of our test suite are based on that and we managed to make these > tests very reliable only with minor work. I'm struggeling with asyncore in SSL tests, mostly because I don't grasp how asyncore works and I have no intention to dig into a deprecated technology. > Even if it's not said this way, in fact this PEP reopens the question > of splitting the stdlib. It would be helpful to have some references > into the PEP about previous discussions on this topic. Recent > discussion: > > "Amber Brown: Batteries Included, But They're Leaking " > http://pyfound.blogspot.com/2019/05/amber-brown-batteries-included-but.html > > By the way, even if it's not directly related, the PEP doesn't address > one of the common question: be able to override a stdlib module with a > more recent version. For example, if the Python provided by your OS > contains a bug in asyncio, you are stuck and cannot easily upgrade it > to get a new version without the fix. As you said, it's not directly related. These topics are not within scope for the PEP. Please open a new thread. I like to keep this thread on topic. > For me, "getopt" is a dead battery. The PEP says "Although users are > encouraged to use argparse instead, the getopt module is still widely > used." > > Well, that makes sense. But so, what is the metric to decide if a > module is "widely used" or not? > > Should we keep a module at soon as at least one user says "I'm using > it!". In other replies, users asked to keep spwd, fileinput, nntplib, > etc. getopt is very short, has no maintainance overhead, and useful for quick hacks. fileinput stays as well for the same reason. > Sorry, I mostly opened questions. I don't have answers. I just wanted > to share that this PEP scares me a little bit. At least, the current > version of the PEP. The PEP will be implemented over a course of two releases and three years. It gives us enough time to address concern or change our minds. Christian From nd at perlig.de Tue May 21 08:31:02 2019 From: nd at perlig.de (=?ISO-8859-1?Q?Andr=E9?= Malo) Date: Tue, 21 May 2019 14:31:02 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <2215566.ODhDVceCri@finnegan> Message-ID: <2640652.eraKMVzp3W@finnegan> On Dienstag, 21. Mai 2019 13:46:34 CEST Christian Heimes wrote: > On 21/05/2019 13.08, Andr? Malo wrote: > > On Montag, 20. Mai 2019 23:27:49 CEST Antoine Pitrou wrote: > >> NNTP is still quite used (often through GMane, but probably not only) > >> so > >> I'd question the removal of nntplib. > >> > >> cgitb used to be used by some Web frameworks in order to format > >> exceptions. Perhaps one should check if that's still the case. > > > > I concur with both of those. > > There's software in production using both. (It doesn't mean it's on pypi > > or even free software). > > There is always somebody who uses a feature. This argument blocks any > innovation or cleanup. Victor just reminded me of https://xkcd.com/1172/ > . > > * The removed modules will be available through PyPI. > * You don't have to start to worry until Python 3.10 is released in over 3 > years from now. * The modules are fully supported in Python 3.8 and 3.9. > Python 3.9 will reach EOL late 2026 or early 2027. Correct. However that's a valid argument for the whole stdlib. I agree with Victor (in the other branch), that we should call the PEP how it's meant. It effectively boils down to: "We don't want to maintain these modules anymore, volunteers step forward within the next 3 years". It would definitely draw a clear line and cut short a lot of discussions (like this one). And it would be perfectly fine, for me at least. nd -- package Hacker::Perl::Another::Just;print qq~@{[reverse split/::/ =>__PACKAGE__]}~; # Andr? Malo # http://www.perlig.de # From christian at python.org Tue May 21 08:37:40 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 14:37:40 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> On 21/05/2019 14.06, Anders Munch wrote: > Fra: Python-Dev [mailto:python-dev-bounces+ajm=flonidan.dk at python.org] P? vegne af Christian Heimes >> * The removed modules will be available through PyPI. > > Will they? That's not the impression I got from the PEP. It's all open source. It's up to the Python community to adopt packages and provide them on PyPI. Python core will not maintain and distribute the packages. I'll merely provide a repository with packages to help kick-starting the process. Christian From steve at holdenweb.com Tue May 21 08:48:06 2019 From: steve at holdenweb.com (Steve Holden) Date: Tue, 21 May 2019 13:48:06 +0100 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> References: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> Message-ID: That seems entirely reasonable. I wonder if the larger community could somehow form an organization (the Dead Parrot SIG?) that would at least curate and monitor efforts to ensure their continued utility? On Tue, May 21, 2019 at 1:40 PM Christian Heimes wrote: > On 21/05/2019 14.06, Anders Munch wrote: > > Fra: Python-Dev [mailto:python-dev-bounces+ajm=flonidan.dk at python.org] > P? vegne af Christian Heimes > >> * The removed modules will be available through PyPI. > > > > Will they? That's not the impression I got from the PEP. > > It's all open source. It's up to the Python community to adopt packages > and provide them on PyPI. > > Python core will not maintain and distribute the packages. I'll merely > provide a repository with packages to help kick-starting the process. > > Christian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ned at nedbatchelder.com Tue May 21 08:54:27 2019 From: ned at nedbatchelder.com (Ned Batchelder) Date: Tue, 21 May 2019 08:54:27 -0400 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> References: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> Message-ID: <7ac61718-1235-cd8e-57e4-0f4abda25993@nedbatchelder.com> On 5/21/19 8:37 AM, Christian Heimes wrote: > On 21/05/2019 14.06, Anders Munch wrote: >> Fra: Python-Dev [mailto:python-dev-bounces+ajm=flonidan.dk at python.org] P? vegne af Christian Heimes >>> * The removed modules will be available through PyPI. >> Will they? That's not the impression I got from the PEP. > It's all open source. It's up to the Python community to adopt packages and provide them on PyPI. > > Python core will not maintain and distribute the packages. I'll merely provide a repository with packages to help kick-starting the process. This statement should be in the PEP in some form. For example: "The modules could be made available on PyPI. The Python core team will not publish or maintain the packages." --Ned. From p.f.moore at gmail.com Tue May 21 09:00:57 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 21 May 2019 14:00:57 +0100 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> Message-ID: On Tue, 21 May 2019 at 13:50, Steve Holden wrote: > > That seems entirely reasonable. I wonder if the larger community could somehow form an organization (the Dead Parrot SIG?) that would at least curate and monitor efforts to ensure their continued utility? I have no idea whether there is enough community support to make this work, but regardless, I strongly support a SIG by that name. It may actually be even *more* amusing if it doesn't have enough members to make it viable :-) Paul From steve at pearwood.info Tue May 21 09:01:13 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 21 May 2019 23:01:13 +1000 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: <20190521130111.GJ4221@ando.pearwood.info> The PEP title is prejudicial and inaccurate. These aren't "dead batteries", these are *working batteries* that you want to remove. If you want a fair and open debate on this, please change the title to something less prejudicial. If this were my PEP, I'd call it "Removing unloved batteries from the standard library". On Mon, May 20, 2019 at 10:15:22PM +0200, Christian Heimes wrote: > Times have changed. The introduction of the cheese shop (PyPI), setuptools, > and later pip, it became simple and straight forward to download and install > packages. Nowadays Python has a rich and vibrant ecosystem of third party > packages. It's pretty much standard to either install packages from PyPI or > use one of the many Python or Linux distributions. Christian, I'm glad that you are privileged enough to find it simple and straight forward to download and install, but for many Python users, it is not so simple or straight forward. Many Python users don't have the privilege of being able to install arbitrary, unvetted packages from PyPI. They get to use only packages from approved vendors, including the stdlib, what they write themselves, and nothing else. Please don't dismiss this part of the Python community just because they don't typically hang around in the same forums we do. I've worked with organisations where downloading and installing software from the internet was grounds for instant dismissal. I've worked with organisations with regulatory requirements to do due-dilegance on their entire software stack, and getting permission to install an unapproved library could take 3-6 months elapsed time and dozens of person-hours, including a full review of the licencing and copyright by lawyers. I've also worked with kids using school computers who don't have either the legal permission or the knowledge to use pip install. Sometimes their school administrators are ... how shall I put this kindly? ... over zealous in their efforts to protect the students from malware and spyware (apart from the school's own spyware, of course...) and rather lacking in their understanding of the difference between piracy and Open Source software. Getting Python installed by the school admiinistrator is one thing, but allowing the kids to run pip and install software themselves is unthinkable. And remember, in some juristictions, installing software from the internet can put you in breach of some draconian laws. At the very least, kids may face expulsion. Despite all the advances in packaging and installation, many people still do have trouble installing third-party software. Internet forums are full of people asking for help troubleshooting pip install. "Simple and straight forward" it might be for an expert, but many Python users are not experts. The Internet is full of difficult, conflicting and/or bad advice about installing software ("just use sudo pip install"). Let's look at this advice from Red Hat: If you want to use third-party packages, create a virtual environment using python3 -m venv --system-site-packages myenv (or for Python 2, install python2-virtualenv and run python2 -m virtualenv --system-site-packages myenv). Then, activate the environment using source myenv/bin/activate, and install packages into it using pip install. The packages will then be available as long as the environment is activated. https://developers.redhat.com/blog/2018/11/14/python-in-rhel-8/ That's some serious barrier to entry right there. We're going from: batteries included to what's a virtual enviroment? what's activation? why isn't pip working for me? I know that saying anything against pip and virtual environments is heresy, but honestly, "just install it from PyPI" is not friendly to beginners or those who just want something that works without a load of extra complexity. -- Steven From christian at python.org Tue May 21 09:24:26 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 15:24:26 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190521130111.GJ4221@ando.pearwood.info> References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: On 21/05/2019 15.01, Steven D'Aprano wrote: > Christian, I'm glad that you are privileged enough to find it simple and > straight forward to download and install, but for many Python users, it > is not so simple or straight forward. > > Many Python users don't have the privilege of being able to install > arbitrary, unvetted packages from PyPI. They get to use only packages > from approved vendors, including the stdlib, what they write themselves, > and nothing else. Please don't dismiss this part of the Python community > just because they don't typically hang around in the same forums we do. > > I've worked with organisations where downloading and installing software > from the internet was grounds for instant dismissal. I've worked with > organisations with regulatory requirements to do due-dilegance on their > entire software stack, and getting permission to install an unapproved > library could take 3-6 months elapsed time and dozens of person-hours, > including a full review of the licencing and copyright by lawyers. > > I've also worked with kids using school computers who don't have either > the legal permission or the knowledge to use pip install. > > Sometimes their school administrators are ... how shall I put this > kindly? ... over zealous in their efforts to protect the students from > malware and spyware (apart from the school's own spyware, of course...) > and rather lacking in their understanding of the difference between > piracy and Open Source software. Getting Python installed by the school > admiinistrator is one thing, but allowing the kids to run pip and > install software themselves is unthinkable. > > And remember, in some juristictions, installing software from the > internet can put you in breach of some draconian laws. At the very > least, kids may face expulsion. This argument has bring brought up several times. For that reason the PEP lists mostly modules that deal with historic and irrelevant data formats. The wave module stays because it is useful in education contexts. The use case is explicitly mentioned in the latest version of the PEP. Do you have a concrete and tangible case example, in which a deprecated module is absolutely necessary, a user gets permission to update Python, but does not get permission to get a PyPI package installed? You could argue that cgi, spwd, and crypt are not history data formats. My counter argument: *If* you have to work in a severely restricted environment, *then* you are surely not allowed to run your own CGI web server or access the system password database. From p.f.moore at gmail.com Tue May 21 09:26:11 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 21 May 2019 14:26:11 +0100 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190521130111.GJ4221@ando.pearwood.info> References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: On Tue, 21 May 2019 at 14:03, Steven D'Aprano wrote: > I know that saying anything against pip and virtual environments is > heresy, but honestly, "just install it from PyPI" is not friendly to > beginners or those who just want something that works without a load of > extra complexity. Speaking as a pip developer, I 100% support this comment. For a major segment of Python's user base, if something isn't in the stdlib (or their preferred distribution - Anaconda or a Linux distro, for example) then it's a significant step upward in complexity. Having said this, I'm not as sure I agree with the rest of Steven's posting. I think that in general, this PEP is sensible and strikes a good balance. It's all very well saying "the Python core devs aren't willing to support these libraries any more" - but being quite that blunt invites a backlash. The PEP demonstrates that we're doing due dilligence over the process, and even if some people disagree with the decisions made, the arguments have been made in public. "You can get equivalents off PyPI" is only a very minor part of the argument here, the main thrust being "... and in general we don't think many people need to". It's a judgement call, and it needs review, but "rarely used" captures the sense well enough (given that we want a short phrase, not an explanatory paragraph...) Regarding the title, It strikes me as fine, it's a slightly light hearted play on the "batteries included" idea, and even if the batteries in question aren't entirely dead, they are definitely pinin' for the fjords. Discussions like this need a bit of levity to keep us all grounded, IMO. Paul From vstinner at redhat.com Tue May 21 09:54:58 2019 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 21 May 2019 15:54:58 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: IMHO we should simply acknowledge this fact by mentioning it in the PEP. We are aware that "pip install" is not always a valid option, but we decided anyway to deprecate/remove modules because <...>. Victor Le mar. 21 mai 2019 ? 15:29, Paul Moore a ?crit : > > On Tue, 21 May 2019 at 14:03, Steven D'Aprano wrote: > > > I know that saying anything against pip and virtual environments is > > heresy, but honestly, "just install it from PyPI" is not friendly to > > beginners or those who just want something that works without a load of > > extra complexity. > > Speaking as a pip developer, I 100% support this comment. For a major > segment of Python's user base, if something isn't in the stdlib (or > their preferred distribution - Anaconda or a Linux distro, for > example) then it's a significant step upward in complexity. > > Having said this, I'm not as sure I agree with the rest of Steven's > posting. I think that in general, this PEP is sensible and strikes a > good balance. It's all very well saying "the Python core devs aren't > willing to support these libraries any more" - but being quite that > blunt invites a backlash. The PEP demonstrates that we're doing due > dilligence over the process, and even if some people disagree with the > decisions made, the arguments have been made in public. "You can get > equivalents off PyPI" is only a very minor part of the argument here, > the main thrust being "... and in general we don't think many people > need to". It's a judgement call, and it needs review, but "rarely > used" captures the sense well enough (given that we want a short > phrase, not an explanatory paragraph...) > > Regarding the title, It strikes me as fine, it's a slightly light > hearted play on the "batteries included" idea, and even if the > batteries in question aren't entirely dead, they are definitely pinin' > for the fjords. Discussions like this need a bit of levity to keep us > all grounded, IMO. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com -- Night gathers, and now my watch begins. It shall not end until my death. From christian at python.org Tue May 21 10:12:42 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 16:12:42 +0200 Subject: [Python-Dev] PEP 594: update 1 Message-ID: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Hi, I have updated the PEP with feedback from discussions. The highlights are: * Deprecate parser module * Keep fileinput module * Elaborate why crypt and spwd are dangerous and bad * Improve sections for cgitb, colorsys, nntplib, and smtpd modules * The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable substitutions. * Mention that socketserver is going to stay for http.server and xmlrpc.server * The future maintenance section now states that the deprecated modules may be adopted by Python community members. https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff-ae358c21fa7968ee3b6c64479e051574 I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback. Christian ------------------------------------------------------------------- PEP: 594 Title: Removing dead batteries from the standard library Author: Christian Heimes Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 20-May-2019 Post-History: 21-May-2019 Abstract ======== This PEP proposed a list of standard library modules to be removed from the standard library. The modules are mostly historic data formats and APIs that have been superseded a long time ago, e.g. Mac OS 9 and Commodore. Rationale ========= Back in the early days of Python, the interpreter came with a large set of useful modules. This was often refrained to as "batteries included" philosophy and was one of the corner stones to Python's success story. Users didn't have to figure out how to download and install separate packages in order to write a simple web server or parse email. Times have changed. The introduction of the cheese shop (PyPI), setuptools, and later pip, it became simple and straight forward to download and install packages. Nowadays Python has a rich and vibrant ecosystem of third party packages. It's pretty much standard to either install packages from PyPI or use one of the many Python or Linux distributions. On the other hand, Python's standard library is piling up cruft, unnecessary duplication of functionality, and dispensable features. This is undesirable for several reasons. * Any additional module increases the maintenance cost for the Python core development team. The team has limited resources, reduced maintenance cost frees development time for other improvements. * Modules in the standard library are generally favored and seen as the de-facto solution for a problem. A majority of users only pick 3rd party modules to replace a stdlib module, when they have a compelling reason, e.g. lxml instead of `xml`. The removal of an unmaintained stdlib module increases the chances of a community contributed module to become widely used. * A lean and mean standard library benefits platforms with limited resources like devices with just a few hundred kilobyte of storage (e.g. BBC Micro:bit). Python on mobile platforms like BeeWare or WebAssembly (e.g. pyodide) also benefit from reduced download size. The modules in the PEP have been selected for deprecation because their removal is either least controversial or most beneficial. For example least controversial are 30 years old multimedia formats like ``sunau`` audio format, which was used on SPARC and NeXT workstations in the late 1980ties. The ``crypt`` module has fundamental flaws that are better solved outside the standard library. This PEP also designates some modules as not scheduled for removal. Some modules have been deprecated for several releases or seem unnecessary at first glance. However it is beneficial to keep the modules in the standard library, mostly for environments where installing a package from PyPI is not an option. This can be cooperate environments or class rooms where external code is not permitted without legal approval. * The usage of FTP is declining, but some files are still provided over the FTP protocol or hosters offer FTP to upload content. Therefore ``ftplib`` is going to stay. * The ``optparse`` and ``getopt`` module are widely used. They are mature modules with very low maintenance overhead. * According to David Beazley [5]_ the ``wave`` module is easy to teach to kids and can make crazy sounds. Making a computer generate crazy sounds is powerful and highly motivating exercise for a 9yo aspiring developer. It's a fun battery to keep. Deprecation schedule ==================== 3.8 --- This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released a few months before Python 2.7 will reach its end of lifetime. We expect that Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and 2020. To reduce churn and to allow a smooth transition from Python 2, Python 3.8 will neither raise `DeprecationWarning` nor remove any modules that have been scheduled for removal. Instead deprecated modules will just be *documented* as deprecated. Optionally modules may emit a `PendingDeprecationWarning`. All deprecated modules will also undergo a feature freeze. No additional features should be added. Bug should still be fixed. 3.9 --- Starting with Python 3.9, deprecated modules will start issuing `DeprecationWarning`. The `parser`_ module is removed and potentially replaced with a new module. All other deprecated modules are fully supported and will receive security updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will be released about 18 months after 3.8.0 (April 2021?) and most likely be supported for 5 years after the release. The estimated EOL of Python 3.9 is in 2026. 3.10 ---- In 3.10 all deprecated modules will be removed from the CPython repository together with tests, documentation, and autoconf rules. PEP acceptance process ====================== 3.8.0b1 is scheduled to be release shortly after the PEP is officially submitted. Since it's improbable that the PEP will pass all stages of the PEP process in time, I propose a two step acceptance process that is analogous Python's two release deprecation process. The first *provisionally accepted* phase targets Python 3.8.0b1. In the first phase no code is changes or removed. Modules are only documented as deprecated. The only exception is the `parser`_ module. It has been documented as deprecated since Python 2.5 and is scheduled for removal for 3.9 to make place for a more advanced parser. The final decision, which modules will be removed and how the removed code is preserved, can be delayed for another year. Deprecated modules ================== The modules are grouped as data encoding, multimedia, network, OS interface, and misc modules. The majority of modules are for old data formats or old APIs. Some others are rarely useful and have better replacements on PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with audio processing. .. csv-table:: Table 1: Proposed modules deprecations :header: "Module", "Deprecated in", "To be removed", "Replacement" :widths: 1, 1, 1, 2 aifc,3.8,3.10,\- asynchat,3.8,3.10,asyncio asyncore,3.8,3.10,asyncio audioop,3.8,3.10,\- binhex,3.8,3.10,\- cgi,3.8,3.10,\- cgitb,3.8,3.10,\- chunk,3.8,3.10,\- colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" fileinput,\-,**keep**,argparse formatter,3.4,3.10,\- fpectl,**3.7**,**3.7**,\- getopt,**3.2**,**keep**,"argparse, optparse" imghdr,3.8,3.10,"filetype, puremagic, python-magic" imp,**3.4**,3.10,importlib lib2to3,\-,**keep**, macpath,**3.7**,**3.8**,\- msilib,3.8,3.10,\- nntplib,3.8,3.10,\- nis,3.8,3.10,\- optparse,\-,**keep**,argparse ossaudiodev,3.8,3.10,\- parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" pipes,3.8,3.10,subprocess smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd sndhdr,3.8,3.10,"filetype, puremagic, python-magic" spwd,3.8,3.10,"python-pam, simplepam" sunau,3.8,3.10,\- uu,3.8,3.10,\- wave,\-,**keep**, xdrlib,3.8,3.10,\- Data encoding modules --------------------- binhex ~~~~~~ The `binhex `_ module encodes and decodes Apple Macintosh binhex4 data. It was originally developed for TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to encode binary email attachments. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** uu ~~ The `uu `_ module provides uuencode format, an old binary encoding format for email from 1980. The uu format has been replaced by MIME. The uu codec is provided by the binascii module. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** xdrlib ~~~~~~ The `xdrlib `_ module supports the Sun External Data Representation Standard. XDR is an old binary serialization format from 1987. These days it's rarely used outside specialized domains like NFS. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Multimedia modules ------------------ aifc ~~~~ The `aifc `_ module provides support for reading and writing AIFF and AIFF-C files. The Audio Interchange File Format is an old audio format from 1988 based on Amiga IFF. It was most commonly used on the Apple Macintosh. These days only few specialized application use AIFF. Module type pure Python (depends on `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute **none** audioop ~~~~~~~ The `audioop `_ module contains helper functions to manipulate raw audio data and adaptive differential pulse-code modulated audio data. The module is implemented in C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ module depend on `audioop`_ for some operations. The byteswap operation in the `wave`_ module can be substituted with little work. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** colorsys ~~~~~~~~ The `colorsys `_ module defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate systems. The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and advanced features. The Pillow library is better suited to transform images between color systems. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `colormath `_, `colour `_ `colorspacious `_, `Pillow `_ chunk ~~~~~ The `chunk `_ module provides support for reading and writing Electronic Arts' Interchange File Format. IFF is an old audio file format originally introduced for Commodore and Amiga. The format is no longer relevant. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** imghdr ~~~~~~ The `imghdr `_ module is a simple tool to guess the image file format from the first 32 bytes of a file or buffer. It supports only a limited amount of formats and neither returns resolution nor color depth. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic `_, `filetype `_, `python-magic `_ ossaudiodev ~~~~~~~~~~~ The `ossaudiodev `_ module provides support for Open Sound System, an interface to sound playback and capture devices. OSS was initially free software, but later support for newer sound devices and improvements were proprietary. Linux community abandoned OSS in favor of ALSA [1]_. Some operation systems like OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** sndhdr ~~~~~~ The `sndhdr `_ module is similar to the `imghdr`_ module but for audio formats. It guesses file format, channels, frame rate, and sample widths from the first 512 bytes of a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and other ancient formats. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute `puremagic `_, `filetype `_, `python-magic `_ sunau ~~~~~ The `sunau `_ module provides support for Sun AU sound format. It's yet another old, obsolete file format. Module type pure Python (depends on `audioop`_ C extension for some operations) Deprecated in 3.8 To be removed in 3.10 Substitute **none** Networking modules ------------------ asynchat ~~~~~~~~ The `asynchat `_ module is build on top of `asyncore`_ and has been deprecated since Python 3.6. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio asyncore ~~~~~~~~ The `asyncore `_ module was the first module for asynchronous socket service clients and servers. It has been replaced by asyncio and is deprecated since Python 3.6. The ``asyncore`` module is also used in stdlib tests. The tests for ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly based on ``asyncore``. These tests must be updated to use asyncio or threading. Module type pure Python Deprecated in 3.6 Removed in 3.10 Substitute asyncio cgi ~~~ The `cgi `_ module is a support module for Common Gateway Interface (CGI) scripts. CGI is deemed as inefficient because every incoming request is handled in a new process. PEP 206 considers the module as *designed poorly and are now near-impossible to fix*. Several people proposed to either keep the cgi module for features like `cgi.parse_qs()` or move `cgi.escape()` to a different module. The functions `cgi.parse_qs` and `cgi.parse_qsl` have been deprecated for a while and are actually aliases for `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The function `cgi.quote` has been deprecated in favor of `html.quote` with secure default values. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** cgitb ~~~~~ The `cgitb `_ module is a helper for the cgi module for configurable tracebacks. The ``cgitb`` module is not used by any major Python web framework (Django, Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an optional debugging middleware. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** smtpd ~~~~~ The `smtpd `_ module provides a simple implementation of a SMTP mail server. The module documentation marks the module as deprecated and recommends ``aiosmtpd`` instead. The deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. Module type pure Python Deprecated in **3.7** To be removed in 3.10 Substitute aiosmtpd nntplib ~~~~~~~ The `nntplib `_ module implements the client side of the Network News Transfer Protocol (nntp). News groups used to be a dominant platform for online discussions. Over the last two decades, news has been slowly but steadily replaced with mailing lists and web-based discussion platforms. Twisted is also `planning `_ to deprecate NNTP support and `pynnt `_ hasn't seen any activity since 2014. This is a good indicator that the public interest in NNTP support is declining. The ``nntplib`` tests have been the cause of additional work in the recent past. Python only contains client side of NNTP. The tests connect to external news server. The servers are sometimes unavailble, too slow, or do not work correctly over IPv6. The situation causes flaky test runs on buildbots. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute **none** Operating system interface -------------------------- crypt ~~~~~ The `crypt `_ module implements password hashing based on ``crypt(3)`` function from ``libcrypt`` or ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor quality and insecure. Users are discouraged to use them. * The module is not available on Windows. Cross-platform application need an alternative implementation any way. * Only DES encryption is guarenteed to be available. DES has an extremely limited key space of 2**56. * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only algorithm that is still secure. However it's in glibc and therefore not commonly available on Linux. * Depending on the platform, the ``crypt`` module is not thread safe. Only implementations with ``crypt_r(3)`` are thread safe. * The module was never useful to interact with system user and password databases. On BSD, macOS, and Linux, all user authentication and password modification operations must go through PAM (pluggable authentication module), see `spwd`_ deprecation. Module type C extension + Python module Deprecated in 3.8 To be removed in 3.10 Substitute `bcrypt `_, `passlib `_, `argon2cffi `_, hashlib module (PBKDF2, scrypt) macpath ~~~~~~~ The `macpath `_ module provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer supported Module type pure Python Deprecated in 3.7 Removed in 3.8 Substitute **none** nis ~~~ The `nis `_ module provides NIS/YP support. Network Information Service / Yellow Pages is an old and deprecated directory service protocol developed by Sun Microsystems. It's designed successor NIS+ from 1992 never took off. For a long time, libc's Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful and more secure replacement of NIS. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute **none** spwd ~~~~ The `spwd `_ module provides direct access to Unix shadow password database using non-standard APIs. In general it's a bad idea to use the spwd. The spwd circumvents system security policies, it does not use the PAM stack, and is only compatible with local user accounts, because it ignores NSS. The use of the ``spwd`` module for access control must be consider a *security bug*, as it bypasses PAM's access control. Further more the ``spwd`` module uses the `shadow(3) `_ APIs. Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This is dangerous and even forbidden for confined services on systems with a security engine like SELinux or AppArmor. Module type C extension Deprecated in 3.8 To be removed in 3.10 Substitute `python-pam `_, `simpleplam `_ Misc modules ------------ formatter ~~~~~~~~~ The `formatter `_ module is an old text formatting module which has been deprecated since Python 3.4. Module type pure Python Deprecated in 3.4 To be removed in 3.10 Substitute *n/a* imp ~~~ The `imp `_ module is the predecessor of the `importlib `_ module. Most functions have been deprecated since Python 3.3 and the module since Python 3.4. Module type C extension Deprecated in 3.4 To be removed in 3.10 Substitute importlib msilib ~~~~~~ The `msilib `_ package is a Windows-only package. It supports the creation of Microsoft Installers (MSI). The package also exposes additional APIs to create cabinet files (CAB). The module is used to facilitate distutils to create MSI installers with ``bdist_msi`` command. In the past it was used to create CPython's official Windows installer, too. Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) as new deployment model [3]_. Module type C extension + Python code Deprecated in 3.8 To be removed in 3.10 Substitute **none** parser ~~~~~~ The `parser `_ module provides an interface to Python?s internal parser and byte-code compiler. The stdlib has superior ways to interact with the parse tree. From Python 2.5 onward, it's much more convenient to cut in at the Abstract Syntax Tree (AST) generation and compilation stage. The ``parser`` module causes additional work. It's C code that must be kept in sync with any change to Python's grammar and internal parser. Pablo wants to remove the parser module and promote lib2to3's pgen2 instead [6]_. Most importantly the presence of the ``parser`` module makes it harder to switch to something more powerful than a LL(1) parser [7]_. Since the ``parser`` module is documented as deprecated since Python 2.5 and a new parsing technology is planned for 3.9, the ``parser`` module is scheduled for removal in 3.9. Module type C extension Deprecated in 3.8, documented as deprecated since **2.5** To be removed in **3.9** Substitute ast, lib2to3.pgen2 pipes ~~~~~ The `pipes `_ module provides helpers to pipe the input of one command into the output of another command. The module is built on top of ``os.popen``. Users are encouraged to use the subprocess module instead. Module type pure Python Deprecated in 3.8 To be removed in 3.10 Substitute subprocess module Removed modules =============== fpectl ------ The `fpectl `_ module was never built by default, its usage was discouraged and considered dangerous. It also required a configure flag that caused an ABI incompatibility. The module was removed in 3.7 by Nathaniel J. Smith in `bpo-29137 `_. Module type C extension + CAPI Deprecated in 3.7 Removed in 3.7 Substitute **none** Modules to keep =============== Some modules were originally proposed for deprecation. fileinput --------- The `fileinput `_ module implements a helpers to iterate over a list of files from ``sys.argv``. The module predates the optparser and argparser module. The same functionality can be implemented with the argparser module. Several core developers expressed their interest to keep the module in the standard library, as it is handy for quick scripts. Module type pure Python lib2to3 ------- The `lib2to3 `_ package provides the ``2to3`` command to transpile Python 2 code to Python 3 code. The package is useful for other tasks besides porting code from Python 2 to 3. For example `black`_ uses it for code reformatting. Module type pure Python getopt ------ The `getopt `_ module mimics C's getopt() option parser. Although users are encouraged to use argparse instead, the getopt module is still widely used. The module is small, simple, and handy for C developers to write simple Python scripts. Module type pure Python Substitute argparse optparse -------- The `optparse `_ module is the predecessor of the argparse module. Although it has been deprecated for many years, it's still too widely used to remove it. Module type pure Python Deprecated in 3.2 Substitute argparse wave ---- The `wave `_ module provides support for the WAV sound format. The module is not deprecated, because The WAV format is still relevant these days. The ``wave`` module is also used in education, e.g. to show kids how to make noise with a computer. The module uses one simple function from the `audioop`_ module to perform byte swapping between little and big endian formats. Before 24 bit WAV support was added, byte swap used to be implemented with the ``array`` module. To remove ``wave``'s dependency on the ``audioop``, the byte swap function could be either be moved to another module (e.g. ``operator``) or the ``array`` module could gain support for 24 bit (3 byte) arrays. Module type pure Python (depends on *byteswap* from `audioop`_ C extension) Deprecated in 3.8 To be removed in 3.10 Substitute *n/a* Future maintenance of removed modules ===================================== The main goal of the PEP is to reduce the burden and workload on the Python core developer team. Therefore removed modules will not be maintained by the core team as separate PyPI packages. However the removed code, tests and documentation may be moved into a new git repository, so community members have a place from which they can pick up and fork code. A first draft of a `legacylib `_ repository is available on my private Github account. The modules could be made available on PyPI. The Python core team will not publish or maintain the packages. It is my hope that members of the Python community will adopt, maintain, and perhaps improve the deprecated modules. It's my hope that some of the deprecated modules will be picked up and adopted by users that actually care about them. For example ``colorsys`` and ``imghdr`` are useful modules, but have limited feature set. A fork of ``imghdr`` can add new features and support for more image formats, without being constrained by Python's release cycle. Most of the modules are in pure Python and can be easily packaged. Some depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ does not depend on any external libraries, it can be shipped in as binary wheels with some effort. Other C modules can be replaced with ctypes or cffi. For example I created `legacycrypt `_ with ``_crypt`` extension reimplemented with a few lines of ctypes code. Discussions =========== * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. * Berker Peksag proposed to deprecate and removed *msilib*. * Brett Cannon recommended to delay active deprecation warnings and removal of modules like *imp* until Python 3.10. Version 3.8 will be released shortly before Python 2 reaches end of lifetime. A delay reduced churn for users that migrate from Python 2 to 3.8. * Brett also came up with the idea to keep lib2to3. The package is useful for other purposes, e.g. `black `_ uses it to reformat Python code. * At one point, distutils was mentioned in the same sentence as this PEP. To avoid lengthy discussion and delay of the PEP, I decided against dealing with distutils. Deprecation of the distutils package will be handled by another PEP. * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) convinced me to keep the `wave`_ module. [4]_ * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ * Andrew Svetlov mentioned the ``socketserver`` module is questionable. However it's used to implement ``http.server`` and ``xmlrpc.server``. The stdlib doesn't have a replacement for the servers, yet. Update history ============== Update 1 -------- * Deprecate `parser`_ module * Keep `fileinput`_ module * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now list suitable substitutions. * Mention that ``socketserver`` is going to stay for ``http.server`` and ``xmlrpc.server`` * The future maintenance section now states that the deprecated modules may be adopted by Python community members. References ========== .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free .. [2] https://man.openbsd.org/ossaudio .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 .. [5] https://twitter.com/dabeaz/status/1130278844479545351 .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html .. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From christian at python.org Tue May 21 10:16:39 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 16:16:39 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: <46d841bc-6fa1-16e6-a13d-dbc3cb63d527@python.org> On 21/05/2019 15.54, Victor Stinner wrote: > IMHO we should simply acknowledge this fact by mentioning it in the > PEP. We are aware that "pip install" is not always a valid option, but > we decided anyway to deprecate/remove modules because <...>. I like the idea. Could you create an issue or PR, please? Christian From ncoghlan at gmail.com Tue May 21 10:30:46 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 May 2019 00:30:46 +1000 Subject: [Python-Dev] deprecation of abstractstaticmethod and abstractclassmethod In-Reply-To: <571659b9-d63e-4bf9-ba06-56e7555b9aa8@python.org> References: <51c99452-91d3-a479-2666-1af554c433e2@stoneleaf.us> <571659b9-d63e-4bf9-ba06-56e7555b9aa8@python.org> Message-ID: On Thu, 16 May 2019 at 06:03, Steve Dower wrote: > > On 15May2019 1248, Nathaniel Smith wrote: > > I don't care about the deprecation either way. But can we fix the > > individual decorators so both orders work? Even if it requires a special > > case in the code, it seems worthwhile to remove a subtle user-visible > > footgun. > > The classmethod and staticmethod objects already have a getter for > __isabstractmethod__ that forwards to the underlying object, so they > should just need a setter as well, right? I believe the problem with the reverse direction relates to abstractmethod correctly forwarding __get__ to the underlying descriptor (or not), rather than having anything to do with the `__isabstractmethod__` property forwarding. However, I *don't* recall if we ever actually tried to make the reverse direction work - it may be that there's a relatively shallow enhancement that would let `@abstractmethod` wrap arbitrary decorators correctly. > I also have no strong opinion about deprecation. But fixing this seems > pretty straightforward and may be a good task for someone's first C > contribution? If the reverse order can be made to work, that would be good, but if that proves intractable, Ethan's suggestion of reversing the documented deprecation makes sense. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From encukou at gmail.com Tue May 21 10:39:44 2019 From: encukou at gmail.com (Petr Viktorin) Date: Tue, 21 May 2019 16:39:44 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <8ade5b62-75d1-589e-3b7b-0678f5e239ec@python.org> References: <8ade5b62-75d1-589e-3b7b-0678f5e239ec@python.org> Message-ID: <0790e769-ff11-ca99-7252-662d9fedeceb@gmail.com> On 5/21/19 12:06 PM, Christian Heimes wrote: > On 21/05/2019 11.49, Nathaniel Smith wrote: >> On Tue, May 21, 2019 at 2:40 AM Walter D?rwald wrote: >>> >>> On 20 May 2019, at 22:15, Christian Heimes wrote: >>> >>>> Hi, >>>> >>>> here is the first version of my PEP 594 to deprecate and eventually >>>> remove modules from the standard library. The PEP started last year >>>> with talk during the Python Language Summit 2018, >>>> https://lwn.net/Articles/755229/. >>>> >>>> [...] >>>> >>>> colorsys >>>> ~~~~~~~~ >>>> >>>> The `colorsys `_ >>>> module >>>> defines color conversion functions between RGB, YIQ, HSL, and HSV >>>> coordinate >>>> systems. The Pillow library provides much faster conversation between >>>> color systems. >>>> >>>> Module type >>>> pure Python >>>> Deprecated in >>>> 3.8 >>>> To be removed in >>>> 3.10 >>>> Substitute >>>> `Pillow `_, >>>> `colorspacious `_ >>> >>> I'm using colorsys constantly as the basis for a tool that converts CSS >>> colors between different coordinate systems. I don't see how that could >>> be done via Pillow (which AFAICT only converts complete images). >>> RGB<->HSV<->HLS conversion seems to be not available (or not obvious) in >>> colorspacious. >> >> Correct, colorspacious doesn't support HSV or HLS. I suppose it would >> be trivial enough to add... >> >> The 'colour' package has them (along with everything else you can >> dream of): https://colour.readthedocs.io/en/latest/colour.models.html > > Nice catch, I just added https://python-colormath.readthedocs.io/en/latest/ to my update PR. I'll add colour to the list, too. > > (It didn't pop up on my radar because I wasn't checking for British spelling) > > Christian Please, don't remove colorsys. HSL and HSV aren't outdated, and AFAIK, the module is not a maintenance burden. The packages on PyPI offer more advanced models. Perhaps they could be linked in the documentation, if we want to endorse them. But just as we're not removing `array` in favor of `numpy`, we shouldn't remove colorsys either. Colorsys is not dead, it's just small. I assume one of the reasons colorspacious doesn't support HSV or HLS is that HSV/HLS tends to be used for quite different purposes than the more fancy color spaces. You'd use HSL/HSV for a simple color picker or a saturation shift on a web page, GUI or game (where you don't have *exact* colors anyway); things like sRGB or CIELab are for color management, photos, printing, etc. From skip.montanaro at gmail.com Tue May 21 10:43:42 2019 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 21 May 2019 09:43:42 -0500 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190521130111.GJ4221@ando.pearwood.info> References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: > If this were my PEP, I'd call it "Removing unloved batteries from the standard library". Or maybe, "Removing obsolete and (potentially) dangerous batteries from the standard library." I can certainly understand why either class of module would be removed. When bsddb185 was tossed out, I put it up on PyPI (https://pypi.org/project/bsddb185/) so the source wouldn't be lost. I never expected to actually need it, but I got pinged for the first time a few months ago. Someone needed help building it. So, no matter the reason for removal, I think it would be reasonable to toss all these modules into GitHub or PyPI. Someone will want them. Skip From guido at python.org Tue May 21 10:45:44 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 21 May 2019 07:45:44 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: Quick note on the pip discussion: if the likely remaining users of a module slated for deprecation are professionals maintaining some legacy code, pip is a fine solution. OTOH if the likely users are beginners, maybe pip is not great. In general I think it's fine to err on the side of caution -- as someone already said, we're only removing *dead* batteries, not *leaky* ones. If there's too much pushback for a particular module, maybe it's not dead yet. We can make another list in a few years. On Tue, May 21, 2019 at 6:28 AM Paul Moore wrote: > On Tue, 21 May 2019 at 14:03, Steven D'Aprano wrote: > > > I know that saying anything against pip and virtual environments is > > heresy, but honestly, "just install it from PyPI" is not friendly to > > beginners or those who just want something that works without a load of > > extra complexity. > > Speaking as a pip developer, I 100% support this comment. For a major > segment of Python's user base, if something isn't in the stdlib (or > their preferred distribution - Anaconda or a Linux distro, for > example) then it's a significant step upward in complexity. > > Having said this, I'm not as sure I agree with the rest of Steven's > posting. I think that in general, this PEP is sensible and strikes a > good balance. It's all very well saying "the Python core devs aren't > willing to support these libraries any more" - but being quite that > blunt invites a backlash. The PEP demonstrates that we're doing due > dilligence over the process, and even if some people disagree with the > decisions made, the arguments have been made in public. "You can get > equivalents off PyPI" is only a very minor part of the argument here, > the main thrust being "... and in general we don't think many people > need to". It's a judgement call, and it needs review, but "rarely > used" captures the sense well enough (given that we want a short > phrase, not an explanatory paragraph...) > > Regarding the title, It strikes me as fine, it's a slightly light > hearted play on the "batteries included" idea, and even if the > batteries in question aren't entirely dead, they are definitely pinin' > for the fjords. Discussions like this need a bit of levity to keep us > all grounded, IMO. > > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 21 10:46:14 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 21 May 2019 07:46:14 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <0790e769-ff11-ca99-7252-662d9fedeceb@gmail.com> References: <8ade5b62-75d1-589e-3b7b-0678f5e239ec@python.org> <0790e769-ff11-ca99-7252-662d9fedeceb@gmail.com> Message-ID: +1. Let's keep colorsys then. On Tue, May 21, 2019 at 7:41 AM Petr Viktorin wrote: > On 5/21/19 12:06 PM, Christian Heimes wrote: > > On 21/05/2019 11.49, Nathaniel Smith wrote: > >> On Tue, May 21, 2019 at 2:40 AM Walter D?rwald > wrote: > >>> > >>> On 20 May 2019, at 22:15, Christian Heimes wrote: > >>> > >>>> Hi, > >>>> > >>>> here is the first version of my PEP 594 to deprecate and eventually > >>>> remove modules from the standard library. The PEP started last year > >>>> with talk during the Python Language Summit 2018, > >>>> https://lwn.net/Articles/755229/. > >>>> > >>>> [...] > >>>> > >>>> colorsys > >>>> ~~~~~~~~ > >>>> > >>>> The `colorsys `_ > >>>> module > >>>> defines color conversion functions between RGB, YIQ, HSL, and HSV > >>>> coordinate > >>>> systems. The Pillow library provides much faster conversation between > >>>> color systems. > >>>> > >>>> Module type > >>>> pure Python > >>>> Deprecated in > >>>> 3.8 > >>>> To be removed in > >>>> 3.10 > >>>> Substitute > >>>> `Pillow `_, > >>>> `colorspacious `_ > >>> > >>> I'm using colorsys constantly as the basis for a tool that converts CSS > >>> colors between different coordinate systems. I don't see how that could > >>> be done via Pillow (which AFAICT only converts complete images). > >>> RGB<->HSV<->HLS conversion seems to be not available (or not obvious) > in > >>> colorspacious. > >> > >> Correct, colorspacious doesn't support HSV or HLS. I suppose it would > >> be trivial enough to add... > >> > >> The 'colour' package has them (along with everything else you can > >> dream of): https://colour.readthedocs.io/en/latest/colour.models.html > > > > Nice catch, I just added > https://python-colormath.readthedocs.io/en/latest/ to my update PR. I'll > add colour to the list, too. > > > > (It didn't pop up on my radar because I wasn't checking for British > spelling) > > > > Christian > > > Please, don't remove colorsys. HSL and HSV aren't outdated, and AFAIK, > the module is not a maintenance burden. > > The packages on PyPI offer more advanced models. Perhaps they could be > linked in the documentation, if we want to endorse them. But just as > we're not removing `array` in favor of `numpy`, we shouldn't remove > colorsys either. Colorsys is not dead, it's just small. > > I assume one of the reasons colorspacious doesn't support HSV or HLS is > that HSV/HLS tends to be used for quite different purposes than the more > fancy color spaces. You'd use HSL/HSV for a simple color picker or a > saturation shift on a web page, GUI or game (where you don't have > *exact* colors anyway); things like sRGB or CIELab are for color > management, photos, printing, etc. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 21 10:51:06 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 22 May 2019 00:51:06 +1000 Subject: [Python-Dev] PEP 558: Defined semantics for locals() Message-ID: Hi folks, After a couple of years hiatus, I finally got back to working on the reference implementation for PEP 558, my proposal to resolve some weird interactions between closures and trace functions in CPython by formally defining the expected semantics of the locals() builtin at function scope, and then ensuring that CPython adheres to those clarified semantics. The full text of the PEP is included below, and the rendered version is available at https://www.python.org/dev/peps/pep-0558/ The gist of the PEP is that: 1. The behaviour when no trace functions are installed and no frame introspection APIs are invoked is fine, so nothing should change in that regard (except that we elevate that behaviour from "the way CPython happens to work" to "the way the language and library reference says that Python implementations are supposed to work", and make sure CPython continues to behave that way even when a trace hook *is* installed) 2. If you get hold of a frame object in CPython (or another implementation that emulates the CPython frame API), whether via a trace hook or via a frame introspection API, then writing to the returned mapping will update the actual function locals and/or closure reference immediately, rather than relying on the FastToLocals/LocalsToFast APIs 3. The LocalsToFast C API changes to always produce RuntimeError (since there's no way for us to make it actually work correctly and consistently in the presence of closures, and the trace hook implementation won't need it any more given the write-through proxy behaviour on frame objects' "f_locals" attribute) The reference implementation still isn't quite done yet, but it's far enough along that I'm happy with the semantics and C API updates proposed in the current version of the PEP. Cheers, Nick. P.S. I'm away this weekend, so I expect the reference implementation to be done late next week, and I'd be submitting the PEP to Nathaniel for formal pronouncement at that point. However, I'm posting this thread now so that there's more time for discussion prior to the 3.8b1 deadline. ============== PEP: 558 Title: Defined semantics for locals() Author: Nick Coghlan BDFL-Delegate: Nathaniel J. Smith Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 2017-09-08 Python-Version: 3.8 Post-History: 2017-09-08, 2019-05-22 Abstract ======== The semantics of the ``locals()`` builtin have historically been underspecified and hence implementation dependent. This PEP proposes formally standardising on the behaviour of the CPython 3.6 reference implementation for most execution scopes, with some adjustments to the behaviour at function scope to make it more predictable and independent of the presence or absence of tracing functions. Rationale ========= While the precise semantics of the ``locals()`` builtin are nominally undefined, in practice, many Python programs depend on it behaving exactly as it behaves in CPython (at least when no tracing functions are installed). Other implementations such as PyPy are currently replicating that behaviour, up to and including replication of local variable mutation bugs that can arise when a trace hook is installed [1]_. While this PEP considers CPython's current behaviour when no trace hooks are installed to be acceptable (and largely desirable), it considers the current behaviour when trace hooks are installed to be problematic, as it causes bugs like [1]_ *without* even reliably enabling the desired functionality of allowing debuggers like ``pdb`` to mutate local variables [3]_. Proposal ======== The expected semantics of the ``locals()`` builtin change based on the current execution scope. For this purpose, the defined scopes of execution are: * module scope: top-level module code, as well as any other code executed using ``exec()`` or ``eval()`` with a single namespace * class scope: code in the body of a ``class`` statement, as well as any other code executed using ``exec()`` or ``eval()`` with separate local and global namespaces * function scope: code in the body of a ``def`` or ``async def`` statement We also allow interpreters to define two "modes" of execution, with only the first mode being considered part of the language specification itself: * regular operation: the way the interpreter behaves by default * tracing mode: the way the interpreter behaves when a trace hook has been registered in one or more threads via an implementation dependent mechanism like ``sys.settrace`` ([4]_) in CPython's ``sys`` module or ``PyEval_SetTrace`` ([5]_) in CPython's C API For regular operation, this PEP proposes elevating the current behaviour of the CPython reference implementation to become part of the language specification. For tracing mode, this PEP proposes changes to CPython's behaviour at function scope that bring the ``locals()`` builtin semantics closer to those used in regular operation, while also making the related frame API semantics clearer and easier for interactive debuggers to rely on. The proposed tracing mode changes also affect the semantics of frame object references obtained through other means, such as via a traceback, or via the ``sys._getframe()`` API. New ``locals()`` documentation ------------------------------ The heart of this proposal is to revise the documentation for the ``locals()`` builtin to read as follows: Return a dictionary representing the current local symbol table, with variable names as the keys, and their currently bound references as the values. This will always be the same dictionary for a given runtime execution frame. At module scope, as well as when using ``exec()`` or ``eval()`` with a single namespace, this function returns the same namespace as ``globals()``. At class scope, it returns the namespace that will be passed to the metaclass constructor. When using ``exec()`` or ``eval()`` with separate local and global namespaces, it returns the local namespace passed in to the function call. At function scope (including for generators and coroutines), it returns a dynamic snapshot of the function's local variables and any nonlocal cell references. In this case, changes made via the snapshot are *not* written back to the corresponding local variables or nonlocal cell references, and any such changes to the snapshot will be overwritten if the snapshot is subsequently refreshed (e.g. by another call to ``locals()``). CPython implementation detail: the dynamic snapshot for the current frame will be implicitly refreshed before each call to the trace function when a trace function is active. For reference, the current documentation of this builtin reads as follows: Update and return a dictionary representing the current local symbol table. Free variables are returned by locals() when it is called in function blocks, but not in class blocks. Note: The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter. (In other words: the status quo is that the semantics and behaviour of ``locals()`` are currently formally implementation defined, whereas the proposed state after this PEP is that the only implementation defined behaviour will be that encountered at function scope when a tracing function is defined, with the behaviour in all other cases being defined by the language and library references) Module scope ------------ At module scope, as well as when using ``exec()`` or ``eval()`` with a single namespace, ``locals()`` must return the same object as ``globals()``, which must be the actual execution namespace (available as ``inspect.currentframe().f_locals`` in implementations that provide access to frame objects). Variable assignments during subsequent code execution in the same scope must dynamically change the contents of the returned mapping, and changes to the returned mapping must change the values bound to local variable names in the execution environment. The semantics at module scope are required to be the same in both tracing mode (if provided by the implementation) and in regular operation. To capture this expectation as part of the language specification, the following paragraph will be added to the documentation for ``locals()``: At module scope, as well as when using ``exec()`` or ``eval()`` with a single namespace, this function returns the same namespace as ``globals()``. This part of the proposal does not require any changes to the reference implementation - it is standardisation of the current behaviour. Class scope ----------- At class scope, as well as when using ``exec()`` or ``eval()`` with separate global and local namespaces, ``locals()`` must return the specified local namespace (which may be supplied by the metaclass ``__prepare__`` method in the case of classes). As for module scope, this must be a direct reference to the actual execution namespace (available as ``inspect.currentframe().f_locals`` in implementations that provide access to frame objects). Variable assignments during subsequent code execution in the same scope must change the contents of the returned mapping, and changes to the returned mapping must change the values bound to local variable names in the execution environment. The mapping returned by ``locals()`` will *not* be used as the actual class namespace underlying the defined class (the class creation process will copy the contents to a fresh dictionary that is only accessible by going through the class machinery). For nested classes defined inside a function, any nonlocal cells referenced from the class scope are *not* included in the ``locals()`` mapping. The semantics at class scope are required to be the same in both tracing mode (if provided by the implementation) and in regular operation. To capture this expectation as part of the language specification, the following two paragraphs will be added to the documentation for ``locals()``: When using ``exec()`` or ``eval()`` with separate local and global namespaces, [this function] returns the given local namespace. At class scope, it returns the namespace that will be passed to the metaclass constructor. This part of the proposal does not require any changes to the reference implementation - it is standardisation of the current behaviour. Function scope -------------- At function scope, interpreter implementations are granted significant freedom to optimise local variable access, and hence are NOT required to permit arbitrary modification of local and nonlocal variable bindings through the mapping returned from ``locals()``. Historically, this leniency has been described in the language specification with the words "The contents of this dictionary should not be modified; changes may not affect the values of local and free variables used by the interpreter." This PEP proposes to change that text to instead say: At function scope (including for generators and coroutines), [this function] returns a dynamic snapshot of the function's local variables and any nonlocal cell references. In this case, changes made via the snapshot are *not* written back to the corresponding local variables or nonlocal cell references, and any such changes to the snapshot will be overwritten if the snapshot is subsequently refreshed (e.g. by another call to ``locals()``). CPython implementation detail: the dynamic snapshot for the currently executing frame will be implicitly refreshed before each call to the trace function when a trace function is active. This part of the proposal *does* require changes to the CPython reference implementation, as while it accurately describes the behaviour in regular operation, the "write back" strategy currently used to support namespace changes from trace functions doesn't comply with it (and also causes the quirky behavioural problems mentioned in the Rationale). CPython Implementation Changes ============================== The current cause of CPython's tracing mode quirks (both the side effects from simply installing a tracing function and the fact that writing values back to function locals only works for the specific function being traced) is the way that locals mutation support for trace hooks is currently implemented: the ``PyFrame_LocalsToFast`` function. When a trace function is installed, CPython currently does the following for function frames (those where the code object uses "fast locals" semantics): 1. Calls ``PyFrame_FastToLocals`` to update the dynamic snapshot 2. Calls the trace hook (with tracing of the hook itself disabled) 3. Calls ``PyFrame_LocalsToFast`` to capture any changes made to the dynamic snapshot This approach is problematic for a few different reasons: * Even if the trace function doesn't mutate the snapshot, the final step resets any cell references back to the state they were in before the trace function was called (this is the root cause of the bug report in [1]_) * If the trace function *does* mutate the snapshot, but then does something that causes the snapshot to be refreshed, those changes are lost (this is one aspect of the bug report in [3]_) * If the trace function attempts to mutate the local variables of a frame other than the one being traced (e.g. ``frame.f_back.f_locals``), those changes will almost certainly be lost (this is another aspect of the bug report in [3]_) * If a ``locals()`` reference is passed to another function, and *that* function mutates the snapshot namespace, then those changes *may* be written back to the execution frame *if* a trace hook is installed The proposed resolution to this problem is to take advantage of the fact that whereas functions typically access their *own* namespace using the language defined ``locals()`` builtin, trace functions necessarily use the implementation dependent ``frame.f_locals`` interface, as a frame reference is what gets passed to hook implementations. Instead of being a direct reference to the dynamic snapshot returned by ``locals()``, ``frame.f_locals`` will be updated to instead return a dedicated proxy type (implemented as a private subclass of the existing ``types.MappingProxyType``) that has two internal attributes not exposed as part of either the Python or public C API: * *mapping*: the dynamic snapshot that is returned by the ``locals()`` builtin * *frame*: the underlying frame that the snapshot is for ``__setitem__`` and ``__delitem__`` operations on the proxy will affect not only the dynamic snapshot, but *also* the corresponding fast local or cell reference on the underlying frame. The ``locals()`` builtin will be made aware of this proxy type, and continue to return a reference to the dynamic snapshot rather than to the write-through proxy. At the C API layer, ``PyEval_GetLocals()`` will implement the same semantics as the Python level ``locals()`` builtin, and a new ``PyFrame_GetPyLocals(frame)`` accessor API will be provided to allow the function level proxy bypass logic to be encapsulated entirely inside the frame implementation. The C level equivalent of accessing ``pyframe.f_locals`` in Python will be a new ``PyFrame_GetLocalsAttr(frame)`` API. Like the Python level descriptor, the new API will implicitly refresh the dynamic snapshot at function scope before returning a reference to the write-through proxy. The ``PyFrame_LocalsToFast()`` function will be changed to always emit ``RuntimeError``, explaining that it is no longer a supported operation, and affected code should be updated to rely on the write-through tracing mode proxy instead. Design Discussion ================= Ensuring ``locals()`` returns a shared snapshot at function scope ----------------------------------------------------------------- The ``locals()`` builtin is a required part of the language, and in the reference implementation it has historically returned a mutable mapping with the following characteristics: * each call to ``locals()`` returns the *same* mapping * for namespaces where ``locals()`` returns a reference to something other than the actual local execution namespace, each call to ``locals()`` updates the mapping with the current state of the local variables and any referenced nonlocal cells * changes to the returned mapping *usually* aren't written back to the local variable bindings or the nonlocal cell references, but write backs can be triggered by doing one of the following: * installing a Python level trace hook (write backs then happen whenever the trace hook is called) * running a function level wildcard import (requires bytecode injection in Py3) * running an ``exec`` statement in the function's scope (Py2 only, since ``exec`` became an ordinary builtin in Python 3) The proposal in this PEP aims to retain the first two properties (to maintain backwards compatibility with as much code as possible) while ensuring that simply installing a trace hook can't enable rebinding of function locals via the ``locals()`` builtin (whereas enabling rebinding via ``frame.f_locals`` inside the tracehook implementation is fully intended). Keeping ``locals()`` as a dynamic snapshot at function scope ------------------------------------------------------------ It would theoretically be possible to change the semantics of the ``locals()`` builtin to return the write-through proxy at function scope, rather than continuing to return a dynamic snapshot. This PEP doesn't (and won't) propose this as it's a backwards incompatible change in practice, even though code that relies on the current behaviour is technically operating in an undefined area of the language specification. Consider the following code snippet:: def example(): x = 1 locals()["x"] = 2 print(x) Even with a trace hook installed, that function will consistently print ``1`` on the current reference interpreter implementation:: >>> example() 1 >>> import sys >>> def basic_hook(*args): ... return basic_hook ... >>> sys.settrace(basic_hook) >>> example() 1 Similarly, ``locals()`` can be passed to the ``exec()`` and ``eval()`` builtins at function scope without risking unexpected rebinding of local variables. Provoking the reference interpreter into incorrectly mutating the local variable state requires a more complex setup where a nested function closes over a variable being rebound in the outer function, and due to the use of either threads, generators, or coroutines, it's possible for a trace function to start running for the nested function before the rebinding operation in the outer function, but finish running after the rebinding operation has taken place (in which case the rebinding will be reverted, which is the bug reported in [1]_). In addition to preserving the de facto semantics which have been in place since PEP 227 introduced nested scopes in Python 2.1, the other benefit of restricting the write-through proxy support to the implementation-defined frame object API is that it means that only interpreter implementations which emulate the full frame API need to offer the write-through capability at all, and that JIT-compiled implementations only need to enable it when a frame introspection API is invoked, or a trace hook is installed, not whenever ``locals()`` is accessed at function scope. What happens with the default args for ``eval()`` and ``exec()``? ----------------------------------------------------------------- These are formally defined as inheriting ``globals()`` and ``locals()`` from the calling scope by default. There isn't any need for the PEP to change these defaults, so it doesn't. Changing the frame API semantics in regular operation ----------------------------------------------------- Earlier versions of this PEP proposed having the semantics of the frame ``f_locals`` attribute depend on whether or not a tracing hook was currently installed - only providing the write-through proxy behaviour when a tracing hook was active, and otherwise behaving the same as the ``locals()`` builtin. That was adopted as the original design proposal for a couple of key reasons, one pragmatic and one more philosophical: * Object allocations and method wrappers aren't free, and tracing functions aren't the only operations that access frame locals from outside the function. Restricting the changes to tracing mode meant that the additional memory and execution time overhead of these changes would as close to zero in regular operation as we can possibly make them. * "Don't change what isn't broken": the current tracing mode problems are caused by a requirement that's specific to tracing mode (support for external rebinding of function local variable references), so it made sense to also restrict any related fixes to tracing mode However, actually attempting to implement and document that dynamic approach highlighted the fact that it makes for a really subtle runtime state dependent behaviour distinction in how ``frame.f_locals`` works, and creates several new edge cases around how ``f_locals`` behaves as trace functions are added and removed. Accordingly, the design was switched to the current one, where ``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always a dynamic snapshot, which is both simpler to implement and easier to explain. Regardless of how the CPython reference implementation chooses to handle this, optimising compilers and interpreters also remain free to impose additional restrictions on debuggers, by making local variable mutation through frame objects an opt-in behaviour that may disable some optimisations (just as the emulation of CPython's frame API is already an opt-in flag in some Python implementations). Historical semantics at function scope -------------------------------------- The current semantics of mutating ``locals()`` and ``frame.f_locals`` in CPython are rather quirky due to historical implementation details: * actual execution uses the fast locals array for local variable bindings and cell references for nonlocal variables * there's a ``PyFrame_FastToLocals`` operation that populates the frame's ``f_locals`` attribute based on the current state of the fast locals array and any referenced cells. This exists for three reasons: * allowing trace functions to read the state of local variables * allowing traceback processors to read the state of local variables * allowing ``locals()`` to read the state of local variables * a direct reference to ``frame.f_locals`` is returned from ``locals()``, so if you hand out multiple concurrent references, then all those references will be to the exact same dictionary * the two common calls to the reverse operation, ``PyFrame_LocalsToFast``, were removed in the migration to Python 3: ``exec`` is no longer a statement (and hence can no longer affect function local namespaces), and the compiler now disallows the use of ``from module import *`` operations at function scope * however, two obscure calling paths remain: ``PyFrame_LocalsToFast`` is called as part of returning from a trace function (which allows debuggers to make changes to the local variable state), and you can also still inject the ``IMPORT_STAR`` opcode when creating a function directly from a code object rather than via the compiler This proposal deliberately *doesn't* formalise these semantics as is, since they only make sense in terms of the historical evolution of the language and the reference implementation, rather than being deliberately designed. Implementation ============== The reference implementation update is in development as a draft pull request on GitHub ([6]_). Acknowledgements ================ Thanks to Nathaniel J. Smith for proposing the write-through proxy idea in [1]_ and pointing out some critical design flaws in earlier iterations of the PEP that attempted to avoid introducing such a proxy. References ========== .. [1] Broken local variable assignment given threads + trace hook + closure (https://bugs.python.org/issue30744) .. [2] Clarify the required behaviour of ``locals()`` (https://bugs.python.org/issue17960) .. [3] Updating function local variables from pdb is unreliable (https://bugs.python.org/issue9633) .. [4] CPython's Python API for installing trace hooks (https://docs.python.org/dev/library/sys.html#sys.settrace) .. [5] CPython's C API for installing trace hooks (https://docs.python.org/3/c-api/init.html#c.PyEval_SetTrace) .. [6] PEP 558 reference implementation (https://github.com/python/cpython/pull/3640/files) Copyright ========= This document has been placed in the public domain. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Tue May 21 11:31:12 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 May 2019 17:31:12 +0200 Subject: [Python-Dev] PEP 594: update 1 References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: <20190521173112.503f4c91@fsol> As I said, if the main annoyance with nntplib is the sporadic test failures, then the relevant tests can be disabled on CI. NNTP itself is still used, even if less and less. Regards Antoine. On Tue, 21 May 2019 16:12:42 +0200 Christian Heimes wrote: > Hi, > > I have updated the PEP with feedback from discussions. The highlights are: > > * Deprecate parser module > * Keep fileinput module > * Elaborate why crypt and spwd are dangerous and bad > * Improve sections for cgitb, colorsys, nntplib, and smtpd modules > * The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable substitutions. > * Mention that socketserver is going to stay for http.server and xmlrpc.server > * The future maintenance section now states that the deprecated modules may be adopted by Python community members. > https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff-ae358c21fa7968ee3b6c64479e051574 > > > I'll be traveling the next couple of days and will only have limited opportunities to respond on feedback. > > Christian > > ------------------------------------------------------------------- > PEP: 594 > Title: Removing dead batteries from the standard library > Author: Christian Heimes > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 20-May-2019 > Post-History: 21-May-2019 > > > Abstract > ======== > > This PEP proposed a list of standard library modules to be removed from the > standard library. The modules are mostly historic data formats and APIs that > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > Rationale > ========= > > Back in the early days of Python, the interpreter came with a large set of > useful modules. This was often refrained to as "batteries included" > philosophy and was one of the corner stones to Python's success story. > Users didn't have to figure out how to download and install separate > packages in order to write a simple web server or parse email. > > Times have changed. The introduction of the cheese shop (PyPI), setuptools, > and later pip, it became simple and straight forward to download and install > packages. Nowadays Python has a rich and vibrant ecosystem of third party > packages. It's pretty much standard to either install packages from PyPI or > use one of the many Python or Linux distributions. > > On the other hand, Python's standard library is piling up cruft, unnecessary > duplication of functionality, and dispensable features. This is undesirable > for several reasons. > > * Any additional module increases the maintenance cost for the Python core > development team. The team has limited resources, reduced maintenance cost > frees development time for other improvements. > * Modules in the standard library are generally favored and seen as the > de-facto solution for a problem. A majority of users only pick 3rd party > modules to replace a stdlib module, when they have a compelling reason, e.g. > lxml instead of `xml`. The removal of an unmaintained stdlib module > increases the chances of a community contributed module to become widely > used. > * A lean and mean standard library benefits platforms with limited resources > like devices with just a few hundred kilobyte of storage (e.g. BBC > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > (e.g. pyodide) also benefit from reduced download size. > > The modules in the PEP have been selected for deprecation because their > removal is either least controversial or most beneficial. For example > least controversial are 30 years old multimedia formats like ``sunau`` > audio format, which was used on SPARC and NeXT workstations in the late > 1980ties. The ``crypt`` module has fundamental flaws that are better solved > outside the standard library. > > This PEP also designates some modules as not scheduled for removal. Some > modules have been deprecated for several releases or seem unnecessary at > first glance. However it is beneficial to keep the modules in the standard > library, mostly for environments where installing a package from PyPI is not > an option. This can be cooperate environments or class rooms where external > code is not permitted without legal approval. > > * The usage of FTP is declining, but some files are still provided over > the FTP protocol or hosters offer FTP to upload content. Therefore > ``ftplib`` is going to stay. > * The ``optparse`` and ``getopt`` module are widely used. They are mature > modules with very low maintenance overhead. > * According to David Beazley [5]_ the ``wave`` module is easy to teach to > kids and can make crazy sounds. Making a computer generate crazy sounds is > powerful and highly motivating exercise for a 9yo aspiring developer. It's > a fun battery to keep. > > > Deprecation schedule > ==================== > > 3.8 > --- > > This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released > a few months before Python 2.7 will reach its end of lifetime. We expect that > Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and > 2020. To reduce churn and to allow a smooth transition from Python 2, > Python 3.8 will neither raise `DeprecationWarning` nor remove any > modules that have been scheduled for removal. Instead deprecated modules will > just be *documented* as deprecated. Optionally modules may emit a > `PendingDeprecationWarning`. > > All deprecated modules will also undergo a feature freeze. No additional > features should be added. Bug should still be fixed. > > 3.9 > --- > > Starting with Python 3.9, deprecated modules will start issuing > `DeprecationWarning`. The `parser`_ module is removed and potentially > replaced with a new module. > > All other deprecated modules are fully supported and will receive security > updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will > be released about 18 months after 3.8.0 (April 2021?) and most likely > be supported for 5 years after the release. The estimated EOL of Python 3.9 > is in 2026. > > 3.10 > ---- > > In 3.10 all deprecated modules will be removed from the CPython repository > together with tests, documentation, and autoconf rules. > > > PEP acceptance process > ====================== > > 3.8.0b1 is scheduled to be release shortly after the PEP is officially > submitted. Since it's improbable that the PEP will pass all stages of the > PEP process in time, I propose a two step acceptance process that is > analogous Python's two release deprecation process. > > The first *provisionally accepted* phase targets Python 3.8.0b1. In the first > phase no code is changes or removed. Modules are only documented as > deprecated. The only exception is the `parser`_ module. It has been > documented as deprecated since Python 2.5 and is scheduled for removal for > 3.9 to make place for a more advanced parser. > > The final decision, which modules will be removed and how the removed code > is preserved, can be delayed for another year. > > > Deprecated modules > ================== > > The modules are grouped as data encoding, multimedia, network, OS interface, > and misc modules. The majority of modules are for old data formats or > old APIs. Some others are rarely useful and have better replacements on > PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with > audio processing. > > .. csv-table:: Table 1: Proposed modules deprecations > :header: "Module", "Deprecated in", "To be removed", "Replacement" > :widths: 1, 1, 1, 2 > > aifc,3.8,3.10,\- > asynchat,3.8,3.10,asyncio > asyncore,3.8,3.10,asyncio > audioop,3.8,3.10,\- > binhex,3.8,3.10,\- > cgi,3.8,3.10,\- > cgitb,3.8,3.10,\- > chunk,3.8,3.10,\- > colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" > crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" > fileinput,\-,**keep**,argparse > formatter,3.4,3.10,\- > fpectl,**3.7**,**3.7**,\- > getopt,**3.2**,**keep**,"argparse, optparse" > imghdr,3.8,3.10,"filetype, puremagic, python-magic" > imp,**3.4**,3.10,importlib > lib2to3,\-,**keep**, > macpath,**3.7**,**3.8**,\- > msilib,3.8,3.10,\- > nntplib,3.8,3.10,\- > nis,3.8,3.10,\- > optparse,\-,**keep**,argparse > ossaudiodev,3.8,3.10,\- > parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" > pipes,3.8,3.10,subprocess > smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd > sndhdr,3.8,3.10,"filetype, puremagic, python-magic" > spwd,3.8,3.10,"python-pam, simplepam" > sunau,3.8,3.10,\- > uu,3.8,3.10,\- > wave,\-,**keep**, > xdrlib,3.8,3.10,\- > > > Data encoding modules > --------------------- > > binhex > ~~~~~~ > > The `binhex `_ module encodes > and decodes Apple Macintosh binhex4 data. It was originally developed for > TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to > encode binary email attachments. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > uu > ~~ > > The `uu `_ module provides > uuencode format, an old binary encoding format for email from 1980. The uu > format has been replaced by MIME. The uu codec is provided by the binascii > module. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > xdrlib > ~~~~~~ > > The `xdrlib `_ module supports > the Sun External Data Representation Standard. XDR is an old binary > serialization format from 1987. These days it's rarely used outside > specialized domains like NFS. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Multimedia modules > ------------------ > > aifc > ~~~~ > > The `aifc `_ module provides > support for reading and writing AIFF and AIFF-C files. The Audio Interchange > File Format is an old audio format from 1988 based on Amiga IFF. It was most > commonly used on the Apple Macintosh. These days only few specialized > application use AIFF. > > Module type > pure Python (depends on `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > audioop > ~~~~~~~ > > The `audioop `_ module > contains helper functions to manipulate raw audio data and adaptive > differential pulse-code modulated audio data. The module is implemented in > C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ > module depend on `audioop`_ for some operations. The byteswap operation in > the `wave`_ module can be substituted with little work. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > colorsys > ~~~~~~~~ > > The `colorsys `_ module > defines color conversion functions between RGB, YIQ, HSL, and HSV coordinate > systems. > > The PyPI packages *colormath*, *colour*, and *colorspacious* provide more and > advanced features. The Pillow library is better suited to transform images > between color systems. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `colormath `_, > `colour `_ > `colorspacious `_, > `Pillow `_ > > chunk > ~~~~~ > > The `chunk `_ module provides > support for reading and writing Electronic Arts' Interchange File Format. > IFF is an old audio file format originally introduced for Commodore and > Amiga. The format is no longer relevant. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > imghdr > ~~~~~~ > > The `imghdr `_ module is a > simple tool to guess the image file format from the first 32 bytes > of a file or buffer. It supports only a limited amount of formats and > neither returns resolution nor color depth. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `puremagic `_, > `filetype `_, > `python-magic `_ > > ossaudiodev > ~~~~~~~~~~~ > > The `ossaudiodev `_ > module provides support for Open Sound System, an interface to sound > playback and capture devices. OSS was initially free software, but later > support for newer sound devices and improvements were proprietary. Linux > community abandoned OSS in favor of ALSA [1]_. Some operation systems like > OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > sndhdr > ~~~~~~ > > The `sndhdr `_ module is > similar to the `imghdr`_ module but for audio formats. It guesses file > format, channels, frame rate, and sample widths from the first 512 bytes of > a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and > other ancient formats. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `puremagic `_, > `filetype `_, > `python-magic `_ > > sunau > ~~~~~ > > The `sunau `_ module provides > support for Sun AU sound format. It's yet another old, obsolete file format. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Networking modules > ------------------ > > asynchat > ~~~~~~~~ > > The `asynchat `_ module > is build on top of `asyncore`_ and has been deprecated since Python 3.6. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > asyncore > ~~~~~~~~ > > The `asyncore `_ module was > the first module for asynchronous socket service clients and servers. It > has been replaced by asyncio and is deprecated since Python 3.6. > > The ``asyncore`` module is also used in stdlib tests. The tests for > ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly > based on ``asyncore``. These tests must be updated to use asyncio or > threading. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > > cgi > ~~~ > > The `cgi `_ module is a support > module for Common Gateway Interface (CGI) scripts. CGI is deemed as > inefficient because every incoming request is handled in a new process. PEP > 206 considers the module as *designed poorly and are now near-impossible > to fix*. > > Several people proposed to either keep the cgi module for features like > `cgi.parse_qs()` or move `cgi.escape()` to a different module. The > functions `cgi.parse_qs` and `cgi.parse_qsl` have been > deprecated for a while and are actually aliases for > `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The > function `cgi.quote` has been deprecated in favor of `html.quote` > with secure default values. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > cgitb > ~~~~~ > > The `cgitb `_ module is a > helper for the cgi module for configurable tracebacks. > > The ``cgitb`` module is not used by any major Python web framework (Django, > Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an > optional debugging middleware. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > smtpd > ~~~~~ > > The `smtpd `_ module provides > a simple implementation of a SMTP mail server. The module documentation > marks the module as deprecated and recommends ``aiosmtpd`` instead. The > deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. > > Module type > pure Python > Deprecated in > **3.7** > To be removed in > 3.10 > Substitute > aiosmtpd > > nntplib > ~~~~~~~ > > The `nntplib `_ module > implements the client side of the Network News Transfer Protocol (nntp). News > groups used to be a dominant platform for online discussions. Over the last > two decades, news has been slowly but steadily replaced with mailing lists > and web-based discussion platforms. Twisted is also > `planning `_ to deprecate NNTP > support and `pynnt `_ hasn't seen any > activity since 2014. This is a good indicator that the public interest in > NNTP support is declining. > > The ``nntplib`` tests have been the cause of additional work in the recent > past. Python only contains client side of NNTP. The tests connect to > external news server. The servers are sometimes unavailble, too slow, or do > not work correctly over IPv6. The situation causes flaky test runs on > buildbots. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Operating system interface > -------------------------- > > crypt > ~~~~~ > > The `crypt `_ module implements > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor > quality and insecure. Users are discouraged to use them. > > * The module is not available on Windows. Cross-platform application need > an alternative implementation any way. > * Only DES encryption is guarenteed to be available. DES has an extremely > limited key space of 2**56. > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > algorithm that is still secure. However it's in glibc and therefore not > commonly available on Linux. > * Depending on the platform, the ``crypt`` module is not thread safe. Only > implementations with ``crypt_r(3)`` are thread safe. > * The module was never useful to interact with system user and password > databases. On BSD, macOS, and Linux, all user authentication and > password modification operations must go through PAM (pluggable > authentication module), see `spwd`_ deprecation. > > Module type > C extension + Python module > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `bcrypt `_, > `passlib `_, > `argon2cffi `_, > hashlib module (PBKDF2, scrypt) > > macpath > ~~~~~~~ > > The `macpath `_ module > provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer > supported > > Module type > pure Python > Deprecated in > 3.7 > Removed in > 3.8 > Substitute > **none** > > nis > ~~~ > > The `nis `_ module provides > NIS/YP support. Network Information Service / Yellow Pages is an old and > deprecated directory service protocol developed by Sun Microsystems. It's > designed successor NIS+ from 1992 never took off. For a long time, libc's > Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more powerful > and more secure replacement of NIS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > spwd > ~~~~ > > The `spwd `_ module provides > direct access to Unix shadow password database using non-standard APIs. > > In general it's a bad idea to use the spwd. The spwd circumvents system > security policies, it does not use the PAM stack, and is only compatible > with local user accounts, because it ignores NSS. The use of the ``spwd`` > module for access control must be consider a *security bug*, as it bypasses > PAM's access control. > > Further more the ``spwd`` module uses the > `shadow(3) `_ APIs. > Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. This > is dangerous and even forbidden for confined services on systems with a > security engine like SELinux or AppArmor. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `python-pam `_, > `simpleplam `_ > > Misc modules > ------------ > > formatter > ~~~~~~~~~ > > The `formatter `_ module > is an old text formatting module which has been deprecated since Python 3.4. > > Module type > pure Python > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > *n/a* > > imp > ~~~ > > The `imp `_ module is the > predecessor of the > `importlib `_ module. Most > functions have been deprecated since Python 3.3 and the module since > Python 3.4. > > Module type > C extension > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > importlib > > msilib > ~~~~~~ > > The `msilib `_ package is a > Windows-only package. It supports the creation of Microsoft Installers (MSI). > The package also exposes additional APIs to create cabinet files (CAB). The > module is used to facilitate distutils to create MSI installers with > ``bdist_msi`` command. In the past it was used to create CPython's official > Windows installer, too. > > Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) > as new deployment model [3]_. > > Module type > C extension + Python code > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > parser > ~~~~~~ > > The `parser `_ module provides > an interface to Python?s internal parser and byte-code compiler. The stdlib > has superior ways to interact with the parse tree. From Python 2.5 onward, > it's much more convenient to cut in at the Abstract Syntax Tree (AST) > generation and compilation stage. > > The ``parser`` module causes additional work. It's C code that must be > kept in sync with any change to Python's grammar and internal parser. > Pablo wants to remove the parser module and promote lib2to3's pgen2 instead > [6]_. > > Most importantly the presence of the ``parser`` module makes it harder to > switch to something more powerful than a LL(1) parser [7]_. Since the > ``parser`` module is documented as deprecated since Python 2.5 and a new > parsing technology is planned for 3.9, the ``parser`` module is scheduled for > removal in 3.9. > > Module type > C extension > Deprecated in > 3.8, documented as deprecated since **2.5** > To be removed in > **3.9** > Substitute > ast, lib2to3.pgen2 > > pipes > ~~~~~ > > The `pipes `_ module provides > helpers to pipe the input of one command into the output of another command. > The module is built on top of ``os.popen``. Users are encouraged to use > the subprocess module instead. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > subprocess module > > Removed modules > =============== > > fpectl > ------ > > The `fpectl `_ module was > never built by default, its usage was discouraged and considered dangerous. > It also required a configure flag that caused an ABI incompatibility. The > module was removed in 3.7 by Nathaniel J. Smith in > `bpo-29137 `_. > > Module type > C extension + CAPI > Deprecated in > 3.7 > Removed in > 3.7 > Substitute > **none** > > > Modules to keep > =============== > > Some modules were originally proposed for deprecation. > > fileinput > --------- > > The `fileinput `_ module > implements a helpers to iterate over a list of files from ``sys.argv``. The > module predates the optparser and argparser module. The same functionality > can be implemented with the argparser module. > > Several core developers expressed their interest to keep the module in the > standard library, as it is handy for quick scripts. > > Module type > pure Python > > lib2to3 > ------- > > The `lib2to3 `_ package provides > the ``2to3`` command to transpile Python 2 code to Python 3 code. > > The package is useful for other tasks besides porting code from Python 2 to > 3. For example `black`_ uses it for code reformatting. > > Module type > pure Python > > getopt > ------ > > The `getopt `_ module mimics > C's getopt() option parser. > > Although users are encouraged to use argparse instead, the getopt module is > still widely used. The module is small, simple, and handy for C developers > to write simple Python scripts. > > Module type > pure Python > Substitute > argparse > > optparse > -------- > > The `optparse `_ module is > the predecessor of the argparse module. > > Although it has been deprecated for many years, it's still too widely used > to remove it. > > Module type > pure Python > Deprecated in > 3.2 > Substitute > argparse > > wave > ---- > > The `wave `_ module provides > support for the WAV sound format. > > The module is not deprecated, because The WAV format is still relevant these > days. The ``wave`` module is also used in education, e.g. to show kids how > to make noise with a computer. > > The module uses one simple function from the `audioop`_ module to perform > byte swapping between little and big endian formats. Before 24 bit WAV > support was added, byte swap used to be implemented with the ``array`` > module. To remove ``wave``'s dependency on the ``audioop``, the byte swap > function could be either be moved to another module (e.g. ``operator``) or > the ``array`` module could gain support for 24 bit (3 byte) arrays. > > Module type > pure Python (depends on *byteswap* from `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > > Future maintenance of removed modules > ===================================== > > The main goal of the PEP is to reduce the burden and workload on the Python > core developer team. Therefore removed modules will not be maintained by > the core team as separate PyPI packages. However the removed code, tests and > documentation may be moved into a new git repository, so community members > have a place from which they can pick up and fork code. > > A first draft of a `legacylib `_ > repository is available on my private Github account. The modules could be > made available on PyPI. The Python core team will not publish or maintain > the packages. It is my hope that members of the Python community will > adopt, maintain, and perhaps improve the deprecated modules. > > It's my hope that some of the deprecated modules will be picked up and > adopted by users that actually care about them. For example ``colorsys`` and > ``imghdr`` are useful modules, but have limited feature set. A fork of > ``imghdr`` can add new features and support for more image formats, without > being constrained by Python's release cycle. > > Most of the modules are in pure Python and can be easily packaged. Some > depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ > does not depend on any external libraries, it can be shipped in as binary > wheels with some effort. Other C modules can be replaced with ctypes or cffi. > For example I created `legacycrypt `_ > with ``_crypt`` extension reimplemented with a few lines of ctypes code. > > > Discussions > =========== > > * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. > * Berker Peksag proposed to deprecate and removed *msilib*. > * Brett Cannon recommended to delay active deprecation warnings and removal > of modules like *imp* until Python 3.10. Version 3.8 will be released > shortly before Python 2 reaches end of lifetime. A delay reduced churn for > users that migrate from Python 2 to 3.8. > * Brett also came up with the idea to keep lib2to3. The package is useful > for other purposes, e.g. `black `_ uses > it to reformat Python code. > * At one point, distutils was mentioned in the same sentence as this PEP. > To avoid lengthy discussion and delay of the PEP, I decided against dealing > with distutils. Deprecation of the distutils package will be handled by > another PEP. > * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) > convinced me to keep the `wave`_ module. [4]_ > * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ > * Andrew Svetlov mentioned the ``socketserver`` module is questionable. > However it's used to implement ``http.server`` and ``xmlrpc.server``. The > stdlib doesn't have a replacement for the servers, yet. > > > Update history > ============== > > Update 1 > -------- > > * Deprecate `parser`_ module > * Keep `fileinput`_ module > * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad > * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ modules > * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now > list suitable substitutions. > * Mention that ``socketserver`` is going to stay for ``http.server`` and > ``xmlrpc.server`` > * The future maintenance section now states that the deprecated modules > may be adopted by Python community members. > > References > ========== > > .. [1] https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free > .. [2] https://man.openbsd.org/ossaudio > .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ > .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 > .. [5] https://twitter.com/dabeaz/status/1130278844479545351 > .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html > .. [7] https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org From guido at python.org Tue May 21 11:35:05 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 21 May 2019 08:35:05 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <20190521173112.503f4c91@fsol> References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> Message-ID: OK, sounds like nntplib can stay ? this time. On Tue, May 21, 2019 at 08:33 Antoine Pitrou wrote: > > As I said, if the main annoyance with nntplib is the sporadic test > failures, then the relevant tests can be disabled on CI. > > NNTP itself is still used, even if less and less. > > Regards > > Antoine. > > > On Tue, 21 May 2019 16:12:42 +0200 > Christian Heimes wrote: > > Hi, > > > > I have updated the PEP with feedback from discussions. The highlights > are: > > > > * Deprecate parser module > > * Keep fileinput module > > * Elaborate why crypt and spwd are dangerous and bad > > * Improve sections for cgitb, colorsys, nntplib, and smtpd modules > > * The colorsys, crypt, imghdr, sndhdr, and spwd sections now list > suitable substitutions. > > * Mention that socketserver is going to stay for http.server and > xmlrpc.server > > * The future maintenance section now states that the deprecated modules > may be adopted by Python community members. > > > https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff-ae358c21fa7968ee3b6c64479e051574 > > > > > > I'll be traveling the next couple of days and will only have limited > opportunities to respond on feedback. > > > > Christian > > > > ------------------------------------------------------------------- > > PEP: 594 > > Title: Removing dead batteries from the standard library > > Author: Christian Heimes > > Status: Draft > > Type: Standards Track > > Content-Type: text/x-rst > > Created: 20-May-2019 > > Post-History: 21-May-2019 > > > > > > Abstract > > ======== > > > > This PEP proposed a list of standard library modules to be removed from > the > > standard library. The modules are mostly historic data formats and APIs > that > > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > > > > Rationale > > ========= > > > > Back in the early days of Python, the interpreter came with a large set > of > > useful modules. This was often refrained to as "batteries included" > > philosophy and was one of the corner stones to Python's success story. > > Users didn't have to figure out how to download and install separate > > packages in order to write a simple web server or parse email. > > > > Times have changed. The introduction of the cheese shop (PyPI), > setuptools, > > and later pip, it became simple and straight forward to download and > install > > packages. Nowadays Python has a rich and vibrant ecosystem of third party > > packages. It's pretty much standard to either install packages from PyPI > or > > use one of the many Python or Linux distributions. > > > > On the other hand, Python's standard library is piling up cruft, > unnecessary > > duplication of functionality, and dispensable features. This is > undesirable > > for several reasons. > > > > * Any additional module increases the maintenance cost for the Python > core > > development team. The team has limited resources, reduced maintenance > cost > > frees development time for other improvements. > > * Modules in the standard library are generally favored and seen as the > > de-facto solution for a problem. A majority of users only pick 3rd > party > > modules to replace a stdlib module, when they have a compelling > reason, e.g. > > lxml instead of `xml`. The removal of an unmaintained stdlib module > > increases the chances of a community contributed module to become > widely > > used. > > * A lean and mean standard library benefits platforms with limited > resources > > like devices with just a few hundred kilobyte of storage (e.g. BBC > > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > > (e.g. pyodide) also benefit from reduced download size. > > > > The modules in the PEP have been selected for deprecation because their > > removal is either least controversial or most beneficial. For example > > least controversial are 30 years old multimedia formats like ``sunau`` > > audio format, which was used on SPARC and NeXT workstations in the late > > 1980ties. The ``crypt`` module has fundamental flaws that are better > solved > > outside the standard library. > > > > This PEP also designates some modules as not scheduled for removal. Some > > modules have been deprecated for several releases or seem unnecessary at > > first glance. However it is beneficial to keep the modules in the > standard > > library, mostly for environments where installing a package from PyPI is > not > > an option. This can be cooperate environments or class rooms where > external > > code is not permitted without legal approval. > > > > * The usage of FTP is declining, but some files are still provided over > > the FTP protocol or hosters offer FTP to upload content. Therefore > > ``ftplib`` is going to stay. > > * The ``optparse`` and ``getopt`` module are widely used. They are mature > > modules with very low maintenance overhead. > > * According to David Beazley [5]_ the ``wave`` module is easy to teach to > > kids and can make crazy sounds. Making a computer generate crazy > sounds is > > powerful and highly motivating exercise for a 9yo aspiring developer. > It's > > a fun battery to keep. > > > > > > Deprecation schedule > > ==================== > > > > 3.8 > > --- > > > > This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be > released > > a few months before Python 2.7 will reach its end of lifetime. We expect > that > > Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and > > 2020. To reduce churn and to allow a smooth transition from Python 2, > > Python 3.8 will neither raise `DeprecationWarning` nor remove any > > modules that have been scheduled for removal. Instead deprecated modules > will > > just be *documented* as deprecated. Optionally modules may emit a > > `PendingDeprecationWarning`. > > > > All deprecated modules will also undergo a feature freeze. No additional > > features should be added. Bug should still be fixed. > > > > 3.9 > > --- > > > > Starting with Python 3.9, deprecated modules will start issuing > > `DeprecationWarning`. The `parser`_ module is removed and potentially > > replaced with a new module. > > > > All other deprecated modules are fully supported and will receive > security > > updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will > > be released about 18 months after 3.8.0 (April 2021?) and most likely > > be supported for 5 years after the release. The estimated EOL of Python > 3.9 > > is in 2026. > > > > 3.10 > > ---- > > > > In 3.10 all deprecated modules will be removed from the CPython > repository > > together with tests, documentation, and autoconf rules. > > > > > > PEP acceptance process > > ====================== > > > > 3.8.0b1 is scheduled to be release shortly after the PEP is officially > > submitted. Since it's improbable that the PEP will pass all stages of the > > PEP process in time, I propose a two step acceptance process that is > > analogous Python's two release deprecation process. > > > > The first *provisionally accepted* phase targets Python 3.8.0b1. In the > first > > phase no code is changes or removed. Modules are only documented as > > deprecated. The only exception is the `parser`_ module. It has been > > documented as deprecated since Python 2.5 and is scheduled for removal > for > > 3.9 to make place for a more advanced parser. > > > > The final decision, which modules will be removed and how the removed > code > > is preserved, can be delayed for another year. > > > > > > Deprecated modules > > ================== > > > > The modules are grouped as data encoding, multimedia, network, OS > interface, > > and misc modules. The majority of modules are for old data formats or > > old APIs. Some others are rarely useful and have better replacements on > > PyPI, e.g. Pillow for image processing or NumPy-based projects to deal > with > > audio processing. > > > > .. csv-table:: Table 1: Proposed modules deprecations > > :header: "Module", "Deprecated in", "To be removed", "Replacement" > > :widths: 1, 1, 1, 2 > > > > aifc,3.8,3.10,\- > > asynchat,3.8,3.10,asyncio > > asyncore,3.8,3.10,asyncio > > audioop,3.8,3.10,\- > > binhex,3.8,3.10,\- > > cgi,3.8,3.10,\- > > cgitb,3.8,3.10,\- > > chunk,3.8,3.10,\- > > colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" > > crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" > > fileinput,\-,**keep**,argparse > > formatter,3.4,3.10,\- > > fpectl,**3.7**,**3.7**,\- > > getopt,**3.2**,**keep**,"argparse, optparse" > > imghdr,3.8,3.10,"filetype, puremagic, python-magic" > > imp,**3.4**,3.10,importlib > > lib2to3,\-,**keep**, > > macpath,**3.7**,**3.8**,\- > > msilib,3.8,3.10,\- > > nntplib,3.8,3.10,\- > > nis,3.8,3.10,\- > > optparse,\-,**keep**,argparse > > ossaudiodev,3.8,3.10,\- > > parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" > > pipes,3.8,3.10,subprocess > > smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd > > sndhdr,3.8,3.10,"filetype, puremagic, python-magic" > > spwd,3.8,3.10,"python-pam, simplepam" > > sunau,3.8,3.10,\- > > uu,3.8,3.10,\- > > wave,\-,**keep**, > > xdrlib,3.8,3.10,\- > > > > > > Data encoding modules > > --------------------- > > > > binhex > > ~~~~~~ > > > > The `binhex `_ module > encodes > > and decodes Apple Macintosh binhex4 data. It was originally developed for > > TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to > > encode binary email attachments. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > uu > > ~~ > > > > The `uu `_ module provides > > uuencode format, an old binary encoding format for email from 1980. The > uu > > format has been replaced by MIME. The uu codec is provided by the > binascii > > module. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > xdrlib > > ~~~~~~ > > > > The `xdrlib `_ module > supports > > the Sun External Data Representation Standard. XDR is an old binary > > serialization format from 1987. These days it's rarely used outside > > specialized domains like NFS. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > > > Multimedia modules > > ------------------ > > > > aifc > > ~~~~ > > > > The `aifc `_ module > provides > > support for reading and writing AIFF and AIFF-C files. The Audio > Interchange > > File Format is an old audio format from 1988 based on Amiga IFF. It was > most > > commonly used on the Apple Macintosh. These days only few specialized > > application use AIFF. > > > > Module type > > pure Python (depends on `audioop`_ C extension) > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > audioop > > ~~~~~~~ > > > > The `audioop `_ module > > contains helper functions to manipulate raw audio data and adaptive > > differential pulse-code modulated audio data. The module is implemented > in > > C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ > > module depend on `audioop`_ for some operations. The byteswap operation > in > > the `wave`_ module can be substituted with little work. > > > > Module type > > C extension > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > colorsys > > ~~~~~~~~ > > > > The `colorsys `_ module > > defines color conversion functions between RGB, YIQ, HSL, and HSV > coordinate > > systems. > > > > The PyPI packages *colormath*, *colour*, and *colorspacious* provide > more and > > advanced features. The Pillow library is better suited to transform > images > > between color systems. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > `colormath `_, > > `colour `_ > > `colorspacious `_, > > `Pillow `_ > > > > chunk > > ~~~~~ > > > > The `chunk `_ module > provides > > support for reading and writing Electronic Arts' Interchange File Format. > > IFF is an old audio file format originally introduced for Commodore and > > Amiga. The format is no longer relevant. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > imghdr > > ~~~~~~ > > > > The `imghdr `_ module is > a > > simple tool to guess the image file format from the first 32 bytes > > of a file or buffer. It supports only a limited amount of formats and > > neither returns resolution nor color depth. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > `puremagic `_, > > `filetype `_, > > `python-magic `_ > > > > ossaudiodev > > ~~~~~~~~~~~ > > > > The `ossaudiodev `_ > > module provides support for Open Sound System, an interface to sound > > playback and capture devices. OSS was initially free software, but later > > support for newer sound devices and improvements were proprietary. Linux > > community abandoned OSS in favor of ALSA [1]_. Some operation systems > like > > OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. > > > > Module type > > C extension > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > sndhdr > > ~~~~~~ > > > > The `sndhdr `_ module is > > similar to the `imghdr`_ module but for audio formats. It guesses file > > format, channels, frame rate, and sample widths from the first 512 bytes > of > > a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and > > other ancient formats. > > > > Module type > > pure Python (depends on `audioop`_ C extension for some operations) > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > `puremagic `_, > > `filetype `_, > > `python-magic `_ > > > > sunau > > ~~~~~ > > > > The `sunau `_ module > provides > > support for Sun AU sound format. It's yet another old, obsolete file > format. > > > > Module type > > pure Python (depends on `audioop`_ C extension for some operations) > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > > > Networking modules > > ------------------ > > > > asynchat > > ~~~~~~~~ > > > > The `asynchat `_ module > > is build on top of `asyncore`_ and has been deprecated since Python 3.6. > > > > Module type > > pure Python > > Deprecated in > > 3.6 > > Removed in > > 3.10 > > Substitute > > asyncio > > > > asyncore > > ~~~~~~~~ > > > > The `asyncore `_ > module was > > the first module for asynchronous socket service clients and servers. It > > has been replaced by asyncio and is deprecated since Python 3.6. > > > > The ``asyncore`` module is also used in stdlib tests. The tests for > > ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly > > based on ``asyncore``. These tests must be updated to use asyncio or > > threading. > > > > Module type > > pure Python > > Deprecated in > > 3.6 > > Removed in > > 3.10 > > Substitute > > asyncio > > > > > > cgi > > ~~~ > > > > The `cgi `_ module is a > support > > module for Common Gateway Interface (CGI) scripts. CGI is deemed as > > inefficient because every incoming request is handled in a new process. > PEP > > 206 considers the module as *designed poorly and are now near-impossible > > to fix*. > > > > Several people proposed to either keep the cgi module for features like > > `cgi.parse_qs()` or move `cgi.escape()` to a different module. The > > functions `cgi.parse_qs` and `cgi.parse_qsl` have been > > deprecated for a while and are actually aliases for > > `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The > > function `cgi.quote` has been deprecated in favor of `html.quote` > > with secure default values. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > > > cgitb > > ~~~~~ > > > > The `cgitb `_ module is a > > helper for the cgi module for configurable tracebacks. > > > > The ``cgitb`` module is not used by any major Python web framework > (Django, > > Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an > > optional debugging middleware. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > smtpd > > ~~~~~ > > > > The `smtpd `_ module > provides > > a simple implementation of a SMTP mail server. The module documentation > > marks the module as deprecated and recommends ``aiosmtpd`` instead. The > > deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. > > > > Module type > > pure Python > > Deprecated in > > **3.7** > > To be removed in > > 3.10 > > Substitute > > aiosmtpd > > > > nntplib > > ~~~~~~~ > > > > The `nntplib `_ module > > implements the client side of the Network News Transfer Protocol (nntp). > News > > groups used to be a dominant platform for online discussions. Over the > last > > two decades, news has been slowly but steadily replaced with mailing > lists > > and web-based discussion platforms. Twisted is also > > `planning `_ to deprecate > NNTP > > support and `pynnt `_ hasn't > seen any > > activity since 2014. This is a good indicator that the public interest in > > NNTP support is declining. > > > > The ``nntplib`` tests have been the cause of additional work in the > recent > > past. Python only contains client side of NNTP. The tests connect to > > external news server. The servers are sometimes unavailble, too slow, or > do > > not work correctly over IPv6. The situation causes flaky test runs on > > buildbots. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > > > Operating system interface > > -------------------------- > > > > crypt > > ~~~~~ > > > > The `crypt `_ module > implements > > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of > poor > > quality and insecure. Users are discouraged to use them. > > > > * The module is not available on Windows. Cross-platform application need > > an alternative implementation any way. > > * Only DES encryption is guarenteed to be available. DES has an extremely > > limited key space of 2**56. > > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > > algorithm that is still secure. However it's in glibc and therefore not > > commonly available on Linux. > > * Depending on the platform, the ``crypt`` module is not thread safe. > Only > > implementations with ``crypt_r(3)`` are thread safe. > > * The module was never useful to interact with system user and password > > databases. On BSD, macOS, and Linux, all user authentication and > > password modification operations must go through PAM (pluggable > > authentication module), see `spwd`_ deprecation. > > > > Module type > > C extension + Python module > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > `bcrypt `_, > > `passlib `_, > > `argon2cffi `_, > > hashlib module (PBKDF2, scrypt) > > > > macpath > > ~~~~~~~ > > > > The `macpath `_ module > > provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no > longer > > supported > > > > Module type > > pure Python > > Deprecated in > > 3.7 > > Removed in > > 3.8 > > Substitute > > **none** > > > > nis > > ~~~ > > > > The `nis `_ module provides > > NIS/YP support. Network Information Service / Yellow Pages is an old and > > deprecated directory service protocol developed by Sun Microsystems. It's > > designed successor NIS+ from 1992 never took off. For a long time, libc's > > Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more > powerful > > and more secure replacement of NIS. > > > > Module type > > C extension > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > spwd > > ~~~~ > > > > The `spwd `_ module > provides > > direct access to Unix shadow password database using non-standard APIs. > > > > In general it's a bad idea to use the spwd. The spwd circumvents system > > security policies, it does not use the PAM stack, and is only compatible > > with local user accounts, because it ignores NSS. The use of the ``spwd`` > > module for access control must be consider a *security bug*, as it > bypasses > > PAM's access control. > > > > Further more the ``spwd`` module uses the > > `shadow(3) `_ APIs. > > Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. > This > > is dangerous and even forbidden for confined services on systems with a > > security engine like SELinux or AppArmor. > > > > Module type > > C extension > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > `python-pam `_, > > `simpleplam `_ > > > > Misc modules > > ------------ > > > > formatter > > ~~~~~~~~~ > > > > The `formatter `_ > module > > is an old text formatting module which has been deprecated since Python > 3.4. > > > > Module type > > pure Python > > Deprecated in > > 3.4 > > To be removed in > > 3.10 > > Substitute > > *n/a* > > > > imp > > ~~~ > > > > The `imp `_ module is the > > predecessor of the > > `importlib `_ module. > Most > > functions have been deprecated since Python 3.3 and the module since > > Python 3.4. > > > > Module type > > C extension > > Deprecated in > > 3.4 > > To be removed in > > 3.10 > > Substitute > > importlib > > > > msilib > > ~~~~~~ > > > > The `msilib `_ package > is a > > Windows-only package. It supports the creation of Microsoft Installers > (MSI). > > The package also exposes additional APIs to create cabinet files (CAB). > The > > module is used to facilitate distutils to create MSI installers with > > ``bdist_msi`` command. In the past it was used to create CPython's > official > > Windows installer, too. > > > > Microsoft is slowly moving away from MSI in favor of Windows 10 Apps > (AppX) > > as new deployment model [3]_. > > > > Module type > > C extension + Python code > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > parser > > ~~~~~~ > > > > The `parser `_ module > provides > > an interface to Python?s internal parser and byte-code compiler. The > stdlib > > has superior ways to interact with the parse tree. From Python 2.5 > onward, > > it's much more convenient to cut in at the Abstract Syntax Tree (AST) > > generation and compilation stage. > > > > The ``parser`` module causes additional work. It's C code that must be > > kept in sync with any change to Python's grammar and internal parser. > > Pablo wants to remove the parser module and promote lib2to3's pgen2 > instead > > [6]_. > > > > Most importantly the presence of the ``parser`` module makes it harder to > > switch to something more powerful than a LL(1) parser [7]_. Since the > > ``parser`` module is documented as deprecated since Python 2.5 and a new > > parsing technology is planned for 3.9, the ``parser`` module is > scheduled for > > removal in 3.9. > > > > Module type > > C extension > > Deprecated in > > 3.8, documented as deprecated since **2.5** > > To be removed in > > **3.9** > > Substitute > > ast, lib2to3.pgen2 > > > > pipes > > ~~~~~ > > > > The `pipes `_ module > provides > > helpers to pipe the input of one command into the output of another > command. > > The module is built on top of ``os.popen``. Users are encouraged to use > > the subprocess module instead. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > subprocess module > > > > Removed modules > > =============== > > > > fpectl > > ------ > > > > The `fpectl `_ module > was > > never built by default, its usage was discouraged and considered > dangerous. > > It also required a configure flag that caused an ABI incompatibility. The > > module was removed in 3.7 by Nathaniel J. Smith in > > `bpo-29137 `_. > > > > Module type > > C extension + CAPI > > Deprecated in > > 3.7 > > Removed in > > 3.7 > > Substitute > > **none** > > > > > > Modules to keep > > =============== > > > > Some modules were originally proposed for deprecation. > > > > fileinput > > --------- > > > > The `fileinput `_ > module > > implements a helpers to iterate over a list of files from ``sys.argv``. > The > > module predates the optparser and argparser module. The same > functionality > > can be implemented with the argparser module. > > > > Several core developers expressed their interest to keep the module in > the > > standard library, as it is handy for quick scripts. > > > > Module type > > pure Python > > > > lib2to3 > > ------- > > > > The `lib2to3 `_ package > provides > > the ``2to3`` command to transpile Python 2 code to Python 3 code. > > > > The package is useful for other tasks besides porting code from Python 2 > to > > 3. For example `black`_ uses it for code reformatting. > > > > Module type > > pure Python > > > > getopt > > ------ > > > > The `getopt `_ module > mimics > > C's getopt() option parser. > > > > Although users are encouraged to use argparse instead, the getopt module > is > > still widely used. The module is small, simple, and handy for C > developers > > to write simple Python scripts. > > > > Module type > > pure Python > > Substitute > > argparse > > > > optparse > > -------- > > > > The `optparse `_ > module is > > the predecessor of the argparse module. > > > > Although it has been deprecated for many years, it's still too widely > used > > to remove it. > > > > Module type > > pure Python > > Deprecated in > > 3.2 > > Substitute > > argparse > > > > wave > > ---- > > > > The `wave `_ module > provides > > support for the WAV sound format. > > > > The module is not deprecated, because The WAV format is still relevant > these > > days. The ``wave`` module is also used in education, e.g. to show kids > how > > to make noise with a computer. > > > > The module uses one simple function from the `audioop`_ module to perform > > byte swapping between little and big endian formats. Before 24 bit WAV > > support was added, byte swap used to be implemented with the ``array`` > > module. To remove ``wave``'s dependency on the ``audioop``, the byte swap > > function could be either be moved to another module (e.g. ``operator``) > or > > the ``array`` module could gain support for 24 bit (3 byte) arrays. > > > > Module type > > pure Python (depends on *byteswap* from `audioop`_ C extension) > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > *n/a* > > > > > > Future maintenance of removed modules > > ===================================== > > > > The main goal of the PEP is to reduce the burden and workload on the > Python > > core developer team. Therefore removed modules will not be maintained by > > the core team as separate PyPI packages. However the removed code, tests > and > > documentation may be moved into a new git repository, so community > members > > have a place from which they can pick up and fork code. > > > > A first draft of a `legacylib `_ > > repository is available on my private Github account. The modules could > be > > made available on PyPI. The Python core team will not publish or maintain > > the packages. It is my hope that members of the Python community will > > adopt, maintain, and perhaps improve the deprecated modules. > > > > It's my hope that some of the deprecated modules will be picked up and > > adopted by users that actually care about them. For example ``colorsys`` > and > > ``imghdr`` are useful modules, but have limited feature set. A fork of > > ``imghdr`` can add new features and support for more image formats, > without > > being constrained by Python's release cycle. > > > > Most of the modules are in pure Python and can be easily packaged. Some > > depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since > `audioop`_ > > does not depend on any external libraries, it can be shipped in as binary > > wheels with some effort. Other C modules can be replaced with ctypes or > cffi. > > For example I created `legacycrypt >`_ > > with ``_crypt`` extension reimplemented with a few lines of ctypes code. > > > > > > Discussions > > =========== > > > > * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. > > * Berker Peksag proposed to deprecate and removed *msilib*. > > * Brett Cannon recommended to delay active deprecation warnings and > removal > > of modules like *imp* until Python 3.10. Version 3.8 will be released > > shortly before Python 2 reaches end of lifetime. A delay reduced churn > for > > users that migrate from Python 2 to 3.8. > > * Brett also came up with the idea to keep lib2to3. The package is useful > > for other purposes, e.g. `black `_ > uses > > it to reformat Python code. > > * At one point, distutils was mentioned in the same sentence as this PEP. > > To avoid lengthy discussion and delay of the PEP, I decided against > dealing > > with distutils. Deprecation of the distutils package will be handled by > > another PEP. > > * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) > > convinced me to keep the `wave`_ module. [4]_ > > * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ > > * Andrew Svetlov mentioned the ``socketserver`` module is questionable. > > However it's used to implement ``http.server`` and ``xmlrpc.server``. > The > > stdlib doesn't have a replacement for the servers, yet. > > > > > > Update history > > ============== > > > > Update 1 > > -------- > > > > * Deprecate `parser`_ module > > * Keep `fileinput`_ module > > * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad > > * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ > modules > > * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections > now > > list suitable substitutions. > > * Mention that ``socketserver`` is going to stay for ``http.server`` and > > ``xmlrpc.server`` > > * The future maintenance section now states that the deprecated modules > > may be adopted by Python community members. > > > > References > > ========== > > > > .. [1] > https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free > > .. [2] https://man.openbsd.org/ossaudio > > .. [3] > https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ > > .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 > > .. [5] https://twitter.com/dabeaz/status/1130278844479545351 > > .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html > > .. [7] > https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 > > > > > > Copyright > > ========= > > > > This document has been placed in the public domain. > > > > > > > > .. > > Local Variables: > > mode: indented-text > > indent-tabs-mode: nil > > sentence-end-double-space: t > > fill-column: 70 > > coding: utf-8 > > End: > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/python-python-dev%40m.gmane.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Tue May 21 11:35:32 2019 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 21 May 2019 08:35:32 -0700 Subject: [Python-Dev] PEP 544 and dunder methods Message-ID: [originally I sent this to typing-sig but I guess it got caught up in moderation.] In PEP 554 [1] it says: - Implement metaclass functionality to detect whether a class is a protocol or not. Add a class attribute _is_protocol = True if that is the case. Verify that a protocol class only has protocol base classes in the MRO (except for object). The phrase "except for object" implies that having "type" in the MRO would not be allowed. Was that intentional? Perhaps I've misunderstood semantics with typing stuff (I haven't followed much of the discussion), but there seems to be a disconnect with how special ("dunder") methods are looked up (i.e. on classes). Effectively, an object's *class* has to implement the relevant "protocol". So disallowing "type" in the MRO means not using the PEP's functionality for "special method" protocols (as well as metaclasses in general). Have I understood that right? Is that on purpose? Thanks for working on this, BTW. It's a nice addition. :) -eric [1] https://www.python.org/dev/peps/pep-0544/#implementation-details -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 21 11:48:58 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 17:48:58 +0200 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> Message-ID: On 21/05/2019 17.35, Guido van Rossum wrote: > OK, sounds like nntplib can stay ? this time.? > > On Tue, May 21, 2019 at 08:33 Antoine Pitrou > wrote: > > > As I said, if the main annoyance with nntplib is the sporadic test > failures, then the relevant tests can be disabled on CI. > > NNTP itself is still used, even if less and less. I'd rather have it moved outside the core and maintained by people that actually care about NNTP support. Greg, you proposed the deprecation of nntplib. Any comments? Christian From steve at holdenweb.com Tue May 21 12:04:49 2019 From: steve at holdenweb.com (Steve Holden) Date: Tue, 21 May 2019 17:04:49 +0100 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> Message-ID: Good point! On Tue, May 21, 2019 at 2:01 PM Paul Moore wrote: > On Tue, 21 May 2019 at 13:50, Steve Holden wrote: > > > > That seems entirely reasonable. I wonder if the larger community could > somehow form an organization (the Dead Parrot SIG?) that would at least > curate and monitor efforts to ensure their continued utility? > > I have no idea whether there is enough community support to make this > work, but regardless, I strongly support a SIG by that name. It may > actually be even *more* amusing if it doesn't have enough members to > make it viable :-) > > Paul > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Tue May 21 12:08:14 2019 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Tue, 21 May 2019 23:08:14 +0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: On Tue, 21 May 2019 at 21:13, Christian Heimes wrote: > crypt > ~~~~~ > > The `crypt `_ module > implements > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor > quality and insecure. Users are discouraged to use them. > > * The module is not available on Windows. Cross-platform application need > an alternative implementation any way. > * Only DES encryption is guarenteed to be available. DES has an extremely > limited key space of 2**56. > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > algorithm that is still secure. However it's in glibc and therefore not > commonly available on Linux. > * Depending on the platform, the ``crypt`` module is not thread safe. Only > implementations with ``crypt_r(3)`` are thread safe. > * The module was never useful to interact with system user and password > databases. This is actually not true. Their main use case is to compare passwords against the shadowed password db: https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L413 A quick search on searchcode.com shows both spwd and crypt modules are used. I am no security expert (and I wasn?t aware they are insecure until now, since the doc doesn?t mention it) but I would prefer seeing these 2 fixed or improved rather than bluntly removed. > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 21 12:09:38 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 18:09:38 +0200 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <20190521173112.503f4c91@fsol> References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> Message-ID: On 21/05/2019 17.31, Antoine Pitrou wrote: > > As I said, if the main annoyance with nntplib is the sporadic test > failures, then the relevant tests can be disabled on CI. > > NNTP itself is still used, even if less and less. I don't like the idea to drop a third of the test cases for nntplib -- and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library. Even a deprecation in 3.8 means that the module has to be maintained until EOL of 3.9, which is around 2026. Christian From christian at python.org Tue May 21 12:26:54 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 18:26:54 +0200 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: On 21/05/2019 18.08, Giampaolo Rodola' wrote: > > > On Tue, 21 May 2019 at 21:13, Christian Heimes > wrote: > > crypt > ~~~~~ > > The `crypt `_ module implements > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor > quality and insecure. Users are discouraged to use them. > > * The module is not available on Windows. Cross-platform application need > ? an alternative implementation any way. > * Only DES encryption is guarenteed to be available. DES has an extremely > ? limited key space of 2**56. > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > ? SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > ? algorithm that is still secure. However it's in glibc and therefore not > ? commonly available on Linux. > * Depending on the platform, the ``crypt`` module is not thread safe. Only > ? implementations with ``crypt_r(3)`` are thread safe. > * The module was never useful to interact with system user and password > ? databases. > > > This is actually not true. Their main use case is to compare passwords against the shadowed password db: > https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L413 > A quick search on searchcode.com shows both spwd and crypt modules are used. I am no security expert (and I wasn?t aware they are insecure until now, since the doc doesn?t mention it) but I would prefer seeing these 2 fixed or improved rather than bluntly removed.? No, the statement is correct. I may have to explain this even further. The approach in pyftpdlib is the wrong and IMO deserves a CVE. The crypt() + spwd() approach is flawed on multiple levels. For example it bypasses account restriction, access control, and login session. It also requires you to run the service as root and with SELinux disabled or an unconfined context -- a bad combination. There is only one correct way to perform a credential check: use PAM. spwd can't be fixed. It could only be replaced with a completely different API that wraps PAM and Windows's authentication API. Christian PS: Authentication, authorization, and identity management are part of my day job at Red Hat. From v+python at g.nevcal.com Tue May 21 12:29:45 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 21 May 2019 09:29:45 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> Message-ID: <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> On 5/20/2019 2:20 PM, Christian Heimes wrote: > On 20/05/2019 23.12, Andrew Svetlov wrote: >> socketserver.py is also questionable > I briefly though about the module, but didn't consider it for removal. The http.server, xmlrpc.server, and logging configuration server are implemented on top of the socketserver. I don't want to remove the socketserver module without a suitable replacement for http.server in the standard library. But http.server could be on the remove list too... it gets mighty little support, has very little functionality, and implements a CGI interface (although that also has very little functionality), and you have the CGI tools on the remove list, rendering the CGI interface implemented by http.server less easily usable. Further, it doesn't directly support https:, and browsers are removing/reducing support for http:. I can't speak to xmlrpc or logging configuration. -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 21 12:42:49 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 18:42:49 +0200 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: <0a9d13be-af4c-0e66-0d70-6a920f22dc46@python.org> On 21/05/2019 18.08, Giampaolo Rodola' wrote: > > > On Tue, 21 May 2019 at 21:13, Christian Heimes > wrote: > > crypt > ~~~~~ > > The `crypt `_ module implements > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor > quality and insecure. Users are discouraged to use them. > > * The module is not available on Windows. Cross-platform application need > ? an alternative implementation any way. > * Only DES encryption is guarenteed to be available. DES has an extremely > ? limited key space of 2**56. > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > ? SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > ? algorithm that is still secure. However it's in glibc and therefore not > ? commonly available on Linux. > * Depending on the platform, the ``crypt`` module is not thread safe. Only > ? implementations with ``crypt_r(3)`` are thread safe. > * The module was never useful to interact with system user and password > ? databases. > > > This is actually not true. Their main use case is to compare passwords against the shadowed password db: > https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L413 > A quick search on searchcode.com shows both spwd and crypt modules are used. I am no security expert (and I wasn?t aware they are insecure until now, since the doc doesn?t mention it) but I would prefer seeing these 2 fixed or improved rather than bluntly removed.? PS: "pw1 != pw2" on line 418 is also vulnerable to side channel attack. You must use a constant timing comparison operator like hmac.compare_digest() to compare password digests. https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L418 From guido at python.org Tue May 21 13:12:50 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 21 May 2019 10:12:50 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> Message-ID: I still like http.server for quick temporary hacks where I want to be able to point a browser at some code I wrote 5 minutes ago and that I plan to discard in an hour. Usually it's running at localhost:8000. Remembering how to use Django, flask or Tornado seems overkill for that purpose. On Tue, May 21, 2019 at 9:39 AM Glenn Linderman wrote: > On 5/20/2019 2:20 PM, Christian Heimes wrote: > > On 20/05/2019 23.12, Andrew Svetlov wrote: > > socketserver.py is also questionable > > I briefly though about the module, but didn't consider it for removal. The http.server, xmlrpc.server, and logging configuration server are implemented on top of the socketserver. I don't want to remove the socketserver module without a suitable replacement for http.server in the standard library. > > > But http.server could be on the remove list too... it gets mighty little > support, has very little functionality, and implements a CGI interface > (although that also has very little functionality), and you have the CGI > tools on the remove list, rendering the CGI interface implemented by > http.server less easily usable. > > Further, it doesn't directly support https:, and browsers are > removing/reducing support for http:. > > I can't speak to xmlrpc or logging configuration. > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From edwin at 211mainstreet.net Tue May 21 13:13:30 2019 From: edwin at 211mainstreet.net (Edwin Zimmerman) Date: Tue, 21 May 2019 13:13:30 -0400 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> Message-ID: <005401d50ff8$839e66c0$8adb3440$@211mainstreet.net> On Tuesday, May 21, 2019 12:30 PM Glenn Linderman wrote On 5/20/2019 2:20 PM, Christian Heimes wrote: On 20/05/2019 23.12, Andrew Svetlov wrote: socketserver.py is also questionable I briefly though about the module, but didn't consider it for removal. The http.server, xmlrpc.server, and logging configuration server are implemented on top of the socketserver. I don't want to remove the socketserver module without a suitable replacement for http.server in the standard library. But http.server could be on the remove list too... it gets mighty little support, has very little functionality, and implements a CGI interface (although that also has very little functionality), and you have the CGI tools on the remove list, rendering the CGI interface implemented by http.server less easily usable. Further, it doesn't directly support https:, and browsers are removing/reducing support for http:. All the same, I?d hate to see http.server leave. It?s doubtful that anyone uses the thing for a production server, but I use it all the time for low-volume development stuff on Windows. -------------- next part -------------- A non-text attachment was scrubbed... Name: winmail.dat Type: application/ms-tnef Size: 6358 bytes Desc: not available URL: From v+python at g.nevcal.com Tue May 21 13:42:20 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 21 May 2019 10:42:20 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> Message-ID: <303943c0-0c48-ec35-cbd9-cbc357dd3e7e@g.nevcal.com> On 5/21/2019 10:12 AM, Guido van Rossum wrote: > I still like http.server for quick temporary hacks where I want to be > able to point a browser at some code I wrote 5 minutes ago and that I > plan to discard in an hour. Usually it's running at localhost:8000. > Remembering how to use Django, flask or Tornado seems overkill for > that purpose. After maintaining my own version of http.server to fix or workaround some of its deficiencies for some years, I discovered bottle.py. It has far more capability, is far better documented, and is just as quick to deploy. While I haven't yet converted all past projects to use bottle.py, it will likely happen in time, unless something even simpler to use is discovered, although I can hardly imagine that happening. I agree that Django, flask, and Tornado have bigger learning curves, due to having their own philosophies on how things should be done... I looked at each on my way to finding bottle.py, and in each case decided their philosophies, while probably useful to people that like or need the capabilities offered, were excessive to learn if the capabilities were not needed. Providing http.server as an included battery is a disservice to novice users, because they might try to use it for something real. > On Tue, May 21, 2019 at 9:39 AM Glenn Linderman > wrote: > > On 5/20/2019 2:20 PM, Christian Heimes wrote: >> On 20/05/2019 23.12, Andrew Svetlov wrote: >>> socketserver.py is also questionable >> I briefly though about the module, but didn't consider it for removal. The http.server, xmlrpc.server, and logging configuration server are implemented on top of the socketserver. I don't want to remove the socketserver module without a suitable replacement for http.server in the standard library. > > But http.server could be on the remove list too... it gets mighty > little support, has very little functionality, and implements a > CGI interface (although that also has very little functionality), > and you have the CGI tools on the remove list, rendering the CGI > interface implemented by http.server less easily usable. > > Further, it doesn't directly support https:, and browsers are > removing/reducing support for http:. > > I can't speak to xmlrpc or logging configuration. > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > > -- > --Guido van Rossum (python.org/~guido ) > /Pronouns: he/him/his //(why is my pronoun?here?)/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Tue May 21 13:45:10 2019 From: bcannon at gmail.com (Brett Cannon) Date: Tue, 21 May 2019 10:45:10 -0700 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: On Mon., May 20, 2019, 16:00 Terry Reedy, wrote: > On 5/20/2019 11:55 AM, Guido van Rossum wrote: > > On Thu, May 16, 2019 at 3:57 PM Steve Dower > > wrote: > > > > [...] > > We still have the policy of not removing modules that exist in the > > Python 2 standard library. But 3.9 won't be covered by that :) > > > > I didn't even remember that. Where's that written down? > > AFAIK, there has been no BDFL pronouncement or coredev vote. It's in PEP 4 since at least Python 3.5: https://www.python.org/dev/peps/pep-0004/#for-modules-existing-in-both-python-2-7-and-python-3-5 . -Brett But it has > become somewhat of a consensus as a result of discussions in multliple > threads about removing modules going back to the 3.4 era. > > PEP 594, just posted, proposes to codify a 'soft' (or perhaps 'slow') > version of the policy: doc deprecation, code deprecation, and code > removal in 3.8, 3.9, 3.10. > > FWIW I am strongly in favor of getting rid of the `parser` module, in > > 3.8 if we can, otherwise in 3.9 (after strong deprecation in 3.8). > > You could request that 'parser' be specifically excluded from the PEP, > along with 'distutils' (and effectively anything else not specifically > named). Or request that it be included, but with an accelerated > schedule. I have a vague idea of why you think it harmful to keep it > around, but a reminder would not hurt ;-). > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Tue May 21 13:37:06 2019 From: phd at phdru.name (Oleg Broytman) Date: Tue, 21 May 2019 19:37:06 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> Message-ID: <20190521173706.ctccjhct4fynjg33@phdru.name> ``http.server`` is used in ``pydoc``: https://github.com/python/cpython/blob/ccb7ca728e09b307f9e9fd36ec40353137e68a3b/Lib/pydoc.py#L2236 On Tue, May 21, 2019 at 10:12:50AM -0700, Guido van Rossum wrote: > I still like http.server for quick temporary hacks where I want to be able > to point a browser at some code I wrote 5 minutes ago and that I plan to > discard in an hour. Usually it's running at localhost:8000. Remembering how > to use Django, flask or Tornado seems overkill for that purpose. > > On Tue, May 21, 2019 at 9:39 AM Glenn Linderman > wrote: > > > On 5/20/2019 2:20 PM, Christian Heimes wrote: > > > > On 20/05/2019 23.12, Andrew Svetlov wrote: > > > > socketserver.py is also questionable > > > > I briefly though about the module, but didn't consider it for removal. The http.server, xmlrpc.server, and logging configuration server are implemented on top of the socketserver. I don't want to remove the socketserver module without a suitable replacement for http.server in the standard library. > > > > > > But http.server could be on the remove list too... it gets mighty little > > support, has very little functionality, and implements a CGI interface > > (although that also has very little functionality), and you have the CGI > > tools on the remove list, rendering the CGI interface implemented by > > http.server less easily usable. > > > > Further, it doesn't directly support https:, and browsers are > > removing/reducing support for http:. > > > > I can't speak to xmlrpc or logging configuration. > -- > --Guido van Rossum (python.org/~guido) > *Pronouns: he/him/his **(why is my pronoun here?)* > Oleg. -- Oleg Broytman https://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From tjreedy at udel.edu Tue May 21 13:59:56 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 21 May 2019 13:59:56 -0400 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190521130111.GJ4221@ando.pearwood.info> References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: On 5/21/2019 9:01 AM, Steven D'Aprano wrote: ... > Many Python users don't have the privilege of being able to install > arbitrary, unvetted packages from PyPI. They get to use only packages > from approved vendors, including the stdlib, what they write themselves, > and nothing else. Please don't dismiss this part of the Python community > just because they don't typically hang around in the same forums we do. ... The problem with this argument, taken by itself, it that it would argue for adding to the stdlib 100s or 1000s of modules or packages that would be useful to many more people than the modules proposed to be dropped. That would require multiplying the number of core developers by perhaps 10. We have thus far decided to not do this. We have instead left sumo distributions with vetted (?) additions to other groups. But as it is, the stdlib is, as far as I last knew, richer than, for instance, the standard C stdlib. Given that presence in the stdlib is restricted, what should be included. Is mere inertia a sufficient reason? I don't think it should be. -- Terry Jan Reedy From bcannon at gmail.com Tue May 21 14:07:08 2019 From: bcannon at gmail.com (Brett Cannon) Date: Tue, 21 May 2019 11:07:08 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <2215566.ODhDVceCri@finnegan> Message-ID: On Tue., May 21, 2019, 04:25 Victor Stinner, wrote: > Le mar. 21 mai 2019 ? 13:18, Andr? Malo a ?crit : > > There's software in production using both. (It doesn't mean it's on pypi > or > > even free software). > > > > What would be the maintenance burden of those modules anyway? (at least > for > > nntp, I guess it's not gonna change). > > The maintenance burden is real even if it's not visible. For example, > test_nntplib is causing frequently issues on our CI: > > https://bugs.python.org/issue19756 > https://bugs.python.org/issue19613 > https://bugs.python.org/issue31850 > > It's failing frequently since 2013, and nobody managed to come with a > fix.. in 6 years. > > There are 11 open issues with "nntp" in their title (8 with exactly > "nntplib" in their title). > I would be curious to know how many feature PRs there might exist as that also is a burden for any of these modules beyond bugs and any general upkeep we do (e.g. someone on Twitter argued that because someone merged a new feature that means that a module should stay, but to me that just says one person cared enough to write a PR and one core dev viewed it as harmless to merge which isn't necessarily representative of usage). -Brett P.S. I have an idea about under-supported modules in the stdlib and feature requests, but it's too early to share (nor the appropriate thread of discussion). > test_nntplib uses the public server news.trigofacile.com which is > operated by Julien ?LIE. Two years ago, Julien asked me if there is > any plan to support the NNTP "COMPRESS" command. > > Victor > > > > > > -- > Night gathers, and now my watch begins. It shall not end until my death. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 21 14:15:36 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 20:15:36 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> Message-ID: <66dc8643-6c91-ac1f-6ff8-ebce4de39a5b@python.org> On 21/05/2019 18.29, Glenn Linderman wrote: > On 5/20/2019 2:20 PM, Christian Heimes wrote: >> On 20/05/2019 23.12, Andrew Svetlov wrote: >>> socketserver.py is also questionable >> I briefly though about the module, but didn't consider it for removal. The http.server, xmlrpc.server, and logging configuration server are implemented on top of the socketserver. I don't want to remove the socketserver module without a suitable replacement for http.server in the standard library. > > But http.server could be on the remove list too... it gets mighty little support, has very little functionality, and implements a CGI interface (although that also has very little functionality), and you have the CGI tools on the remove list, rendering the CGI interface implemented by http.server less easily usable. > > Further, it doesn't directly support https:, and browsers are removing/reducing support for http:. > > I can't speak to xmlrpc or logging configuration. Hi, thanks for bringing this topic up. Initially I considered http.server, too. But as Guido put it, it's both used and useful for local testing and quick hacks. I'm already facing opposition for modules that are less controversial and useful than http.server, too. I have two remarks: 1) The http.server does not act as a CGI server by default. In CGI server mode, it does not depend on the cgi module. 2) The lack of HTTPS support is not a major problem for connections on localhost. There is a RFC draft to consider any connection to "localhost" as secure, https://tools.ietf.org/html/draft-west-let-localhost-be-localhost-06 Christian From tjreedy at udel.edu Tue May 21 14:15:28 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 21 May 2019 14:15:28 -0400 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: On 5/21/2019 10:12 AM, Christian Heimes wrote: > ------------------------------------------------------------------- > PEP: 594 > Title: Removing dead batteries from the standard library 'dead' seems controversial. 'nearly useless' should be less so. I think 'after 2.7 end-of-life' is worth adding, as the schedule hinges around that. Result: "Removing nearly useless batteries from the standard library after 2.7 end-of-life." > Abstract > ======== > Rationale > ========= Part of the point of the PEP is to gather together nearly all proposals in one place so users can respond in a timely fashion. > Deprecation schedule > ==================== I think it worth adding that some modules were deprecated years ago but kept until after 2.7 end-of-life to ease 3.x transition. Removing modules is not a sudden new idea. Also, the proposed schedule (with one exception) is a slower alternative than proposals to add a deprecation warning (if not one already) in 3.8 and remove in 3.9. -- Terry Jan Reedy From solipsis at pitrou.net Tue May 21 14:15:54 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 21 May 2019 20:15:54 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: <20190521201554.67270532@fsol> On Tue, 21 May 2019 13:59:56 -0400 Terry Reedy wrote: > On 5/21/2019 9:01 AM, Steven D'Aprano wrote: > ... > > Many Python users don't have the privilege of being able to install > > arbitrary, unvetted packages from PyPI. They get to use only packages > > from approved vendors, including the stdlib, what they write themselves, > > and nothing else. Please don't dismiss this part of the Python community > > just because they don't typically hang around in the same forums we do. > ... > > The problem with this argument, taken by itself, it that it would argue > for adding to the stdlib 100s or 1000s of modules or packages that would > be useful to many more people than the modules proposed to be dropped. > That would require multiplying the number of core developers by perhaps > 10. We have thus far decided to not do this. > > We have instead left sumo distributions with vetted (?) additions to > other groups. But as it is, the stdlib is, as far as I last knew, > richer than, for instance, the standard C stdlib. > > Given that presence in the stdlib is restricted, what should be > included. Is mere inertia a sufficient reason? I don't think it should be. One fundamental thing the stdlib provides is stability. If you use an API from the stdlib today, chances are it'll still work in 10 years, even on the most recent Python version. Good luck relying on that with most third-party packages. But if stdlib modules can disappear when we decide they don't get enough usage, the stability promise disappears as well. Regards Antoine. From christian at python.org Tue May 21 14:16:49 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 20:16:49 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <8ade5b62-75d1-589e-3b7b-0678f5e239ec@python.org> <0790e769-ff11-ca99-7252-662d9fedeceb@gmail.com> Message-ID: <9e4b8d2b-11e8-3ebe-60c1-84ff50e3b915@python.org> On 21/05/2019 16.46, Guido van Rossum wrote: > +1. Let's keep colorsys then. I let colorsys off the hock, https://github.com/python/peps/pull/1070 Thanks for your feedback, Walter and Petr! Christian From bcannon at gmail.com Tue May 21 14:16:38 2019 From: bcannon at gmail.com (Brett Cannon) Date: Tue, 21 May 2019 11:16:38 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> Message-ID: On Tue., May 21, 2019, 09:10 Christian Heimes, wrote: > On 21/05/2019 17.31, Antoine Pitrou wrote: > > > > As I said, if the main annoyance with nntplib is the sporadic test > > failures, then the relevant tests can be disabled on CI. > > > > NNTP itself is still used, even if less and less. > > I don't like the idea to drop a third of the test cases for nntplib -- and > the 30% that actually test that Python's NNTP library can talk to an actual > NNTP server. IMHO it's more beneficial for core development and for nntplib > to have it maintained by somebody that cares about the library. > Ditto for me. If the code isn't going to be properly tested then it's already unsupported, at which point we're back to wanting to deprecate it. -Brett > Even a deprecation in 3.8 means that the module has to be maintained until > EOL of 3.9, which is around 2026. > > Christian > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Tue May 21 14:18:07 2019 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 22 May 2019 01:18:07 +0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: On Tue, 21 May 2019 at 23:26, Christian Heimes wrote: > On 21/05/2019 18.08, Giampaolo Rodola' wrote: > > > > > > On Tue, 21 May 2019 at 21:13, Christian Heimes > wrote: > > > > crypt > > ~~~~~ > > > > The `crypt `_ module > implements > > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, > of poor > > quality and insecure. Users are discouraged to use them. > > > > * The module is not available on Windows. Cross-platform application > need > > an alternative implementation any way. > > * Only DES encryption is guarenteed to be available. DES has an > extremely > > limited key space of 2**56. > > * MD5, salted SHA256, salted SHA512, and Blowfish are optional > extension. > > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the > only > > algorithm that is still secure. However it's in glibc and > therefore not > > commonly available on Linux. > > * Depending on the platform, the ``crypt`` module is not thread > safe. Only > > implementations with ``crypt_r(3)`` are thread safe. > > * The module was never useful to interact with system user and > password > > databases. > > > > > > This is actually not true. Their main use case is to compare passwords > against the shadowed password db: > > > https://github.com/giampaolo/pyftpdlib/blob/ee7b36c701b78b2d36e938c42d08dbfbad55a34f/pyftpdlib/authorizers.py#L413 > > A quick search on searchcode.com shows both > spwd and crypt modules are used. I am no security expert (and I wasn?t > aware they are insecure until now, since the doc doesn?t mention it) but I > would prefer seeing these 2 fixed or improved rather than bluntly removed. > > No, the statement is correct. I may have to explain this even further. > > The approach in pyftpdlib is the wrong and IMO deserves a CVE. The crypt() > + spwd() approach is flawed on multiple levels. For example it bypasses > account restriction, access control, and login session. It also requires > you to run the service as root and with SELinux disabled or an unconfined > context -- a bad combination. There is only one correct way to perform a > credential check: use PAM. > > spwd can't be fixed. It could only be replaced with a completely different > API that wraps PAM and Windows's authentication API. > > Christian > > PS: Authentication, authorization, and identity management are part of my > day job at Red Hat. > Got it. I had no idea. Since you mentioned the CVE it looks like spwd/crypt doc deserve a warning. This is probably out of the scope of the PEP, but I wonder if the 3 third-party alternatives mentioned in the PEP are mature enough and could be evaluated for stdlib inclusion (the part re. PAM / password-checking at least). Perhaps spwd/crypt could be deprecated in 3.8 and the alternative added in 3.9 before the 3.10 removal. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From antoine at python.org Tue May 21 14:19:56 2019 From: antoine at python.org (Antoine Pitrou) Date: Tue, 21 May 2019 20:19:56 +0200 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> Message-ID: <27ba695b-3219-a628-acef-4e0af649e2de@python.org> Le 21/05/2019 ? 20:16, Brett Cannon a ?crit?: > > > On Tue., May 21, 2019, 09:10 Christian Heimes, > wrote: > > On 21/05/2019 17.31, Antoine Pitrou wrote: > > > > As I said, if the main annoyance with nntplib is the sporadic test > > failures, then the relevant tests can be disabled on CI. > > > > NNTP itself is still used, even if less and less. > > I don't like the idea to drop a third of the test cases for nntplib > -- and the 30% that actually test that Python's NNTP library can > talk to an actual NNTP server. IMHO it's more beneficial for core > development and for nntplib to have it maintained by somebody that > cares about the library. > > > Ditto for me. If the code isn't going to be properly tested then it's > already unsupported, at which point we're back to wanting to deprecate it. So I should never have added those tests and then we wouldn't be talking about removing nntplib. Regards Antoine. From christian at python.org Tue May 21 14:34:18 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 20:34:18 +0200 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: <3ff5c895-92b8-2276-6a06-617c610ce589@python.org> On 21/05/2019 20.18, Giampaolo Rodola' wrote: > No, the statement is correct. I may have to explain this even further. > > The approach in pyftpdlib is the wrong and IMO deserves a CVE. The crypt() + spwd() approach is flawed on multiple levels. For example it bypasses account restriction, access control, and login session. It also requires you to run the service as root and with SELinux disabled or an unconfined context -- a bad combination. There is only one correct way to perform a credential check: use PAM. > > spwd can't be fixed. It could only be replaced with a completely different API that wraps PAM and Windows's authentication API. > > Christian > > PS: Authentication, authorization, and identity management are part of my day job at Red Hat. > > > Got it. I had no idea. Since you mentioned the CVE it looks like spwd/crypt doc deserve a warning. This is probably out of the scope of the PEP, but I wonder if the 3 third-party alternatives mentioned in the PEP are mature enough and could be evaluated for stdlib inclusion (the part re. PAM / password-checking at least). Perhaps spwd/crypt could be deprecated in 3.8 and the alternative added in 3.9 before the 3.10 removal. Sorry, I didn't even occur to me that anybody was still using spwd. I would have added a warning much earlier. There is now https://bugs.python.org/issue36997 to track the problem. Could you do me a favor and open a feature request? From guido at python.org Tue May 21 14:35:28 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 21 May 2019 11:35:28 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <66dc8643-6c91-ac1f-6ff8-ebce4de39a5b@python.org> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <66dc8643-6c91-ac1f-6ff8-ebce4de39a5b@python.org> Message-ID: On Tue, May 21, 2019 at 11:17 AM Christian Heimes wrote: > I'm already facing opposition for modules that are less controversial and > useful than http.server, too. There's another argument here. This is an "omnibus" PEP, meaning it proposes many unrelated changes. In order to get a consensus to pass the PEP, it may be necessary to compromise. IOW I would recommend removing modules from the PEP that bring up strong opposition, *even* if you yourself feel strongly that those modules should be removed. The vast majority of modules on the list hasn't elicited any kind of feedback at all -- those are clearly safe to remove (many people are probably, like myself, hard-pressed to remember what they do). I'm not saying drop anything from the list that elicits any pushback, but once the debate has gone back and forth twice, it may be a hint that a module still has fans. Threatening to open a CVE is more likely to reduce support for the PEP than it is to convince anyone. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 21 15:00:10 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 21 May 2019 12:00:10 -0700 Subject: [Python-Dev] PEP 544 and dunder methods In-Reply-To: References: Message-ID: (I did see your post in the moderation queue and let it through -- however the mypy team, which would be most responsible for answering your question, has been busy fighting some fires.) I'm trying to understand what the PEP is saying and how you're interpreting, and I'm not getting very far. I know for a fact that Protocols work great with dunder methods. In fact the very first example in the PEP uses `__len__` and `__iter__`, and there are several other examples using `__iter__`, so I also know that this isn't an accident of mypy's implementation. And mypy knows about the special rules for dunder method lookup. So what's left to figure out is what the bullet you quote means, and whether it should be clarified, or whether it is literally correct (about what the implementation does). My guess, without delving into the implementation, is that a Protocol is *always* about the class, and that this is entirely a red herring. Looking at the implementation (e.g. the Python 3 implementation at https://github.com/python/typing/blob/master/typing_extensions/src_py3/typing_extensions.py#L1077) doesn't give me much clarity. The code seems to be highly optimized and not so readable. But I do see, at line 1159, what looks like the check that the bullet refers to. The point here is simply to make sure that a protocol doesn't inherit from a non-protocol class. Besides `object` there are several other exceptions that appear to me to be related to internals of the runtime typing implementation. If you are still not satisfied, I recommend that you try to construct a counter-example and test it against the typing_extensions modules in Python 3.7 or 3.8. Thanks! --Guido On Tue, May 21, 2019 at 8:40 AM Eric Snow wrote: > [originally I sent this to typing-sig but I guess it got caught up in > moderation.] > > In PEP 554 [1] it says: > > > - Implement metaclass functionality to detect whether a class is a > protocol or not. Add a class attribute _is_protocol = True if that is > the case. Verify that a protocol class only has protocol base classes in > the MRO (except for object). > > The phrase "except for object" implies that having "type" in the MRO would > not be allowed. Was that intentional? > > Perhaps I've misunderstood semantics with typing stuff (I haven't followed > much of the discussion), but there seems to be a disconnect with how > special ("dunder") methods are looked up (i.e. on classes). Effectively, > an object's *class* has to implement the relevant "protocol". So > disallowing "type" in the MRO means not using the PEP's functionality for > "special method" protocols (as well as metaclasses in general). Have I > understood that right? Is that on purpose? > > Thanks for working on this, BTW. It's a nice addition. :) > > -eric > > > [1] https://www.python.org/dev/peps/pep-0544/#implementation-details > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue May 21 15:12:03 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 21:12:03 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <66dc8643-6c91-ac1f-6ff8-ebce4de39a5b@python.org> Message-ID: On 21/05/2019 20.35, Guido van Rossum wrote: > On Tue, May 21, 2019 at 11:17 AM Christian Heimes > wrote: > > I'm already facing opposition for modules that are less controversial and useful than http.server, too. > > > There's another argument here. This is an "omnibus" PEP, meaning it proposes many unrelated changes. In order to get a consensus to pass the PEP, it may be necessary to compromise. IOW I would recommend removing modules from the PEP that bring up strong opposition, *even* if you yourself feel strongly that those modules should be removed. > > The vast majority of modules on the list hasn't elicited any kind of feedback at all -- those are clearly safe to remove (many people are probably, like myself, hard-pressed to remember what they do). I'm not saying drop anything from the list that elicits any pushback, but once the debate has gone back and forth twice, it may be a hint that a module still has fans. Threatening to open a CVE is more likely to reduce support for the PEP than it is to convince anyone. It was not a threat, but an illustration how critical the flaw with spwd + crypt is. The approach performs only authentication and completely bypasses any authorization. It does not take any login restrictions into account like account enabled flag, host/service based access control, IP restriction, credential strength, and so on. I would give the issue a CVSS rating between 8.3 (high) to 9.6 (critical), perhaps CVSS:3.0/AV:N/AC:L/PR:L/UI:N/S:U/C:H/I:H/A:L By the way, Giampaolo and I have known each other for many years. I know that he'll address the issue and file a CVE himself. From nas-python at arctrix.com Tue May 21 15:59:48 2019 From: nas-python at arctrix.com (Neil Schemenauer) Date: Tue, 21 May 2019 13:59:48 -0600 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: <20190521195948.b4cklbgdzomyfpza@python.ca> On 2019-05-21, Terry Reedy wrote: > The problem with this argument, taken by itself, it that it would argue for > adding to the stdlib 100s or 1000s of modules or packages that would be > useful to many more people than the modules proposed to be dropped. I don't think it does. We are not talking about 100s or 1000s of modules. We are talking about modules which have been in Python's stdlib for years or decades. If I have a script that uses one of these modules and it gets removed, my script breaks. Installing it from PyPI is not really a great solution. We are going to be breaking working scripts just like if we add new language keywords, etc. I think we need to be extremely careful with trying to maintain backwards compatibility, at least as far as we reasonably can. The problem I have with this PEP is that I think it both too aggressive and too conservative at the same time. For almost all modules on the list, I'm sure there will be many people who are harmed by its removal. OTOH, having to maintain all of the modules in the stdlib is a heavy burden. Also, new users can be lured into using a module that is not really the best solution anymore. Here is an alternative, straw man, proposal. Split the CPython repo into two parts: - core Python: minimal possible stdlib - everything else When Python is released, provide installers for a Python that only includes the "core" part and a second installer that includes everything else. I realize this is more work for the release team. Hopefully with some scripting, it will not be too labour intensive. The core Python installer should become the recommended installer. People who need backwards compability with older versions of Python can download the big installer package. To help the people who need 100s or 1000s of extra PyPI packages, we could develop a tool that creates a "sumo" Python installer, grabbing packages from PyPI and building a installer package. To install that package, you would not need network access. That doesn't need to happen right away. Also, maybe other Python distributions can fill that need if core Python devs don't want to build it. Regards, Neil From steve.dower at python.org Tue May 21 16:30:12 2019 From: steve.dower at python.org (Steve Dower) Date: Tue, 21 May 2019 13:30:12 -0700 Subject: [Python-Dev] Python in next Windows 10 update Message-ID: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Hi all Just sharing this here because I think it's important for us to be aware of it - I'm not trying to promote or sell anything here :) (Those who were at the language summit have seen this already.) In the next Windows 10 update that starts rolling out today, we (Microsoft) have added "python.exe" and "python3.exe" commands that are installed on PATH *by default* and will open the Microsoft Store at the page where we (Python core team) publish our build. This makes it a 1-2 click process to get from a clean machine to having a usable Python install ("python.exe" -> opens Store -> "Get it Free" -> "python.exe" now works!) The associated blog post: https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/ Here are answers to a few questions that I assume will come up, at least from this audience that understands the issues better than most: * if someone had installed Python and put it on PATH with our installer, this new command *does not* interfere * if someone had manually modified their own PATH, they *may* see some interference (but we [Microsoft] decided this was an acceptable risk) * the Python 3.7 installed from the store will not auto-update to 3.8, but when 3.8 is released we (Microsoft) will update the redirect to point at it * if you pass arguments to the redirect command, it just exits with an error code - you only get the Store page if you run it without arguments * once the Store package is installed, the redirect command is replaced (this required a new feature in the OS). If you install with the regular installer and update PATH, or active a venv, it will add it *before* the redirect. So these scenarios should be all good. I'm happy to answer other questions here. The long-term contact for this integration is python (at) microsoft.com, which right now will come to me. And on a personal note, I'm very excited that we (Microsoft) got the approval to do this. Getting *anything* added to Windows is a big task, so it's a reflection of the popularity and support for Python that's growing within Microsoft that we were able to make this happen. That's due to every contributor, both to the core runtime and the ecosystem. I hope this will only help us improve the availability of Python for users and make it an easier choice for dev tasks in the future. Cheers, Steve From v+python at g.nevcal.com Tue May 21 15:23:38 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 21 May 2019 12:23:38 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <66dc8643-6c91-ac1f-6ff8-ebce4de39a5b@python.org> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <66dc8643-6c91-ac1f-6ff8-ebce4de39a5b@python.org> Message-ID: <45e08fce-8690-753a-1720-09b401f4cc6a@g.nevcal.com> On 5/21/2019 11:15 AM, Christian Heimes wrote: > On 21/05/2019 18.29, Glenn Linderman wrote: >> On 5/20/2019 2:20 PM, Christian Heimes wrote: >>> On 20/05/2019 23.12, Andrew Svetlov wrote: >>>> socketserver.py is also questionable >>> I briefly though about the module, but didn't consider it for removal. The http.server, xmlrpc.server, and logging configuration server are implemented on top of the socketserver. I don't want to remove the socketserver module without a suitable replacement for http.server in the standard library. >> But http.server could be on the remove list too... it gets mighty little support, has very little functionality, and implements a CGI interface (although that also has very little functionality), and you have the CGI tools on the remove list, rendering the CGI interface implemented by http.server less easily usable. >> >> Further, it doesn't directly support https:, and browsers are removing/reducing support for http:. >> >> I can't speak to xmlrpc or logging configuration. > Hi, > > thanks for bringing this topic up. Initially I considered http.server, too. But as Guido put it, it's both used and useful for local testing and quick hacks. I'm already facing opposition for modules that are less controversial and useful than http.server, too. Indeed. That's why they were created in the first place (http.server, and those other modules). But these days there are better alternatives. That's the point of the PEP, if I understand correctly... get rid of the junkware, point to the betterware, and avoid the need to learn the betterware later, because you wasted time and effort using the junkware at first, because it was "included". > I have two remarks: > > 1) The http.server does not act as a CGI server by default. In CGI server mode, it does not depend on the cgi module. CGIHTTPRequestHandler is part of the package. While it does not depend on the cgi module, its existence encourages the users to search for, find, and use the cgi and cgitb modules when using CGIHTTPRequestHandler. I certainly did in past projects, prior to finding bottle.py. > 2) The lack of HTTPS support is not a major problem for connections on localhost. There is a RFC draft to consider any connection to "localhost" as secure, https://tools.ietf.org/html/draft-west-let-localhost-be-localhost-06 Which certainly didn't exist when Chrome (noting the author's organization) and other browsers first started implementing feature restrictions for http, and while I haven't tested this since the RFC you mention was published, it was certainly a difficulty I faced when attempting to use features with security restrictions for local testing a couple years ago. > Christian -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue May 21 17:00:44 2019 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 21 May 2019 17:00:44 -0400 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: <40CD7FD8-EF5D-4CC2-8851-8AE120539B7F@trueblade.com> That?s great, Steve. Thanks for all of the work (by you and others) on this. -- Eric V. Smith True Blade Systems, Inc (301) 859-4544 > On May 21, 2019, at 4:30 PM, Steve Dower wrote: > > Hi all > > Just sharing this here because I think it's important for us to be aware of it - I'm not trying to promote or sell anything here :) (Those who were at the language summit have seen this already.) > > In the next Windows 10 update that starts rolling out today, we (Microsoft) have added "python.exe" and "python3.exe" commands that are installed on PATH *by default* and will open the Microsoft Store at the page where we (Python core team) publish our build. > > This makes it a 1-2 click process to get from a clean machine to having a usable Python install ("python.exe" -> opens Store -> "Get it Free" -> "python.exe" now works!) > > The associated blog post: > > https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/ > > Here are answers to a few questions that I assume will come up, at least from this audience that understands the issues better than most: > > * if someone had installed Python and put it on PATH with our installer, this new command *does not* interfere > * if someone had manually modified their own PATH, they *may* see some interference (but we [Microsoft] decided this was an acceptable risk) > * the Python 3.7 installed from the store will not auto-update to 3.8, but when 3.8 is released we (Microsoft) will update the redirect to point at it > * if you pass arguments to the redirect command, it just exits with an error code - you only get the Store page if you run it without arguments > * once the Store package is installed, the redirect command is replaced (this required a new feature in the OS). If you install with the regular installer and update PATH, or active a venv, it will add it *before* the redirect. So these scenarios should be all good. > > I'm happy to answer other questions here. The long-term contact for this integration is python (at) microsoft.com, which right now will come to me. > > And on a personal note, I'm very excited that we (Microsoft) got the approval to do this. Getting *anything* added to Windows is a big task, so it's a reflection of the popularity and support for Python that's growing within Microsoft that we were able to make this happen. That's due to every contributor, both to the core runtime and the ecosystem. I hope this will only help us improve the availability of Python for users and make it an easier choice for dev tasks in the future. > > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/eric%2Ba-python-dev%40trueblade.com From njs at pobox.com Tue May 21 17:00:37 2019 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 21 May 2019 14:00:37 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <303943c0-0c48-ec35-cbd9-cbc357dd3e7e@g.nevcal.com> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <303943c0-0c48-ec35-cbd9-cbc357dd3e7e@g.nevcal.com> Message-ID: On Tue, May 21, 2019 at 10:43 AM Glenn Linderman wrote: > After maintaining my own version of http.server to fix or workaround some of its deficiencies for some years, I discovered bottle.py. It has far more capability, is far better documented, and is just as quick to deploy. While I haven't yet converted all past projects to use bottle.py, it will likely happen in time, unless something even simpler to use is discovered, although I can hardly imagine that happening. bottle.py uses http.server for its local development mode (the one you see in their quickstart example at the top of their README). Same with flask, django, and probably a bunch of other frameworks. It's *very* widely used. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Tue May 21 17:04:42 2019 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 21 May 2019 14:04:42 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <2215566.ODhDVceCri@finnegan> Message-ID: On Tue, May 21, 2019 at 4:25 AM Victor Stinner wrote: > > Le mar. 21 mai 2019 ? 13:18, Andr? Malo a ?crit : > > There's software in production using both. (It doesn't mean it's on pypi or > > even free software). > > > > What would be the maintenance burden of those modules anyway? (at least for > > nntp, I guess it's not gonna change). > > The maintenance burden is real even if it's not visible. For example, > test_nntplib is causing frequently issues on our CI: > > https://bugs.python.org/issue19756 > https://bugs.python.org/issue19613 > https://bugs.python.org/issue31850 > > It's failing frequently since 2013, and nobody managed to come with a > fix.. in 6 years. If the tests don't work and the module is unmaintained, then maybe we should disable the tests and put a prominent notice in the docs saying that it's unmaintained and any use is at your own risk. It's not a pleasant thing to do, but if that's the reality of the situation then it's probably better to be up front about it than to stick our fingers in our ears and waste folks time with spurious test failures. And perhaps someone would actually step up to maintain it. -n -- Nathaniel J. Smith -- https://vorpus.org From christian at python.org Tue May 21 17:42:53 2019 From: christian at python.org (Christian Heimes) Date: Tue, 21 May 2019 23:42:53 +0200 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: On 21/05/2019 22.30, Steve Dower wrote: > Hi all > > Just sharing this here because I think it's important for us to be aware of it - I'm not trying to promote or sell anything here :) (Those who were at the language summit have seen this already.) > > In the next Windows 10 update that starts rolling out today, we (Microsoft) have added "python.exe" and "python3.exe" commands that are installed on PATH *by default* and will open the Microsoft Store at the page where we (Python core team) publish our build. > > This makes it a 1-2 click process to get from a clean machine to having a usable Python install ("python.exe" -> opens Store -> "Get it Free" -> "python.exe" now works!) > > The associated blog post: > > https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/ Thanks for doing this and lowering the entree barrier for every Windows user. :) Christian From vstinner at redhat.com Tue May 21 18:19:13 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 22 May 2019 00:19:13 +0200 Subject: [Python-Dev] Unrelated PRs linked to issues on bugs.python.org Message-ID: Hi, Since one or two weeks, I noticed that the bot which links GitHub pull requests to bugs.python.org issues started to link PRs to unrelated issues. Example: https://github.com/python/cpython/pull/13148 (merged 2 hours ago) just added to https://bugs.python.org/issue35363 (closed at the end of last year). I cannot find "35363" (bpo number) in https://github.com/python/cpython/pull/13148 nor in https://github.com/python/cpython/pull/13148/files Does anyone have an idea why unrelated PRs are added to issues? I don't know which bot is responsible for this task, nor who operates this bot. So in case of doubt, I chose to send a message to python-dev. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From vstinner at redhat.com Tue May 21 19:06:34 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 22 May 2019 01:06:34 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <2215566.ODhDVceCri@finnegan> Message-ID: Le mar. 21 mai 2019 ? 23:05, Nathaniel Smith a ?crit : > If the tests don't work and the module is unmaintained, then maybe we > should disable the tests and put a prominent notice in the docs saying > that it's unmaintained and any use is at your own risk. It's not a > pleasant thing to do, but if that's the reality of the situation then > it's probably better to be up front about it than to stick our fingers > in our ears and waste folks time with spurious test failures. And > perhaps someone would actually step up to maintain it. Well, I already did that in the past. Lib/test/test_ftplib.py: @skipUnless(False, "FIXME: bpo-32706") => skipped since 2018-01-29 Lib/test/test_nntplib.py: @unittest.skipIf(True, "FIXME: see bpo-32128") => skipped since 2017-11-25. Oh, I even closed https://bugs.python.org/issue32128. Well, I'm kind of tired of waiting for fixes in nntplib, so I decided to immediately close the issue. I'm quite sure that nobody will fix it "soon" :-) (hint: there FIXME is still there) I'm not sure about documenting the module as "unmaintained". Victor -- Night gathers, and now my watch begins. It shall not end until my death. From v+python at g.nevcal.com Tue May 21 19:11:46 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 21 May 2019 16:11:46 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <303943c0-0c48-ec35-cbd9-cbc357dd3e7e@g.nevcal.com> Message-ID: On 5/21/2019 2:00 PM, Nathaniel Smith wrote: > On Tue, May 21, 2019 at 10:43 AM Glenn Linderman wrote: >> After maintaining my own version of http.server to fix or workaround some of its deficiencies for some years, I discovered bottle.py. It has far more capability, is far better documented, and is just as quick to deploy. While I haven't yet converted all past projects to use bottle.py, it will likely happen in time, unless something even simpler to use is discovered, although I can hardly imagine that happening. > bottle.py uses http.server for its local development mode (the one you > see in their quickstart example at the top of their README). Same with > flask, django, and probably a bunch of other frameworks. It's *very* > widely used. > > -n > The source for bottle.py version 0.13-dev has an import for http.client, but not http.server. I hadn't tracked down every indirect dependency in the bottle.py source code, but it seems that if one uses the "default server" for bottle, that it is "wsgiref", imported from wsgiref.simple_server, and that in turn does import BaseHTTPRequestHandler and HTTPServer from http.server. It is the higher-level code in http.server that has significant deficiencies that have caused me problems over the years... a "SimpleHTTPRequestHandler" that is so simple it doesn't do POST, PUT or PASTE, a "CGIHTTPRequestHandler" that only implements part of the CGI protocol, only CGI support in POST, no support for PUT or PASTE, and no support for https, and not much bug fix activity in those areas. Maybe http.server should be split into the "basic parts" (used by bottle.py, and other frameworks), and the "higher-level parts", which could then be discarded by this PEP! At this point, though, I'd have to agree that the whole should not be discarded. Thanks for making me dig deeper. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Tue May 21 19:15:13 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 22 May 2019 01:15:13 +0200 Subject: [Python-Dev] Parser module in the stdlib In-Reply-To: References: Message-ID: Le ven. 17 mai 2019 ? 00:44, Nathaniel Smith a ?crit : > Will the folks using forks be happy to switch to the stdlib version? > For example I can imagine that if black wants to process 3.7 input > code while running on 3.6, it might prefer a parser on PyPI even if > the stdlib version were public, since the PyPI version can be updated > independently of the host Python. Pablo wrote: > I think is missing an important gap in the stdlib and the closest thing we have > (the current parser module) is not useful for any of that. Also, the core to generating > the hypothetical new package (with some new API over it may be) is already undocumented > as an implementation detail of lib2to3 (and some people are already using it directly). IMHO it's way better to first mature a module on PyPI to ensure that it becomes: * battle-tested * have a well defined and *stable* API * have a good test suite Right now, everything looks too vague to me. Reminder: the stdlib basically only gets a release every 2 years. If you forget a super important feature, you have to wait 2 years to add it. Even for bugfixes, you also have to wait months if not years until all vendors upgrade their Python package... I'm not sure that putting it directly in the stdlib is a good idea. Victor -- Night gathers, and now my watch begins. It shall not end until my death. From ethan at stoneleaf.us Tue May 21 19:21:19 2019 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 21 May 2019 16:21:19 -0700 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: On 05/21/2019 01:30 PM, Steve Dower wrote: > In the next Windows 10 update that starts rolling out today, we (Microsoft) have added "python.exe" and "python3.exe" commands that are installed on PATH *by default* and will open the Microsoft Store at the page where we (Python core team) publish our build. > > This makes it a 1-2 click process to get from a clean machine to having a usable Python install ("python.exe" -> opens Store -> "Get it Free" -> "python.exe" now works!) This is great! I'm now actually looking forward to an update! :-) -- ~Ethan~ From python at mrabarnett.plus.com Tue May 21 19:21:13 2019 From: python at mrabarnett.plus.com (MRAB) Date: Wed, 22 May 2019 00:21:13 +0100 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: <1b8710f5-4cd9-8c2c-6e9c-5baf96b2ea44@mrabarnett.plus.com> On 2019-05-21 21:30, Steve Dower wrote: [snip] > The associated blog post: > > https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/ > > Here are answers to a few questions that I assume will come up, at least > from this audience that understands the issues better than most: > > * if someone had installed Python and put it on PATH with our installer, > this new command *does not* interfere > * if someone had manually modified their own PATH, they *may* see some > interference (but we [Microsoft] decided this was an acceptable risk) > * the Python 3.7 installed from the store will not auto-update to 3.8, > but when 3.8 is released we (Microsoft) will update the redirect to > point at it > * if you pass arguments to the redirect command, it just exits with an > error code - you only get the Store page if you run it without arguments > * once the Store package is installed, the redirect command is replaced > (this required a new feature in the OS). If you install with the regular > installer and update PATH, or active a venv, it will add it *before* the > redirect. So these scenarios should be all good. > > I'm happy to answer other questions here. The long-term contact for this > integration is python (at) microsoft.com, which right now will come to me. > Does it behave nicely with py.exe? Earlier this year I installed an extension in Visual Studio Code for programming Arduino. It installed its own version of Python 2.7 (I'm on Python 3) and changed the file associations to point to that, thus breaking my scripts. (I wasn't keen on its customised icons, either.) I was somewhat unhappy at that... From steve.dower at python.org Tue May 21 19:40:23 2019 From: steve.dower at python.org (Steve Dower) Date: Tue, 21 May 2019 16:40:23 -0700 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <1b8710f5-4cd9-8c2c-6e9c-5baf96b2ea44@mrabarnett.plus.com> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> <1b8710f5-4cd9-8c2c-6e9c-5baf96b2ea44@mrabarnett.plus.com> Message-ID: <6658d7df-ea21-e43d-71b8-3a65737032e1@python.org> On 21May2019 1621, MRAB wrote: > Does it behave nicely with py.exe? This is still something of an open issue with the Store package for Python - py.exe doesn't look for the registry keys in a way that will find them (due to some very obscure compatibility quirks). The Store package does not include py.exe though, so this only comes up if you install a second copy with the regular installer. And that would have to be a 3.8 alpha release to have any chance of getting the fix, since if you install a 3.7 release along with the 3.7 Store package then the regular install is likely going to shadow the Store package anyway, and I wasn't planning on adding to 3.6's launcher at this stage. So far I haven't heard much feedback about this being a real issue - it's almost all been theoretical questions rather than "why did py.exe just fail?" issues. I expect they'll come up, and when they do I'm sure they'll point towards a good solution. > Earlier this year I installed an extension in Visual Studio Code for > programming Arduino. It installed its own version of Python 2.7 (I'm on > Python 3) and changed the file associations to point to that, thus > breaking my scripts. (I wasn't keen on its customised icons, either.) I > was somewhat unhappy at that... The Store package should change otherwise unassigned file associations to point at it when it is installed. If you already have file associations, you might get a prompt next time you double-click a .py file, but it shouldn't just steal them. Cheers, Steve From vstinner at redhat.com Tue May 21 20:01:52 2019 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 22 May 2019 02:01:52 +0200 Subject: [Python-Dev] PEP 587 "Python Initialization Configuration" version 4 In-Reply-To: <20190520213135.201de6f2@fsol> References: <20190520213135.201de6f2@fsol> Message-ID: Hi, Le lun. 20 mai 2019 ? 21:36, Antoine Pitrou a ?crit : > - Since PyInitError can be "ok", I'd rather call it "PyInitStatus". Oh, I like this name :-) By the way, I'm not comfortable with "PyInitError_Failed()" name since it's true if it's an error (failure) ... or "an exit". What do you think of the name "PyInitStatus_IsDone()"? It would give the pattern: PyInitStatus status = Py_InitializeFromConfig(&config); if (PyInitStatus_IsDone(status)) { PyConfig_Clear(&config); Py_ExitInitStatus(status); } Does it make more sense than the current code below? PyInitError err = Py_InitializeFromConfig(&config); if (PyInitError_Failed(err)) { PyConfig_Clear(&config); Py_ExitInitError(err); } > - The PyConfig example sets "module_search_paths" but not > "module_search_paths_set". Is it an error? If "module_search_paths_set" is set to 1, PyConfig_Read() leaves "module_search_paths" unchanged. So it becomes possible to use Python with sys.path = []. Extract of the mentioned example (simplified): --- PyConfig_Read(&config); PyWideStringList_Append(&config.module_search_paths, L"/path/to/more/modules"); --- PyConfig_Read() is responsible to set "module_search_paths_set" to 1. So if PyConfig_Read() is called again (which is always the case in Py_InitializeFromConfig() to ensure that all PyConfig fields are set), our customized module_search_paths is preserved. > - "PyConfig_InitPythonConfig" vs "PyConfig_InitIsolatedConfig": why not > "PyConfig_InitConfig" and "PyConfig_InitIsolatedConfig"? You should read "PyConfig_InitPythonConfig" as: same configuration than regular Python executable. Python 3.7 Py_Initialize() doesn't provide you exactly the same configuration than regular Python executable: command line arguments of sys.argv are not parsed, C standard streams are not configured, etc. Moreover, maybe someone will come up with a 3rd flavor of config, so I prefer to give them explicit names ;-) A single default configuration doesn't fit with all use cases, so PEP 587 provides two different flavors :-) > - Why is there an "isolated" field in addition to the "IsolatedConfig" > functions? There is a single PyConfig structure in the API. It makes more sense for the Python Configuration which is *not* isolated by default. So you can opt-in for an isolated Python configuration, as shown in the example: https://www.python.org/dev/peps/pep-0587/#python-configuration Moreover, well, I have to store the information somewhere :-) The idea of the "Isolated Configuration" is to start from the most isolated configuration, but then to can tune it for your needs. You might want to opt-in for isolated=0 to have a behavior closer to the regular Python. > - PyConfig.isolated is not documented. Oops, fixed (in my current draft). > - "Configuration files are still used with this configuration": why is > this? Shouldn't they be ignored in an isolated configuration? In short, I didn't touch the Path Configuration. You have to explicitly set 5 fields to prevent Python to compute the Path Configuration: * exec_prefix * executable * prefix * module_search_paths_set * module_search_paths https://www.python.org/dev/peps/pep-0587/#path-configuration > - In "Comparison of Python and Isolated Configurations": what does -1 > mean here? Hum, the most complete documentation for the all configuration fields currently lives at: https://github.com/python/cpython/blob/master/Include/cpython/coreconfig.h The design of the PEP 587 is that fields "explicitly" by the user prevents Python to override them. The -1 value means "unset": Python reads the value from the environment. For example, dev_mode equals to -1 means that PYTHONDEVMODE env var and -X dev command line variables are checked: they set dev_mode to 1 is present, otherwise dev_mode is set to 0. Setting dev_mode to 0 ensures that it cannot be enabled. > Overall, this looks like a great improvement. My only worry is that > interactions between the various options seem complicated and > difficult to understand. Perhaps we will need some detailed > documentation and examples. "interactions between the various options" should be documentation at: https://www.python.org/dev/peps/pep-0587/#priority-and-rules My plan is also to move PEP 587 into the documentation, around: http://docs.python.org/dev/c-api/init.html The documentation can be enhanced later. Victor From bcannon at gmail.com Tue May 21 20:44:16 2019 From: bcannon at gmail.com (Brett Cannon) Date: Tue, 21 May 2019 17:44:16 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <27ba695b-3219-a628-acef-4e0af649e2de@python.org> References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> Message-ID: On Tue., May 21, 2019, 11:29 Antoine Pitrou, wrote: > > Le 21/05/2019 ? 20:16, Brett Cannon a ?crit : > > > > > > On Tue., May 21, 2019, 09:10 Christian Heimes, > > wrote: > > > > On 21/05/2019 17.31, Antoine Pitrou wrote: > > > > > > As I said, if the main annoyance with nntplib is the sporadic test > > > failures, then the relevant tests can be disabled on CI. > > > > > > NNTP itself is still used, even if less and less. > > > > I don't like the idea to drop a third of the test cases for nntplib > > -- and the 30% that actually test that Python's NNTP library can > > talk to an actual NNTP server. IMHO it's more beneficial for core > > development and for nntplib to have it maintained by somebody that > > cares about the library. > > > > > > Ditto for me. If the code isn't going to be properly tested then it's > > already unsupported, at which point we're back to wanting to deprecate > it. > > So I should never have added those tests and then we wouldn't be talking > about removing nntplib. > Not necessarily. I suspect it still would have been listed, you would have objected, and someone may have looked at the tests to point out it was only unit tests with no integration tests against a live service like e.g. the pythontest.net-related tests and so it was already neglected. It also doesn't help that no one is listed in the experts index for the module either. -Brett > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericsnowcurrently at gmail.com Tue May 21 20:53:37 2019 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 21 May 2019 17:53:37 -0700 Subject: [Python-Dev] PEP 544 and dunder methods In-Reply-To: References: Message-ID: On Tue, May 21, 2019, 12:00 Guido van Rossum wrote: > My guess, without delving into the implementation, is that a Protocol is > *always* about the class, and that this is entirely a red herring. > I think you're right. It makes sense. I must have missed it somehow. FYI, some protocols (like pickle) are looked up on the instance. I'm on my phone or I'd point you to the BPO issue. :) -eric > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 21 21:19:42 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 21 May 2019 18:19:42 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> Message-ID: I repeat my position: on an omnibus PEP like this, if there's this much debate about one module, it should be struck from the list. On Tue, May 21, 2019 at 5:46 PM Brett Cannon wrote: > > > On Tue., May 21, 2019, 11:29 Antoine Pitrou, wrote: > >> >> Le 21/05/2019 ? 20:16, Brett Cannon a ?crit : >> > >> > >> > On Tue., May 21, 2019, 09:10 Christian Heimes, > > > wrote: >> > >> > On 21/05/2019 17.31, Antoine Pitrou wrote: >> > > >> > > As I said, if the main annoyance with nntplib is the sporadic test >> > > failures, then the relevant tests can be disabled on CI. >> > > >> > > NNTP itself is still used, even if less and less. >> > >> > I don't like the idea to drop a third of the test cases for nntplib >> > -- and the 30% that actually test that Python's NNTP library can >> > talk to an actual NNTP server. IMHO it's more beneficial for core >> > development and for nntplib to have it maintained by somebody that >> > cares about the library. >> > >> > >> > Ditto for me. If the code isn't going to be properly tested then it's >> > already unsupported, at which point we're back to wanting to deprecate >> it. >> >> So I should never have added those tests and then we wouldn't be talking >> about removing nntplib. >> > > Not necessarily. I suspect it still would have been listed, you would have > objected, and someone may have looked at the tests to point out it was only > unit tests with no integration tests against a live service like e.g. the > pythontest.net-related tests and so it was already neglected. > > It also doesn't help that no one is listed in the experts index for the > module either. > > -Brett > > >> Regards >> >> Antoine. >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From mariatta.wijaya at gmail.com Tue May 21 22:15:17 2019 From: mariatta.wijaya at gmail.com (Mariatta) Date: Wed, 22 May 2019 04:15:17 +0200 Subject: [Python-Dev] Unrelated PRs linked to issues on bugs.python.org In-Reply-To: References: Message-ID: One possible reason is when we wrote the wrong boo number in the PR, and then changed it to the correct one. Not sure if this was the case. The linking of PR number in boo is done in bpo. There was a similar issue: https://github.com/python/bugs.python.org/issues/25 On Wed, May 22, 2019, 12:21 AM Victor Stinner wrote: > Hi, > > Since one or two weeks, I noticed that the bot which links GitHub pull > requests to bugs.python.org issues started to link PRs to unrelated > issues. > > Example: https://github.com/python/cpython/pull/13148 (merged 2 hours > ago) just added to https://bugs.python.org/issue35363 (closed at the > end of last year). > > I cannot find "35363" (bpo number) in > https://github.com/python/cpython/pull/13148 nor in > https://github.com/python/cpython/pull/13148/files > > Does anyone have an idea why unrelated PRs are added to issues? > > I don't know which bot is responsible for this task, nor who operates > this bot. So in case of doubt, I chose to send a message to > python-dev. > > Victor > -- > Night gathers, and now my watch begins. It shall not end until my death. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mariatta%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From bcannon at gmail.com Tue May 21 23:04:11 2019 From: bcannon at gmail.com (Brett Cannon) Date: Tue, 21 May 2019 20:04:11 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190521195948.b4cklbgdzomyfpza@python.ca> References: <20190521130111.GJ4221@ando.pearwood.info> <20190521195948.b4cklbgdzomyfpza@python.ca> Message-ID: On Tue., May 21, 2019, 13:07 Neil Schemenauer, wrote: > On 2019-05-21, Terry Reedy wrote: > > The problem with this argument, taken by itself, it that it would argue > for > > adding to the stdlib 100s or 1000s of modules or packages that would be > > useful to many more people than the modules proposed to be dropped. > > I don't think it does. We are not talking about 100s or 1000s of > modules. We are talking about modules which have been in Python's > stdlib for years or decades. If I have a script that uses one of > these modules and it gets removed, my script breaks. > > Installing it from PyPI is not really a great solution. We are > going to be breaking working scripts just like if we add new > language keywords, etc. I think we need to be extremely careful > with trying to maintain backwards compatibility, at least as far as > we reasonably can. > > The problem I have with this PEP is that I think it both too > aggressive and too conservative at the same time. For almost all > modules on the list, I'm sure there will be many people who are > harmed by its removal. OTOH, having to maintain all of the modules > in the stdlib is a heavy burden. Also, new users can be lured into > using a module that is not really the best solution anymore. > > Here is an alternative, straw man, proposal. Split the CPython repo > into two parts: > > - core Python: minimal possible stdlib > - everything else > > When Python is released, provide installers for a Python that only > includes the "core" part and a second installer that includes > everything else. I realize this is more work for the release team. > Hopefully with some scripting, it will not be too labour intensive. > > The core Python installer should become the recommended installer. > People who need backwards compability with older versions of Python > can download the big installer package. > How to this lighten the burden of maintaining those modules which aren't in the core install? Are you suggesting modules in the core install get serious focus and the rest is more of a legacy, unsupported release for some time to give people an extended period to shift to the core install? Or do you have something else in mind? -Brett > To help the people who need 100s or 1000s of extra PyPI packages, we > could develop a tool that creates a "sumo" Python installer, > grabbing packages from PyPI and building a installer package. To > install that package, you would not need network access. That > doesn't need to happen right away. Also, maybe other Python > distributions can fill that need if core Python devs don't want to > build it. > > Regards, > > Neil > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arfrever.fta at gmail.com Wed May 22 00:20:44 2019 From: arfrever.fta at gmail.com (Arfrever Frehtes Taifersar Arahesis) Date: Wed, 22 May 2019 06:20:44 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> Message-ID: 2019-05-21 00:06 UTC+02:00, Christian Heimes wrote: > On 20/05/2019 23.27, Antoine Pitrou wrote: >> Removing the crypt module would remove support for system-standard >> password files. I don't understand the rationale. > > Applications *must* not access system-standard password files directly. On > any sanely and securely configured systems, application cannot even access > system password files like /etc/shadow. Access restrictions and system > security policies will prevent read access. Also applications cannot assume > that users are present in any user file. They may come from LDAP, SSSD, > ActiveDirectory, or other sources. > > The correct way to interact with system users is to use the proper APIs, > that are NSS (name service switch) and PAM (pluggable authentication > modules). NSS looks up and enumerate users and groups. PAM performs password > validation and much, much, much more. The pwd and grp modules use the > correct APIs to interact with NSS. If you need to check or change passwords, > you must go through PAM. It is possible to have a modern Linux desktop system with PAM not installed at all, and therefore not used. Examples of packages in Gentoo Linux which have OPTIONAL dependency on PAM: shadow, sudo, openssh, libcap, systemd, util-linux, screen, cronie, polkit, cups, sddm, kscreenlocker, xscreensaver (So a KDE Plasma desktop environment and its direct and indirect dependencies can be installed without PAM.) The suggested substitutes for spwd module, i.e. python-pam and simpleplam, look like they would not work on a PAM-free system. -- Arfrever Frehtes Taifersar Arahesis From turnbull.stephen.fw at u.tsukuba.ac.jp Wed May 22 00:59:59 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 22 May 2019 13:59:59 +0900 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> References: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> Message-ID: <23780.55119.651028.7615@turnbull.sk.tsukuba.ac.jp> Christian Heimes writes: > It's all open source. It's up to the Python community to adopt > packages and provide them on PyPI. > > Python core will not maintain and distribute the packages. I'll > merely provide a repository with packages to help kick-starting the > process. This looks to me like an opening to a special class of supply chain attacks. I realize that PyPI is not yet particularly robust to such attacks, and we have seen "similar name" attacks (malware uploaded under a name similar to a popular package). ISTM that this approach to implementing the PEP will enable "identical name" attacks. (By download count, stdlib packages are as popular as Python. :-) It now appears that there's been substantial pushback against removing packages that could be characterized as "obsolete and superseded but still in use", so this may not be a sufficient great risk to be worth addressing. I guess this post is already a warning to those who are taking care of the "similar name" malware that this class of attacks will be opened up. One thing we *could* do that would require moderate effort would be to put them up on PyPI ourselves, and require that would-be maintainers be given a (light) vetting before handing over the keys. (Maybe just require that they be subscribers to the Dead Parrot SIG? :-) Steve From random832 at fastmail.com Wed May 22 01:38:41 2019 From: random832 at fastmail.com (Random832) Date: Wed, 22 May 2019 01:38:41 -0400 Subject: [Python-Dev] =?utf-8?q?PEP_594=3A_Removing_dead_batteries_from_t?= =?utf-8?q?he_standard_library?= In-Reply-To: <66dc8643-6c91-ac1f-6ff8-ebce4de39a5b@python.org> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <66dc8643-6c91-ac1f-6ff8-ebce4de39a5b@python.org> Message-ID: <8f3e0f49-17c3-45c7-93be-8c6d803b548d@www.fastmail.com> On Tue, May 21, 2019, at 14:17, Christian Heimes wrote: > thanks for bringing this topic up. Initially I considered http.server, > too. But as Guido put it, it's both used and useful for local testing > and quick hacks. I'm already facing opposition for modules that are > less controversial and useful than http.server, too. Could the high-level functionality of http.server (and perhaps socketserver as a whole) be rebuilt on top of asyncio, with serve_forever implicitly starting an asyncio loop with the server as the only task? Or are the paradigms too different? From robertc at robertcollins.net Wed May 22 01:40:48 2019 From: robertc at robertcollins.net (Robert Collins) Date: Wed, 22 May 2019 17:40:48 +1200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <23780.55119.651028.7615@turnbull.sk.tsukuba.ac.jp> References: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> <23780.55119.651028.7615@turnbull.sk.tsukuba.ac.jp> Message-ID: This vector exists today for all new stdlib modules: once added, any existing dependency could include that name to cater it to be imported on prior python versions. Rob On Wed, 22 May 2019, 17:03 Stephen J. Turnbull, < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Christian Heimes writes: > > > It's all open source. It's up to the Python community to adopt > > packages and provide them on PyPI. > > > > Python core will not maintain and distribute the packages. I'll > > merely provide a repository with packages to help kick-starting the > > process. > > This looks to me like an opening to a special class of supply chain > attacks. I realize that PyPI is not yet particularly robust to such > attacks, and we have seen "similar name" attacks (malware uploaded > under a name similar to a popular package). ISTM that this approach > to implementing the PEP will enable "identical name" attacks. (By > download count, stdlib packages are as popular as Python. :-) > > It now appears that there's been substantial pushback against removing > packages that could be characterized as "obsolete and superseded but > still in use", so this may not be a sufficient great risk to be worth > addressing. I guess this post is already a warning to those who are > taking care of the "similar name" malware that this class of attacks > will be opened up. > > One thing we *could* do that would require moderate effort would be to > put them up on PyPI ourselves, and require that would-be maintainers > be given a (light) vetting before handing over the keys. (Maybe just > require that they be subscribers to the Dead Parrot SIG? :-) > > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/robertc%40robertcollins.net > -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Wed May 22 02:30:41 2019 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Wed, 22 May 2019 13:30:41 +0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190520232749.442c45c7@fsol> References: <20190520232749.442c45c7@fsol> Message-ID: On Tue, 21 May 2019 at 04:30, Antoine Pitrou wrote: > > NNTP is still quite used (often through GMane, but probably not only) so > I'd question the removal of nntplib. I concur nntplib should be left alone. There are possibly even less used network protocols such as telnet (tenetlib) which are not targeted by this PEP but could have been by following the same logic. FTP is another one that despite no longer popular, still has a constant user base (you?d be surprised by the amount of traffic we got over FTP in the last company I worked for). Overall, I think the bar for a module removal should be set very high, especially for ?standard? things such as these network protocols, that despite being old are not likely to change. That means that also the maintenance burden for python-dev will be low or close to none after all. It seems to me also spwd/crypt modules fall into this category (all UNIX platforms have the shadow password db, which is nice to have in python out of the box). In that case the removal appears to be more justified by the security implications than them being not widely used though, so I would use more caution and treat them differently (eg. opt for doc warning + investigate a possible replacement). Also note that spwd could be used to parse /etc/passwd, despite I admit its primary use case is password checking. Certain users may even not care about the additional security provided by PAM (eg. internal devop scripts). -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From nas-python at arctrix.com Wed May 22 02:39:05 2019 From: nas-python at arctrix.com (Neil Schemenauer) Date: Wed, 22 May 2019 00:39:05 -0600 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190521130111.GJ4221@ando.pearwood.info> <20190521195948.b4cklbgdzomyfpza@python.ca> Message-ID: <20190522063905.gvhfbe4h7eocddkr@python.ca> On 2019-05-21, Brett Cannon wrote: > On Tue., May 21, 2019, 13:07 Neil Schemenauer, > wrote: > > Here is an alternative, straw man, proposal. Split the CPython repo > > into two parts: > > > > - core Python: minimal possible stdlib > > - everything else > > How to this lighten the burden of maintaining those modules which aren't in > the core install? Are you suggesting modules in the core install get > serious focus and the rest is more of a legacy, unsupported release for > some time to give people an extended period to shift to the core install? > Or do you have something else in mind? It would give us the freedom to choose how we want to do it. It would give a lightweight Python install for the people who don't need all the batteries, much lighter than what the PEP 594 strategy could provide. For CI, we can decide what should be tested. Obviously the core part is always tested. Initially, we could continue testing all parts of non-core. Later, we could decide that certain parts of non-core get tested less often (e.g. full nntplib tests only nightly). BTW, I think it would be great if automated nightly jobs could also also run tests for PyPI modules like NumPy, requests, Pandas, etc. The installers could offer options as to which parts of the non-core library to install. Modules that no longer receive the same quality of development and testing could get moved to a "deprecated" section. Users who want the best backwards compatibility would install everything. If we want to remove something from the "deprecated" section, I think we should give a lot of notice. A couple of years is not enough. Here is a sketch for a Linux-like package system: python3-stdlib-base all recommended stdlib packages (e.g. same as stdlib after PEP 594 removals) python3-stdlib-deprecated packages suggested for removal by PEP 594 python3-stdlib-broken packages that have bugs or are really not recommended to be used. I'm not sure if we need this but stuff like crypt could go here. python3-stdlib-all depends on the above three packages Ideally these packages don't contain the module files themselves. Instead, they depend on individual packages for each module. E.g. python3-stdlib-deprecated would depend on python3-nntplib. So, someone could install python3-stdlib-base and python3-nntplib if that's all they need. I'm not familiar with the internals of 'pip' but I would guess we could do a similar thing by creating meta PyPI packages that correspond to these sets of packages. So, someone could download the small "core" Python installer and then run: pip install stdlib-base or something like that. From christian at python.org Wed May 22 04:07:31 2019 From: christian at python.org (Christian Heimes) Date: Wed, 22 May 2019 10:07:31 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> Message-ID: On 22/05/2019 06.20, Arfrever Frehtes Taifersar Arahesis wrote: > 2019-05-21 00:06 UTC+02:00, Christian Heimes wrote: >> On 20/05/2019 23.27, Antoine Pitrou wrote: >>> Removing the crypt module would remove support for system-standard >>> password files. I don't understand the rationale. >> >> Applications *must* not access system-standard password files directly. On >> any sanely and securely configured systems, application cannot even access >> system password files like /etc/shadow. Access restrictions and system >> security policies will prevent read access. Also applications cannot assume >> that users are present in any user file. They may come from LDAP, SSSD, >> ActiveDirectory, or other sources. >> >> The correct way to interact with system users is to use the proper APIs, >> that are NSS (name service switch) and PAM (pluggable authentication >> modules). NSS looks up and enumerate users and groups. PAM performs password >> validation and much, much, much more. The pwd and grp modules use the >> correct APIs to interact with NSS. If you need to check or change passwords, >> you must go through PAM. > > It is possible to have a modern Linux desktop system with PAM not > installed at all, and therefore not used. > > Examples of packages in Gentoo Linux which have OPTIONAL dependency on PAM: > shadow, sudo, openssh, libcap, systemd, util-linux, screen, cronie, > polkit, cups, sddm, kscreenlocker, xscreensaver > (So a KDE Plasma desktop environment and its direct and indirect > dependencies can be installed without PAM.) > > The suggested substitutes for spwd module, i.e. python-pam and > simpleplam, look like they would not work on a PAM-free system. Thanks for bringing this up. I don't think we need to care about this care. A PAM-free Linux system is an IMHO very special and exotic case. It's certainly not a setup anybody should run on a server. There are a lot of good reasons to use PAM. I'll update the BPO with reasons soonish. By the way, the /etc/shadow shadow(5) format is trivial and can be parsed with a few lines of code. There is no need to use spwd. Christian From christian at python.org Wed May 22 05:14:41 2019 From: christian at python.org (Christian Heimes) Date: Wed, 22 May 2019 11:14:41 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> Message-ID: On 22/05/2019 08.30, Giampaolo Rodola' wrote: > > > On Tue, 21 May 2019 at 04:30, Antoine Pitrou > wrote: > > > NNTP is still quite used (often through GMane, but probably not only) so > I'd question the removal of nntplib. > > > I concur nntplib should be left alone. There are possibly even less used network protocols such as telnet (tenetlib) which are not targeted by this PEP but could have been by following the same logic. FTP is another one that despite no longer popular, still has a constant user base (you?d be surprised by the amount of traffic we got over FTP in the last company I worked for). Overall, I think the bar for a module removal should be set very high, especially for ?standard? things such as these network protocols, that despite being old are not likely to change. That means that also the maintenance burden for python-dev will be low or close to none after all. * To be honest, I missed telnetlib. https://github.com/python/peps/pull/1073 * The maintenance burden for nntplib is actually high, because tests are flaky. * nntplib has no maintainer and is missing features like COMPRESS extension. > It seems to me also spwd/crypt modules fall into this category (all UNIX platforms have the shadow password db, which is nice to have in python out of the box). In that case the removal appears to be more justified by the security implications than them being not widely used though, so I would use more caution and treat them differently (eg. opt for doc warning + investigate a possible replacement). Also note that spwd could be used to parse /etc/passwd, despite I admit its primary use case is password checking. Certain users may even not care about the additional security provided by PAM (eg. internal devop scripts). spwd and crypt are dead batteries, because their usefulness has been surpassed about two decades (!) ago. They are also very dangerous batteries because they leak hydrofluoric acid at scale. It's as least as bad as the acid + bathtub scene from the first season of Breaking Bad [1]. HF is nasty [2]. I can reveal more details in a week or two. Christian [1] https://breakingbad.fandom.com/wiki/Hydrofluoric_acid [2] https://en.wikipedia.org/wiki/Hydrofluoric_acid#Health_and_safety From turnbull.stephen.fw at u.tsukuba.ac.jp Wed May 22 05:26:24 2019 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 22 May 2019 18:26:24 +0900 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190521195948.b4cklbgdzomyfpza@python.ca> References: <20190521130111.GJ4221@ando.pearwood.info> <20190521195948.b4cklbgdzomyfpza@python.ca> Message-ID: <23781.5568.183871.73668@turnbull.sk.tsukuba.ac.jp> Neil Schemenauer writes: > Here is an alternative, straw man, proposal. Split the CPython repo > into two parts: > > - core Python: minimal possible stdlib > - everything else I take issue with the characterization of "straw man," it's a practical idea that turns out to be not so easy to implement. We tried to do this with XEmacs, and found it just didn't work for us. It requires a lot of effort to pare a distribution down to a minimal core. (In fact, we never got close.) We gave up on that idea after only a year, and this was an age and an environment that did not have pip-like support for installation of 3rd-party packages (which you deprecate as "not really a great solution"), and a lot of people were still downloading at ISDN speeds or worse. The demand for a minimal core was correspondingly greater than today, I think. We did bundle most applications such as MUAs and even C/C++ development support into sumo packages (one for general use, one for packages that didn't function at all without the "MULE" multilingual environment which was optional in XEmacs). But we still ended up with many packages staying in the core distribution, because other parts of the core or the installation process used them, and as you say "If I have a script that uses one of these modules and it gets removed, my script breaks." That annoyed core developers as much as anyone, and they generally had priorities that did not emphasize minimalism (to say the least). > When Python is released, provide installers for a Python that only > includes the "core" part and a second installer that includes > everything else. I realize this is more work for the release team. > Hopefully with some scripting, it will not be too labour intensive. Creating the distributions (core and sumo) was not particularly labor-intensive for us because we mostly didn't distribute binaries. A few moderately baroque scripts did the trick. Automation has advanced, but due to the amount of it it's still fragile, especially when you're producing turn-key binary installers. As I mentioned above, it wasn't the release process that was the bottleneck, it was other factors that made it hard to reduce the core. We had an "xemacs-base" package that provided a set of "extra tools" that some packages required, and libraries moved in and out of it pretty much every release as some developer got tired of telling people to install xemacs-base (back to core) or working around missing features (move it to xemacs-base where it could be updated easily). These are the same things that Python maintainers mention as pros and cons of having things in stdlib or out on PyPI. We learned there is a sweet spot. I know we didn't hit it most of the time, but it's very clear that both extremes are hard, if not infeasible, to maintain. (At the maximalist extreme, even GNU Emacs doesn't really try to include everything written in Emacs Lisp any more.) > To help the people who need 100s or 1000s of extra PyPI packages, > we could develop a tool that creates a "sumo" Python installer, > grabbing packages from PyPI and building a installer package. This could be done by any third party (it wouldn't even need to be a public distribution, it could easily be an enterprise that open sources an internal tool or a GSoC project). And it could be hosted on PyPI itself, since it's not the kind of thing that enterprises need to install on 100,000 workstations. > To install that package, you would not need network access. AIUI, it's not the network access that is usually the problem these days, although it can be annoying if net access is flaky or if you have to sneakernet USBs around. Still, "sneakernet" is basically a one-off cost. The problem is that collections of random software from untrusted sources are verboten in the enterprise. (Ah, those Walnut Creek CDs were wonderful! I wonder what a modern AV tool would find in them?!) I suppose that in those situations people already have the necessary tooling to roll the sumo or provide an intranet repository; the problem is navigating the red tape. > That doesn't need to happen right away. Also, maybe other Python > distributions can fill that need if core Python devs don't want to > build it. I don't think this is about what core devs do or don't want to build. I also don't feel a strong sense among the core developers that they don't want to maintain a "batteries included" standard library. Sure, there's Christian's PEP, but there's also deliberate activity to promote more participation at the core (ie, committer) level. AIUI, Christian's PEP is about transparency of what is already happening (WONTFIX-ing or long periods of inactivity on issues with modules of "low importance" to the active core developers), and a first step toward making a managed process for the deprecation of modules we can't afford to maintain given others we have committed to maintain. Steve From J.Demeyer at UGent.be Wed May 22 05:41:49 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 22 May 2019 11:41:49 +0200 Subject: [Python-Dev] Missing testcase for bpo-34125 Message-ID: <5CE5195D.9030903@UGent.be> Could somebody please merge https://github.com/python/cpython/pull/13461 It adds a missing testcase for bpo-34125. This is testing code which is affected by PEP 590, so missing this test might accidentally break CPython if we screw up with implementing PEP 590. Thanks, Jeroen. From steve at pearwood.info Wed May 22 05:56:55 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 May 2019 19:56:55 +1000 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: <20190522095653.GK4221@ando.pearwood.info> Let me be clear: I do not oppose the removal of modules where necessary, but I do not like this PEP as it stands. But full credit to Christian for graciously accepting feedback; I also acknowledge that if this PEP is accepted, we still have at least two releases to change our minds about individual modules. My largest issue with the PEP as a whole is that I don't like lumping a whole lot of unrelated modules together in one PEP. I believe it encourages a risky sense of "we ought to clean house and throw out all this junk!" rather than a measured consideration of pros and cons of each module individually. I would rather see a serious of much smaller mini-PEPs, one per module. They need not be large: we can take as a given that we can and should remove "dead batteries" (but not merely unloved batteries in good working order, or slightly leaking batteries that just need a bit of care.) Each mini-PEP need only lay out the case for and against removal of one module (or perhaps a small number of closely related modules, not defend the principle in the abstract. As I see it, there is a problem with this approach of a single omnibus PEP. A single PEP has reduced visibility for users of any specific module. This can easily lead to users of a module being under- represented in the discussion even if they are subscribed to Python-Dev. Let's say Alice is a heavy user of the "aardvark" module. If Alice sees a PEP "Deprecate and remove aardvark", she is likely to speak up and air her objections. But if Alice sees a single PEP "Remove dead batteries", she's likely to think "of course the aardvark library is not a dead battery, people like me us it all the time" and not realise she needs to make her objection known. Especially if the discussion thread is large. This thread has already reached 70 messages. "To meaningfully contribute, I will need to read all 70 messages in the Dead Battery thread, most of which are about modules I don't care about." versus: "I need to read 5 messages in the Remove Aardvark thread." So I think that, relative to having one mini-PEP per module, a single PEP may be unintentionally biased towards the "Yes remove the lot" case and under-representing those who use some of these unloved batteries. The problem is compounded by the prejudicial language used: nobody thinks of the modules they use as "dead batteries". How can they be dead when they work fine? E.g. the binhex module *just works* -- if there are any bugs in it, I haven't come across them. binhex is not an obsolete file format for people like me who sometimes needs to read and write old Mac files. It might be an old and unloved battery, but its not dead. > On the other hand, Python's standard library is piling up cruft, unnecessary > duplication of functionality, and dispensable features. This is undesirable > for several reasons. What you are calling cruft and dispensable, others call useful and indispensable. We've already seen at least three modules removed from the list of "cruft": colorsys, nttplib, fileinput. There may have been more by now -- I haven't caught up on all 70 posts yet. I also think the PEP doesn't make the case for some of the things being asserted. For example: > * Any additional module increases the maintenance cost for the Python core > development team. The team has limited resources, reduced maintenance cost > frees development time for other improvements. This may often be the case, but not always. If a module is stable, there's little maintenance cost to speak of. It works, there are (hopefully!) no known bugs, or at least no *serious* bugs, so why touch it? A module that isn't being changed could mean two things: 1. It is essentially complete. It does what it does, more or less correctly, and doesn't need new features or bug fixes. 2. It is suffering from bitrot. I think the PEP fails to make the case for 2 over 1. > * Modules in the standard library are generally favored and seen as the > de-facto solution for a problem. A majority of users only pick 3rd party > modules to replace a stdlib module, when they have a compelling reason, e.g. > lxml instead of `xml`. Is this a problem that we need to fix? > The removal of an unmaintained stdlib module > increases the chances of a community contributed module to become widely > used. I don't think the conclusion is justified. It could just as easily lead to a lack of any such library (the would-be users lack the ability or time to maintain one), or a plethora of alternatives: "Should I use binhex 1.2, or MacBinHex 0.5, or BinnyMacHexFace 5.7, or xehnib 0.8?, or ... " > * A lean and mean standard library benefits platforms with limited resources > like devices with just a few hundred kilobyte of storage (e.g. BBC > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > (e.g. pyodide) also benefit from reduced download size. And I think this argument needs to be strongly challenged. I don't think the CPython stdlib should be cut down to be "lean and mean" for the sake of people using Python on micro-platforms. Let the tiny Pythons choose their own standard library. Some more comments: > All deprecated modules will also undergo a feature freeze. No additional > features should be added. Bug should still be fixed. I disagree with this. If people are requesting or contributing features to a module, that's a good sign that it *is* in use and isn't dead and we shouldn't remove it. It seems perverse to say "we ought to remove this module because nobody is maintaining it" and then reject attempts to maintain it! You'll notice that so far, apart from the example of binhex above, I haven't come out in favour or against removing any specific library. That's intentional: I would like to seperate criticism of the PEP as a document from arguments about which modules should stay. Thank you, -- Steven From steve at pearwood.info Wed May 22 06:03:55 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 May 2019 20:03:55 +1000 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <005401d50ff8$839e66c0$8adb3440$@211mainstreet.net> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <005401d50ff8$839e66c0$8adb3440$@211mainstreet.net> Message-ID: <20190522100355.GL4221@ando.pearwood.info> On Tue, May 21, 2019 at 01:13:30PM -0400, Edwin Zimmerman wrote: [...] [-- Attachment #2: winmail.dat --] [-- Type: application/ms-tnef, Encoding: base64, Size: 8.4K --] Wow! I haven't see one of those for *years* -- I haven't noticed one for about 15 years, and I thought it was an obsolete format that even Windows email stopped using. I think the presence of that is somewhat ironic on a thread arguing that we ought to remove working modules for obsolete formats. -- Steven From solipsis at pitrou.net Wed May 22 06:12:37 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 May 2019 12:12:37 +0200 Subject: [Python-Dev] PEP 594: update 1 References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> Message-ID: <20190522121237.021cbec4@fsol> On Tue, 21 May 2019 17:44:16 -0700 Brett Cannon wrote: > > > > So I should never have added those tests and then we wouldn't be talking > > about removing nntplib. > > > > Not necessarily. I suspect it still would have been listed, you would have > objected, and someone may have looked at the tests to point out it was only > unit tests with no integration tests against a live service like e.g. the > pythontest.net-related tests and so it was already neglected. But that's moving the goalposts. The PEP talks about "cruft, unnecessary duplication of functionality, and dispensable features". Not modules which have an insufficient test suite. > It also doesn't help that no one is listed in the experts index for the > module either. Right, but that's the case for many modules. Regards Antoine. From songofacandy at gmail.com Wed May 22 06:16:29 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Wed, 22 May 2019 19:16:29 +0900 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190522095653.GK4221@ando.pearwood.info> References: <20190522095653.GK4221@ando.pearwood.info> Message-ID: 2019?5?22?(?) 18:57 Steven D'Aprano : > > > All deprecated modules will also undergo a feature freeze. No additional > > features should be added. Bug should still be fixed. > > I disagree with this. If people are requesting or contributing features > to a module, that's a good sign that it *is* in use and isn't dead > and we shouldn't remove it. > If people are requesting or contributing for a module but we can not handle them for long time, I think it's sign we can not maintain the module well. It is not dead, but leaking battery. Maybe, the module requires some domain specific knowledge and it is very far from Python core development. Domain expert will maintain it well than Python core developers. So I don't think "it is still used" is not always equal to "Python core developers should maintain it" or "it should be shipped with Python". Regards, -- Inada Naoki From steve at pearwood.info Wed May 22 06:19:21 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 May 2019 20:19:21 +1000 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> Message-ID: <20190522101921.GM4221@ando.pearwood.info> On Wed, May 22, 2019 at 10:07:31AM +0200, Christian Heimes wrote: > On 22/05/2019 06.20, Arfrever Frehtes Taifersar Arahesis wrote: > > It is possible to have a modern Linux desktop system with PAM not > > installed at all, and therefore not used. [...] Christian wrote: > Thanks for bringing this up. I don't think we need to care about this care. > > A PAM-free Linux system is an IMHO very special and exotic case. It's > certainly not a setup anybody should run on a server. I've heard of rare cases of people running Python on Linux desktops... *wink* > There are a lot > of good reasons to use PAM. I'll update the BPO with reasons soonish. I don't think this PEP should become a document about "Why you should use PAM". I appreciate that from your perspective as a Red Hat security guy, you want everyone to use best practices as you see them, but it isn't Python's position to convince Linux distros or users to use PAM. To put it another way... I think that if you want to make the case for PAM, put it on the web (a blog?) and link to it. As far as the spwd module is concerned, on the one hand you're saying "we should remove it because nobody should ever read from /etc/shadow", and then on the other hand you're all "but go ahead and read /etc/shadow if you like, it is perfectly trivial to do": > By the way, the /etc/shadow shadow(5) format is trivial and can be > parsed with a few lines of code. There is no need to use spwd. so I think you're undercutting your own argument. If reading from /etc/shadow is such a bad thing that we must remove it, why tell people that they can parse it themselves? Not that we could stop them, even if we wanted to. -- Steven From steve at pearwood.info Wed May 22 06:37:28 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 May 2019 20:37:28 +1000 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <23780.55119.651028.7615@turnbull.sk.tsukuba.ac.jp> References: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> <23780.55119.651028.7615@turnbull.sk.tsukuba.ac.jp> Message-ID: <20190522103726.GN4221@ando.pearwood.info> On Wed, May 22, 2019 at 01:59:59PM +0900, Stephen J. Turnbull wrote: > This looks to me like an opening to a special class of supply chain > attacks. [...] > One thing we *could* do that would require moderate effort would be to > put them up on PyPI ourselves, and require that would-be maintainers > be given a (light) vetting before handing over the keys. (Maybe just > require that they be subscribers to the Dead Parrot SIG? :-) Because black hat hackers don't know how to subscribe to a SIG? *wink* I'm just gonna leave this here... https://www.ietf.org/rfc/rfc3514.txt Python is open source, anyone can fork any module from the std lib for any purposes. We can't stop that. But your earlier point about supply chain attacks is very valid. Modules we shift out of the stdlib become more vulnerable to supply chain attacks, because more people will have to download them, giving more opportunity for typo-squatter attacks. -- Steven From steve at pearwood.info Wed May 22 06:48:23 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 22 May 2019 20:48:23 +1000 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190521130111.GJ4221@ando.pearwood.info> Message-ID: <20190522104822.GO4221@ando.pearwood.info> On Tue, May 21, 2019 at 01:59:56PM -0400, Terry Reedy wrote: > On 5/21/2019 9:01 AM, Steven D'Aprano wrote: > ... > >Many Python users don't have the privilege of being able to install > >arbitrary, unvetted packages from PyPI. They get to use only packages > >from approved vendors, including the stdlib, what they write themselves, > >and nothing else. Please don't dismiss this part of the Python community > >just because they don't typically hang around in the same forums we do. > ... > > The problem with this argument, taken by itself, it that it would argue > for adding to the stdlib 100s or 1000s of modules or packages that would > be useful to many more people than the modules proposed to be dropped. No -- taken *by itself* it is only an argument against removing what already exists (or at least to be conservative in what we remove). That is all I'm saying. It requires an *additional* argument that we add anything new, and I'm not making that argument here. The additional argument is valid as a counter to "just put it on PyPI". The argument goes like this: - if we agree that the aardvark module is useful and appropriate for the std lib (not every useful module is, for many reasons!) we could still decide not to add it; - if we add it to the std lib, it will be available to 100% of users of the std lib; - but if we say "put it on PyPI", it will only be available to (let's say) 85% of users. But I'm not making that argument here because I'm not asking to add anything. -- Steven From christian at python.org Wed May 22 07:09:34 2019 From: christian at python.org (Christian Heimes) Date: Wed, 22 May 2019 13:09:34 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <303943c0-0c48-ec35-cbd9-cbc357dd3e7e@g.nevcal.com> Message-ID: <8eba9ec6-5f24-ccad-1218-999173f9f252@python.org> On 22/05/2019 01.11, Glenn Linderman wrote: > On 5/21/2019 2:00 PM, Nathaniel Smith wrote: >> On Tue, May 21, 2019 at 10:43 AM Glenn Linderman wrote: >>> After maintaining my own version of http.server to fix or workaround some of its deficiencies for some years, I discovered bottle.py. It has far more capability, is far better documented, and is just as quick to deploy. While I haven't yet converted all past projects to use bottle.py, it will likely happen in time, unless something even simpler to use is discovered, although I can hardly imagine that happening. >> bottle.py uses http.server for its local development mode (the one you >> see in their quickstart example at the top of their README). Same with >> flask, django, and probably a bunch of other frameworks. It's *very* >> widely used. >> >> -n >> > The source for bottle.py version 0.13-dev has an import for http.client, but not http.server. I hadn't tracked down every indirect dependency in the bottle.py source code, but it seems that if one uses the "default server" for bottle, that it is "wsgiref", imported from wsgiref.simple_server, and that in turn does import BaseHTTPRequestHandler and HTTPServer from http.server. > > It is the higher-level code in http.server that has significant deficiencies that have caused me problems over the years... a "SimpleHTTPRequestHandler" that is so simple it doesn't do POST, PUT or PASTE, a "CGIHTTPRequestHandler" that only implements part of the CGI protocol, only CGI support in POST, no support for PUT or PASTE, and no support for https, and not much bug fix activity in those areas. > > Maybe http.server should be split into the "basic parts" (used by bottle.py, and other frameworks), and the "higher-level parts", which could then be discarded by this PEP! At this point, though, I'd have to agree that the whole should not be discarded. Thanks for making me dig deeper. The idea has merrit. However I feel its out of scope for the PEP. The http.server module and socketserver module are still widely used for debug and toy examples. Could you please open a bug to track your proposal? We may pursue it in a couple of years from now. From christian at python.org Wed May 22 07:31:18 2019 From: christian at python.org (Christian Heimes) Date: Wed, 22 May 2019 13:31:18 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190522101921.GM4221@ando.pearwood.info> References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> <20190522101921.GM4221@ando.pearwood.info> Message-ID: On 22/05/2019 12.19, Steven D'Aprano wrote: > I don't think this PEP should become a document about "Why you should > use PAM". I appreciate that from your perspective as a Red Hat security > guy, you want everyone to use best practices as you see them, but it > isn't Python's position to convince Linux distros or users to use PAM. I think the PEP should make clear why spwd is bad and pining for The Fjords. The document should point users to correct alternatives. There is no correct and secure way to use the spwd module to verify user accounts. Any use of spwd for logins introduces critical security bugs. By the way, all relevant BSD, Linux, and Darwin (macOS) distributions come with PAM support. Almost all use PAM by default. AFAIK only the minimal Alpine container does not have PAM installed by default. This is not Red Hat trying to evangelize the world. PAM is *the* industry standards on Unix-like OS. > To put it another way... I think that if you want to make the case for > PAM, put it on the web (a blog?) and link to it. > > As far as the spwd module is concerned, on the one hand you're saying > "we should remove it because nobody should ever read from /etc/shadow", > and then on the other hand you're all "but go ahead and read /etc/shadow > if you like, it is perfectly trivial to do": > >> By the way, the /etc/shadow shadow(5) format is trivial and can be >> parsed with a few lines of code. There is no need to use spwd. > > so I think you're undercutting your own argument. If reading from > /etc/shadow is such a bad thing that we must remove it, why tell people > that they can parse it themselves? > > Not that we could stop them, even if we wanted to. Steven, I feel like you are turning my own words and arguments against me. From christian at python.org Wed May 22 08:19:02 2019 From: christian at python.org (Christian Heimes) Date: Wed, 22 May 2019 14:19:02 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <23780.55119.651028.7615@turnbull.sk.tsukuba.ac.jp> References: <58e5c9ff-3e5f-5146-3bd9-7feeacdb0aea@python.org> <23780.55119.651028.7615@turnbull.sk.tsukuba.ac.jp> Message-ID: <54787dcb-559c-ed45-1ec4-7535480d2b60@python.org> On 22/05/2019 06.59, Stephen J. Turnbull wrote: > Christian Heimes writes: > > > It's all open source. It's up to the Python community to adopt > > packages and provide them on PyPI. > > > > Python core will not maintain and distribute the packages. I'll > > merely provide a repository with packages to help kick-starting the > > process. > > This looks to me like an opening to a special class of supply chain > attacks. I realize that PyPI is not yet particularly robust to such > attacks, and we have seen "similar name" attacks (malware uploaded > under a name similar to a popular package). ISTM that this approach > to implementing the PEP will enable "identical name" attacks. (By > download count, stdlib packages are as popular as Python. :-) I don't consider this an argument against my proposal, but an argument in favor of improving PyPI. I propose a deal: If you get PEP 453 (ensurepip) revoked, ensurepip removed from the standard library, and the recommendation for the requests package on urllib.request replaced with a big, fat security warning, then I'll reconsider my proposal to recommend PyPI. :) My PEP acts in good faith. As long as CPython's stdlib ships pip and embraces PyPI, I don't see any reason to distrust PyPI. Yes, PyPI is not Fort Knox. In my humble opinion it's more than secure enough for my proposal. Christian From christian at python.org Wed May 22 08:26:05 2019 From: christian at python.org (Christian Heimes) Date: Wed, 22 May 2019 14:26:05 +0200 Subject: [Python-Dev] PEP 594: discussion-to discuss.python.org Message-ID: <20f8aa2b-0941-be11-39ba-1b4c626aadc6@python.org> Please use https://discuss.python.org/t/pep-594-removing-dead-batteries-from-the-standard-library/1704 for feedback and discussion. Thank you, Christian From christian at python.org Wed May 22 08:27:39 2019 From: christian at python.org (Christian Heimes) Date: Wed, 22 May 2019 14:27:39 +0200 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> Message-ID: On 22/05/2019 02.44, Brett Cannon wrote: > It also doesn't help that no one is listed in the experts index for the module either.. Excellent point! The PEP now lists the presence / absence of experts. Christian From berker.peksag at gmail.com Wed May 22 08:42:27 2019 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Wed, 22 May 2019 15:42:27 +0300 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> Message-ID: On Tue, May 21, 2019 at 7:11 PM Christian Heimes wrote: > > On 21/05/2019 17.31, Antoine Pitrou wrote: > > > > As I said, if the main annoyance with nntplib is the sporadic test > > failures, then the relevant tests can be disabled on CI. > > > > NNTP itself is still used, even if less and less. > > I don't like the idea to drop a third of the test cases for nntplib -- and the 30% that actually test that Python's NNTP library can talk to an actual NNTP server. IMHO it's more beneficial for core development and for nntplib to have it maintained by somebody that cares about the library. https://github.com/python/cpython/pull/9461 should fix most of the annoyances with test_nntplib. It uses a local server instead of an external one. --Berker From p.f.moore at gmail.com Wed May 22 09:39:02 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 22 May 2019 14:39:02 +0100 Subject: [Python-Dev] PEP 594: discussion-to discuss.python.org In-Reply-To: <20f8aa2b-0941-be11-39ba-1b4c626aadc6@python.org> References: <20f8aa2b-0941-be11-39ba-1b4c626aadc6@python.org> Message-ID: On Wed, 22 May 2019 at 13:28, Christian Heimes wrote: > > Please use https://discuss.python.org/t/pep-594-removing-dead-batteries-from-the-standard-library/1704 for feedback and discussion. > > Thank you, > Christian Out of curiosity, why? There's been a lot of useful discussion here - why now split the discussion, risk rehashing points already made over here, and making it harder for people to locate the history of the PEP in future? I'd see no problem directing the discussions to Discourse when the PEP was first announced, but why change part way through the process? Paul From encukou at gmail.com Wed May 22 09:51:02 2019 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 22 May 2019 15:51:02 +0200 Subject: [Python-Dev] PEP 590 (Vectorcall) discussion In-Reply-To: <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> References: <6c8356e3-f9b2-c39e-63c4-17f146d326b7@hotpy.org> <15b8a3d7-00ed-a5eb-475c-a3adee671b5f@hotpy.org> <5C9FEF82.50207@UGent.be> <421f8182-4bc8-b8cf-82d6-ca4a4fbd2013@hotpy.org> <50d675b4-839c-6502-ad1a-a33ea9330000@redhat.com> Message-ID: <06b65b36-dbb0-2d43-9845-80c08154afa1@gmail.com> Discussion on PEP 590 (Vectorcall) has been split over several PRs, issues and e-mails, so let me post an update. I am planning to approve PEP 590 with the following changes, if Mark doesn't object to them: * https://github.com/python/peps/pull/1064 (Mark the main API as private to allow changes in Python 3.9) * https://github.com/python/peps/pull/1066 (Use size_t for "number of arguments + flag") The resulting text, for reference: PEP: 590 Title: Vectorcall: a fast calling protocol for CPython Author: Mark Shannon , Jeroen Demeyer BDFL-Delegate: Petr Viktorin Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 29-Mar-2019 Python-Version: 3.8 Post-History: Abstract ======== This PEP introduces a new C API to optimize calls of objects. It introduces a new "vectorcall" protocol and calling convention. This is based on the "fastcall" convention, which is already used internally by CPython. The new features can be used by any user-defined extension class. Most of the new API is private in CPython 3.8. The plan is to finalize semantics and make it public in Python 3.9. **NOTE**: This PEP deals only with the Python/C API, it does not affect the Python language or standard library. Motivation ========== The choice of a calling convention impacts the performance and flexibility of code on either side of the call. Often there is tension between performance and flexibility. The current ``tp_call`` [2]_ calling convention is sufficiently flexible to cover all cases, but its performance is poor. The poor performance is largely a result of having to create intermediate tuples, and possibly intermediate dicts, during the call. This is mitigated in CPython by including special-case code to speed up calls to Python and builtin functions. Unfortunately, this means that other callables such as classes and third party extension objects are called using the slower, more general ``tp_call`` calling convention. This PEP proposes that the calling convention used internally for Python and builtin functions is generalized and published so that all calls can benefit from better performance. The new proposed calling convention is not fully general, but covers the large majority of calls. It is designed to remove the overhead of temporary object creation and multiple indirections. Another source of inefficiency in the ``tp_call`` convention is that it has one function pointer per class, rather than per object. This is inefficient for calls to classes as several intermediate objects need to be created. For a class ``cls``, at least one intermediate object is created for each call in the sequence ``type.__call__``, ``cls.__new__``, ``cls.__init__``. This PEP proposes an interface for use by extension modules. Such interfaces cannot effectively be tested, or designed, without having the consumers in the loop. For that reason, we provide private (underscore-prefixed) names. The API may change (based on consumer feedback) in Python 3.9, where we expect it to be finalized, and the underscores removed. Specification ============= The function pointer type ------------------------- Calls are made through a function pointer taking the following parameters: * ``PyObject *callable``: The called object * ``PyObject *const *args``: A vector of arguments * ``size_t nargs``: The number of arguments plus the optional flag ``PY_VECTORCALL_ARGUMENTS_OFFSET`` (see below) * ``PyObject *kwnames``: Either ``NULL`` or a tuple with the names of the keyword arguments This is implemented by the function pointer type: ``typedef PyObject *(*vectorcallfunc)(PyObject *callable, PyObject *const *args, size_t nargs, PyObject *kwnames);`` Changes to the ``PyTypeObject`` struct -------------------------------------- The unused slot ``printfunc tp_print`` is replaced with ``tp_vectorcall_offset``. It has the type ``Py_ssize_t``. A new ``tp_flags`` flag is added, ``_Py_TPFLAGS_HAVE_VECTORCALL``, which must be set for any class that uses the vectorcall protocol. If ``_Py_TPFLAGS_HAVE_VECTORCALL`` is set, then ``tp_vectorcall_offset`` must be a positive integer. It is the offset into the object of the vectorcall function pointer of type ``vectorcallfunc``. This pointer may be ``NULL``, in which case the behavior is the same as if ``_Py_TPFLAGS_HAVE_VECTORCALL`` was not set. The ``tp_print`` slot is reused as the ``tp_vectorcall_offset`` slot to make it easier for for external projects to backport the vectorcall protocol to earlier Python versions. In particular, the Cython project has shown interest in doing that (see https://mail.python.org/pipermail/python-dev/2018-June/153927.html). Descriptor behavior ------------------- One additional type flag is specified: ``Py_TPFLAGS_METHOD_DESCRIPTOR``. ``Py_TPFLAGS_METHOD_DESCRIPTOR`` should be set if the callable uses the descriptor protocol to create a bound method-like object. This is used by the interpreter to avoid creating temporary objects when calling methods (see ``_PyObject_GetMethod`` and the ``LOAD_METHOD``/``CALL_METHOD`` opcodes). Concretely, if ``Py_TPFLAGS_METHOD_DESCRIPTOR`` is set for ``type(func)``, then: - ``func.__get__(obj, cls)(*args, **kwds)`` (with ``obj`` not None) must be equivalent to ``func(obj, *args, **kwds)``. - ``func.__get__(None, cls)(*args, **kwds)`` must be equivalent to ``func(*args, **kwds)``. There are no restrictions on the object ``func.__get__(obj, cls)``. The latter is not required to implement the vectorcall protocol. The call -------- The call takes the form ``((vectorcallfunc)(((char *)o)+offset))(o, args, n, kwnames)`` where ``offset`` is ``Py_TYPE(o)->tp_vectorcall_offset``. The caller is responsible for creating the ``kwnames`` tuple and ensuring that there are no duplicates in it. ``n`` is the number of postional arguments plus possibly the ``PY_VECTORCALL_ARGUMENTS_OFFSET`` flag. PY_VECTORCALL_ARGUMENTS_OFFSET ------------------------------ The flag ``PY_VECTORCALL_ARGUMENTS_OFFSET`` should be added to ``n`` if the callee is allowed to temporarily change ``args[-1]``. In other words, this can be used if ``args`` points to argument 1 in the allocated vector. The callee must restore the value of ``args[-1]`` before returning. Whenever they can do so cheaply (without allocation), callers are encouraged to use ``PY_VECTORCALL_ARGUMENTS_OFFSET``. Doing so will allow callables such as bound methods to make their onward calls cheaply. The bytecode interpreter already allocates space on the stack for the callable, so it can use this trick at no additional cost. See [3]_ for an example of how ``PY_VECTORCALL_ARGUMENTS_OFFSET`` is used by a callee to avoid allocation. For getting the actual number of arguments from the parameter ``n``, the macro ``PyVectorcall_NARGS(n)`` must be used. This allows for future changes or extensions. New C API and changes to CPython ================================ The following functions or macros are added to the C API: - ``PyObject *_PyObject_Vectorcall(PyObject *obj, PyObject *const *args, size_t nargs, PyObject *keywords)``: Calls ``obj`` with the given arguments. Note that ``nargs`` may include the flag ``PY_VECTORCALL_ARGUMENTS_OFFSET``. The actual number of positional arguments is given by ``PyVectorcall_NARGS(nargs)``. The argument ``keywords`` is a tuple of keyword names or ``NULL``. An empty tuple has the same effect as passing ``NULL``. This uses either the vectorcall protocol or ``tp_call`` internally; if neither is supported, an exception is raised. - ``PyObject *PyVectorcall_Call(PyObject *obj, PyObject *tuple, PyObject *dict)``: Call the object (which must support vectorcall) with the old ``*args`` and ``**kwargs`` calling convention. This is mostly meant to put in the ``tp_call`` slot. - ``Py_ssize_t PyVectorcall_NARGS(size_t nargs)``: Given a vectorcall ``nargs`` argument, return the actual number of arguments. Currently equivalent to ``nargs & ~PY_VECTORCALL_ARGUMENTS_OFFSET``. Subclassing ----------- Extension types inherit the type flag ``_Py_TPFLAGS_HAVE_VECTORCALL`` and the value ``tp_vectorcall_offset`` from the base class, provided that they implement ``tp_call`` the same way as the base class. Additionally, the flag ``Py_TPFLAGS_METHOD_DESCRIPTOR`` is inherited if ``tp_descr_get`` is implemented the same way as the base class. Heap types never inherit the vectorcall protocol because that would not be safe (heap types can be changed dynamically). This restriction may be lifted in the future, but that would require special-casing ``__call__`` in ``type.__setattribute__``. Finalizing the API ================== The underscore in the names ``_PyObject_Vectorcall`` and ``_Py_TPFLAGS_HAVE_VECTORCALL`` indicates that this API may change in minor Python versions. When finalized (which is planned for Python 3.9), they will be renamed to ``PyObject_Vectorcall`` and ``Py_TPFLAGS_HAVE_VECTORCALL``. The old underscore-prefixed names will remain available as aliases. The new API will be documented as normal, but will warn of the above. Semantics for the other names introduced in this PEP (``PyVectorcall_NARGS``, ``PyVectorcall_Call``, ``Py_TPFLAGS_METHOD_DESCRIPTOR``, ``PY_VECTORCALL_ARGUMENTS_OFFSET``) are final. Internal CPython changes ======================== Changes to existing classes --------------------------- The ``function``, ``builtin_function_or_method``, ``method_descriptor``, ``method``, ``wrapper_descriptor``, ``method-wrapper`` classes will use the vectorcall protocol (not all of these will be changed in the initial implementation). For ``builtin_function_or_method`` and ``method_descriptor`` (which use the ``PyMethodDef`` data structure), one could implement a specific vectorcall wrapper for every existing calling convention. Whether or not it is worth doing that remains to be seen. Using the vectorcall protocol for classes ----------------------------------------- For a class ``cls``, creating a new instance using ``cls(xxx)`` requires multiple calls. At least one intermediate object is created for each call in the sequence ``type.__call__``, ``cls.__new__``, ``cls.__init__``. So it makes a lot of sense to use vectorcall for calling classes. This really means implementing the vectorcall protocol for ``type``. Some of the most commonly used classes will use this protocol, probably ``range``, ``list``, ``str``, and ``type``. The ``PyMethodDef`` protocol and Argument Clinic ------------------------------------------------ Argument Clinic [4]_ automatically generates wrapper functions around lower-level callables, providing safe unboxing of primitive types and other safety checks. Argument Clinic could be extended to generate wrapper objects conforming to the new ``vectorcall`` protocol. This will allow execution to flow from the caller to the Argument Clinic generated wrapper and thence to the hand-written code with only a single indirection. Third-party extension classes using vectorcall ============================================== To enable call performance on a par with Python functions and built-in functions, third-party callables should include a ``vectorcallfunc`` function pointer, set ``tp_vectorcall_offset`` to the correct value and add the ``_Py_TPFLAGS_HAVE_VECTORCALL`` flag. Any class that does this must implement the ``tp_call`` function and make sure its behaviour is consistent with the ``vectorcallfunc`` function. Setting ``tp_call`` to ``PyVectorcall_Call`` is sufficient. Performance implications of these changes ========================================= This PEP should not have much impact on the performance of existing code (neither in the positive nor the negative sense). It is mainly meant to allow efficient new code to be written, not to make existing code faster. Nevertheless, this PEP optimizes for ``METH_FASTCALL`` functions. Performance of functions using ``METH_VARARGS`` will become slightly worse. Stable ABI ========== Nothing from this PEP is added to the stable ABI (PEP 384). Alternative Suggestions ======================= bpo-29259 --------- PEP 590 is close to what was proposed in bpo-29259 [#bpo29259]_. The main difference is that this PEP stores the function pointer in the instance rather than in the class. This makes more sense for implementing functions in C, where every instance corresponds to a different C function. It also allows optimizing ``type.__call__``, which is not possible with bpo-29259. PEP 576 and PEP 580 ------------------- Both PEP 576 and PEP 580 are designed to enable 3rd party objects to be both expressive and performant (on a par with CPython objects). The purpose of this PEP is provide a uniform way to call objects in the CPython ecosystem that is both expressive and as performant as possible. This PEP is broader in scope than PEP 576 and uses variable rather than fixed offset function-pointers. The underlying calling convention is similar. Because PEP 576 only allows a fixed offset for the function pointer, it would not allow the improvements to any objects with constraints on their layout. PEP 580 proposes a major change to the ``PyMethodDef`` protocol used to define builtin functions. This PEP provides a more general and simpler mechanism in the form of a new calling convention. This PEP also extends the ``PyMethodDef`` protocol, but merely to formalise existing conventions. Other rejected approaches ------------------------- A longer, 6 argument, form combining both the vector and optional tuple and dictionary arguments was considered. However, it was found that the code to convert between it and the old ``tp_call`` form was overly cumbersome and inefficient. Also, since only 4 arguments are passed in registers on x64 Windows, the two extra arguments would have non-neglible costs. Removing any special cases and making all calls use the ``tp_call`` form was also considered. However, unless a much more efficient way was found to create and destroy tuples, and to a lesser extent dictionaries, then it would be too slow. Acknowledgements ================ Victor Stinner for developing the original "fastcall" calling convention internally to CPython. This PEP codifies and extends his work. References ========== .. [#bpo29259] Add tp_fastcall to PyTypeObject: support FASTCALL calling convention for all callable objects, https://bugs.python.org/issue29259 .. [2] tp_call/PyObject_Call calling convention https://docs.python.org/3/c-api/typeobj.html#c.PyTypeObject.tp_call .. [3] Using PY_VECTORCALL_ARGUMENTS_OFFSET in callee https://github.com/markshannon/cpython/blob/vectorcall-minimal/Objects/classobject.c#L53 .. [4] Argument Clinic https://docs.python.org/3/howto/clinic.html Reference implementation ======================== A minimal implementation can be found at https://github.com/markshannon/cpython/tree/vectorcall-minimal Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From guido at python.org Wed May 22 10:07:21 2019 From: guido at python.org (Guido van Rossum) Date: Wed, 22 May 2019 07:07:21 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> Message-ID: Christian, Please don't touch nntplib. Also I think telnetlib should stay. On Wed, May 22, 2019 at 5:44 AM Berker Peksa? wrote: > On Tue, May 21, 2019 at 7:11 PM Christian Heimes > wrote: > > > > On 21/05/2019 17.31, Antoine Pitrou wrote: > > > > > > As I said, if the main annoyance with nntplib is the sporadic test > > > failures, then the relevant tests can be disabled on CI. > > > > > > NNTP itself is still used, even if less and less. > > > > I don't like the idea to drop a third of the test cases for nntplib -- > and the 30% that actually test that Python's NNTP library can talk to an > actual NNTP server. IMHO it's more beneficial for core development and for > nntplib to have it maintained by somebody that cares about the library. > > https://github.com/python/cpython/pull/9461 should fix most of the > annoyances with test_nntplib. It uses a local server instead of an > external one. > > --Berker > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 22 11:19:48 2019 From: brett at python.org (Brett Cannon) Date: Wed, 22 May 2019 08:19:48 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <20190522121237.021cbec4@fsol> References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> <20190522121237.021cbec4@fsol> Message-ID: On Wed., May 22, 2019, 03:13 Antoine Pitrou, wrote: > On Tue, 21 May 2019 17:44:16 -0700 > Brett Cannon wrote: > > > > > > So I should never have added those tests and then we wouldn't be > talking > > > about removing nntplib. > > > > > > > Not necessarily. I suspect it still would have been listed, you would > have > > objected, and someone may have looked at the tests to point out it was > only > > unit tests with no integration tests against a live service like e.g. the > > pythontest.net-related tests and so it was already neglected. > > But that's moving the goalposts. I was just replying to your hypothetical with another hypothetical. The PEP talks about "cruft, > unnecessary duplication of functionality, and dispensable features". Not > modules which have an insufficient test suite. > To me, untested code is a potential signal of cruft. > > It also doesn't help that no one is listed in the experts index for the > > module either. > > Right, but that's the case for many modules. > I never said I didn't view this PEP as conservative. ? Anyway, my support of dropping nntplib is +0 so I personally support Christian if he wants to keep on the list but I'm also not that bothered if he wants to take it off the list. -Brett > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed May 22 12:16:28 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 22 May 2019 12:16:28 -0400 Subject: [Python-Dev] Missing testcase for bpo-34125 In-Reply-To: <5CE5195D.9030903@UGent.be> References: <5CE5195D.9030903@UGent.be> Message-ID: On 5/22/2019 5:41 AM, Jeroen Demeyer wrote: > Could somebody please merge https://github.com/python/cpython/pull/13461 > > It adds a missing testcase for bpo-34125. This is testing code which is > affected by PEP 590, so missing this test might accidentally break > CPython if we screw up with implementing PEP 590. Merged by Robert Collins. -- Terry Jan Reedy From v+python at g.nevcal.com Wed May 22 14:56:55 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 22 May 2019 11:56:55 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <23781.5568.183871.73668@turnbull.sk.tsukuba.ac.jp> References: <20190521130111.GJ4221@ando.pearwood.info> <20190521195948.b4cklbgdzomyfpza@python.ca> <23781.5568.183871.73668@turnbull.sk.tsukuba.ac.jp> Message-ID: Between this discussion and Steve Dower's recently referenced blog post at https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/from which I quote below: > It?s been widely known for many years that Windows is the only > mainstream operating system that does not include a Python interpreter > out of the box. and Victor's and Eric's work to make embedding Python easier, it makes me wonder when emacs will include a Python interpreter as an extension language. On 5/22/2019 2:26 AM, Stephen J. Turnbull wrote: > Neil Schemenauer writes: > > > Here is an alternative, straw man, proposal. Split the CPython repo > > into two parts: > > > > - core Python: minimal possible stdlib > > - everything else > > I take issue with the characterization of "straw man," it's a > practical idea that turns out to be not so easy to implement. > > We tried to do this with XEmacs, and found it just didn't work for us. > It requires a lot of effort to pare a distribution down to a minimal > core. (In fact, we never got close.) We gave up on that idea after > only a year, and this was an age and an environment that did not have > pip-like support for installation of 3rd-party packages (which you > deprecate as "not really a great solution"), and a lot of people were > still downloading at ISDN speeds or worse. The demand for a minimal > core was correspondingly greater than today, I think. > > We did bundle most applications such as MUAs and even C/C++ > development support into sumo packages (one for general use, one for > packages that didn't function at all without the "MULE" multilingual > environment which was optional in XEmacs). But we still ended up with > many packages staying in the core distribution, because other parts of > the core or the installation process used them, and as you say "If I > have a script that uses one of these modules and it gets removed, my > script breaks." That annoyed core developers as much as anyone, and > they generally had priorities that did not emphasize minimalism (to > say the least). > > > When Python is released, provide installers for a Python that only > > includes the "core" part and a second installer that includes > > everything else. I realize this is more work for the release team. > > Hopefully with some scripting, it will not be too labour intensive. > > Creating the distributions (core and sumo) was not particularly > labor-intensive for us because we mostly didn't distribute binaries. > A few moderately baroque scripts did the trick. Automation has > advanced, but due to the amount of it it's still fragile, especially > when you're producing turn-key binary installers. > > As I mentioned above, it wasn't the release process that was the > bottleneck, it was other factors that made it hard to reduce the > core. We had an "xemacs-base" package that provided a set of "extra > tools" that some packages required, and libraries moved in and out of > it pretty much every release as some developer got tired of telling > people to install xemacs-base (back to core) or working around missing > features (move it to xemacs-base where it could be updated easily). > > These are the same things that Python maintainers mention as pros and > cons of having things in stdlib or out on PyPI. We learned there is a > sweet spot. I know we didn't hit it most of the time, but it's very > clear that both extremes are hard, if not infeasible, to maintain. > (At the maximalist extreme, even GNU Emacs doesn't really try to > include everything written in Emacs Lisp any more.) > > > To help the people who need 100s or 1000s of extra PyPI packages, > > we could develop a tool that creates a "sumo" Python installer, > > grabbing packages from PyPI and building a installer package. > > This could be done by any third party (it wouldn't even need to be a > public distribution, it could easily be an enterprise that open > sources an internal tool or a GSoC project). And it could be hosted > on PyPI itself, since it's not the kind of thing that enterprises need > to install on 100,000 workstations. > > > To install that package, you would not need network access. > > AIUI, it's not the network access that is usually the problem these > days, although it can be annoying if net access is flaky or if you > have to sneakernet USBs around. Still, "sneakernet" is basically a > one-off cost. The problem is that collections of random software from > untrusted sources are verboten in the enterprise. (Ah, those Walnut > Creek CDs were wonderful! I wonder what a modern AV tool would find > in them?!) I suppose that in those situations people already have the > necessary tooling to roll the sumo or provide an intranet repository; > the problem is navigating the red tape. > > > That doesn't need to happen right away. Also, maybe other Python > > distributions can fill that need if core Python devs don't want to > > build it. > > I don't think this is about what core devs do or don't want to build. > I also don't feel a strong sense among the core developers that they > don't want to maintain a "batteries included" standard library. Sure, > there's Christian's PEP, but there's also deliberate activity to > promote more participation at the core (ie, committer) level. AIUI, > Christian's PEP is about transparency of what is already happening > (WONTFIX-ing or long periods of inactivity on issues with modules of > "low importance" to the active core developers), and a first step > toward making a managed process for the deprecation of modules we > can't afford to maintain given others we have committed to maintain. > > Steve -------------- next part -------------- An HTML attachment was scrubbed... URL: From sean.wallitsch at dreamworks.com Wed May 22 14:49:27 2019 From: sean.wallitsch at dreamworks.com (Sean Wallitsch) Date: Wed, 22 May 2019 11:49:27 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> <20190522121237.021cbec4@fsol> Message-ID: Dear python-dev, I'm writing to provide some feedback on PEP-594, primarily the proposed deprecation and reason for the removal of the aifc and audioop libraries. The post production film industry continues to make heavy use of AIFFs, as completely uncompressed audio is preferred. Support for the consumer alternatives (ALAC, FLAC) is virtually non-existent, with no movement towards adoption of those formats. Even Apple's own professional editing tool Final Cut Pro does not support ALAC. Many of the applications also support WAV, but not all. Removal of this module from the standard library is complicated by the fact that a large number of film industry facilities have extremely limited internet access for security reasons. This does not make it impossible to get a library from pypi, but speaking to those devs has made me aware of what a painful process that is for them. They have benefited greatly from aifc's inclusion in the standard library. While development of the aifc and audioop libraries has slowed, this looks consistent with mature projects and file formats that are not undergoing constant change. Checking the bug tracker for aifc and audioop does not paint a picture of these being broken or leaky modules requiring a lot of dev time, and there are 2 community sourced pull requests in progress right now. The film industry may not make up a large chunk of python devs, but it is python that makes tent pole films like Avengers possible. Without the python digital asset management pipelines, films of that scale wouldn't be possible. Thank you for your time and consideration, Sean Wallitsch Lead Technical Director, DreamWorks Animation On Wed, May 22, 2019 at 8:21 AM Brett Cannon wrote: > > > On Wed., May 22, 2019, 03:13 Antoine Pitrou, wrote: > >> On Tue, 21 May 2019 17:44:16 -0700 >> Brett Cannon wrote: >> > > >> > > So I should never have added those tests and then we wouldn't be >> talking >> > > about removing nntplib. >> > > >> > >> > Not necessarily. I suspect it still would have been listed, you would >> have >> > objected, and someone may have looked at the tests to point out it was >> only >> > unit tests with no integration tests against a live service like e.g. >> the >> > pythontest.net-related tests and so it was already neglected. >> >> But that's moving the goalposts. > > > I was just replying to your hypothetical with another hypothetical. > > The PEP talks about "cruft, >> unnecessary duplication of functionality, and dispensable features". Not >> modules which have an insufficient test suite. >> > > To me, untested code is a potential signal of cruft. > > >> > It also doesn't help that no one is listed in the experts index for the >> > module either. >> >> Right, but that's the case for many modules. >> > > I never said I didn't view this PEP as conservative. ? > > Anyway, my support of dropping nntplib is +0 so I personally support > Christian if he wants to keep on the list but I'm also not that bothered if > he wants to take it off the list. > > -Brett > > >> Regards >> >> Antoine. >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/sean.wallitsch%40dreamworks.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Wed May 22 15:15:20 2019 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 22 May 2019 14:15:20 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. Message-ID: There's a Stackoverflow report[1] I suspect is worth looking into, but it requires far more RAM (over 80GB) than I have). The OP whittled it down to a reasonably brief & straightforward pure Python 3 program. It builds a ternary search tree, with perhaps a billion nodes. The problem is that it "takes forever" for Python to reclaim the tree storage when it goes out of scope (the author waited at least hours). Alas, the OP said it takes about 45 minutes to build the tree, and the problem goes away if the tree is materially smaller. So it takes a long time just to try once. With a tree about 10x smaller, for me it takes about 45 seconds for Python to reclaim the tree storage. The tree is deep enough that the "trashcan" may be implicated, and the node objects are small enough that obmalloc carves them out of a (relatively) great many arenas. Those are two ways in which Python _may_ be to blame. The sheer number of objects involved may be provoking edge cases we normally never see. But, for a start, it would be good to know if anyone else can actually reproduce the problem. [1] https://stackoverflow.com/questions/56228799/python-hangs-indefinitely-trying-to-delete-deeply-recursive-object From njs at pobox.com Wed May 22 15:34:58 2019 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 22 May 2019 12:34:58 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> <20190522121237.021cbec4@fsol> Message-ID: On Wed, May 22, 2019, 12:14 Sean Wallitsch wrote: > Dear python-dev, > > I'm writing to provide some feedback on PEP-594, primarily the proposed > deprecation and reason for the removal of the aifc and audioop libraries. > > The post production film industry continues to make heavy use of AIFFs, as > completely uncompressed audio is preferred. Support for the consumer > alternatives (ALAC, FLAC) is virtually non-existent, with no movement > towards adoption of those formats. Even Apple's own professional editing > tool Final Cut Pro does not support ALAC. Many of the applications also > support WAV, but not all. > > Removal of this module from the standard library is complicated by the > fact that a large number of film industry facilities have extremely limited > internet access for security reasons. This does not make it impossible to > get a library from pypi, but speaking to those devs has made me aware of > what a painful process that is for them. They have benefited greatly from > aifc's inclusion in the standard library. > That's really helpful data, thank you! Is audioop also used? You mention both aifc and audioop at the beginning and end of your message, but all the details in the middle focus on just aifc. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From oscar.j.benjamin at gmail.com Wed May 22 15:37:32 2019 From: oscar.j.benjamin at gmail.com (Oscar Benjamin) Date: Wed, 22 May 2019 20:37:32 +0100 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: On Tue, 21 May 2019 at 21:32, Steve Dower wrote: > > In the next Windows 10 update that starts rolling out today, we > (Microsoft) have added "python.exe" and "python3.exe" commands that are > installed on PATH *by default* and will open the Microsoft Store at the > page where we (Python core team) publish our build. > > This makes it a 1-2 click process to get from a clean machine to having > a usable Python install ("python.exe" -> opens Store -> "Get it Free" -> > "python.exe" now works!) This is great. Thanks for this Steve. A lot of novices struggle to get Python installed and running on Windows. Not being on PATH is a particular problem for them so this looks very useful. Of course it doesn't take long before a novice programmer needs some other libraries (numpy, matploblib etc for my students) so I wonder how that part works. Is this doing a single-user or system wide install? After install is IDLE available via start menu? How does pip fit into this? Will pip be on PATH after install? -- Oscar From njs at pobox.com Wed May 22 15:42:04 2019 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 22 May 2019 12:42:04 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> <20190522101921.GM4221@ando.pearwood.info> Message-ID: On Wed, May 22, 2019, 04:32 Christian Heimes wrote: > On 22/05/2019 12.19, Steven D'Aprano wrote: > > I don't think this PEP should become a document about "Why you should > > use PAM". I appreciate that from your perspective as a Red Hat security > > guy, you want everyone to use best practices as you see them, but it > > isn't Python's position to convince Linux distros or users to use PAM. > > I think the PEP should make clear why spwd is bad and pining for The > Fjords. The document should point users to correct alternatives. There is > no correct and secure way to use the spwd module to verify user accounts. > Any use of spwd for logins introduces critical security bugs. > > By the way, all relevant BSD, Linux, and Darwin (macOS) distributions come > with PAM support. Almost all use PAM by default. AFAIK only the minimal > Alpine container does not have PAM installed by default. This is not Red > Hat trying to evangelize the world. PAM is *the* industry standards on > Unix-like OS. > The removal of spwd seems reasonable to me, and I don't think you need to write 20 seperate PEPs for each module, but I do think you should split the spwd/crypt modules off into their own PEP. The discussion about these modules is qualitatively different than some of the others (the security implications etc.), and trying to mix qualitatively different discussions always makes people frustrated. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Wed May 22 16:09:39 2019 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 22 May 2019 23:09:39 +0300 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: On 21.05.2019 23:30, Steve Dower wrote: > Hi all > > Just sharing this here because I think it's important for us to be aware of it - I'm not trying to promote or sell anything here :) (Those > who were at the language summit have seen this already.) > > In the next Windows 10 update that starts rolling out today, we (Microsoft) have added "python.exe" and "python3.exe" commands that are > installed on PATH *by default* and will open the Microsoft Store at the page where we (Python core team) publish our build. > > This makes it a 1-2 click process to get from a clean machine to having a usable Python install ("python.exe" -> opens Store -> "Get it > Free" -> "python.exe" now works!) > > The associated blog post: > > https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/ > > Here are answers to a few questions that I assume will come up, at least from this audience that understands the issues better than most: > > * if someone had installed Python and put it on PATH with our installer, this new command *does not* interfere > * if someone had manually modified their own PATH, they *may* see some interference (but we [Microsoft] decided this was an acceptable risk) > * the Python 3.7 installed from the store will not auto-update to 3.8, but when 3.8 is released we (Microsoft) will update the redirect to > point at it > * if you pass arguments to the redirect command, it just exits with an error code - you only get the Store page if you run it without > arguments > * once the Store package is installed, the redirect command is replaced (this required a new feature in the OS). If you install with the > regular installer and update PATH, or active a venv, it will add it *before* the redirect. So these scenarios should be all good. > > I'm happy to answer other questions here. The long-term contact for this integration is python (at) microsoft.com, which right now will > come to me. > As someone whose job is to diagnose and fix problems with running software: Are there patches in your release? Do you provide corresponding sources and debug symbols for it? > And on a personal note, I'm very excited that we (Microsoft) got the approval to do this. Getting *anything* added to Windows is a big > task, so it's a reflection of the popularity and support for Python that's growing within Microsoft that we were able to make this happen. > That's due to every contributor, both to the core runtime and the ecosystem. I hope this will only help us improve the availability of > Python for users and make it an easier choice for dev tasks in the future. > > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From sean.wallitsch at dreamworks.com Wed May 22 16:21:39 2019 From: sean.wallitsch at dreamworks.com (Sean Wallitsch) Date: Wed, 22 May 2019 13:21:39 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> <20190522121237.021cbec4@fsol> Message-ID: My apologies for that oversight. My understanding is that many of the methods present in aifc depend heavily on audioop for reading and writing. On Wed, May 22, 2019 at 12:35 PM Nathaniel Smith wrote: > On Wed, May 22, 2019, 12:14 Sean Wallitsch > wrote: > >> Dear python-dev, >> >> I'm writing to provide some feedback on PEP-594, primarily the proposed >> deprecation and reason for the removal of the aifc and audioop libraries. >> >> The post production film industry continues to make heavy use of AIFFs, >> as completely uncompressed audio is preferred. Support for the consumer >> alternatives (ALAC, FLAC) is virtually non-existent, with no movement >> towards adoption of those formats. Even Apple's own professional editing >> tool Final Cut Pro does not support ALAC. Many of the applications also >> support WAV, but not all. >> >> Removal of this module from the standard library is complicated by the >> fact that a large number of film industry facilities have extremely limited >> internet access for security reasons. This does not make it impossible to >> get a library from pypi, but speaking to those devs has made me aware of >> what a painful process that is for them. They have benefited greatly from >> aifc's inclusion in the standard library. >> > > That's really helpful data, thank you! > > Is audioop also used? You mention both aifc and audioop at the beginning > and end of your message, but all the details in the middle focus on just > aifc. > > -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed May 22 16:48:40 2019 From: steve.dower at python.org (Steve Dower) Date: Wed, 22 May 2019 13:48:40 -0700 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: <8eafa528-62e6-6fd0-0114-0743205e124a@python.org> On 22May2019 1237, Oscar Benjamin wrote: > On Tue, 21 May 2019 at 21:32, Steve Dower wrote: >> >> In the next Windows 10 update that starts rolling out today, we >> (Microsoft) have added "python.exe" and "python3.exe" commands that are >> installed on PATH *by default* and will open the Microsoft Store at the >> page where we (Python core team) publish our build. >> >> This makes it a 1-2 click process to get from a clean machine to having >> a usable Python install ("python.exe" -> opens Store -> "Get it Free" -> >> "python.exe" now works!) > > This is great. Thanks for this Steve. A lot of novices struggle to get > Python installed and running on Windows. Not being on PATH is a > particular problem for them so this looks very useful. Of course it > doesn't take long before a novice programmer needs some other > libraries (numpy, matploblib etc for my students) so I wonder how that > part works. pip/pip3/pip3.7 all work as expected (python -m pip has a bug that will be fixed once we get a new version of pip bundled - I've already made the fix on their side to correct it, but the workaround is to pass --user). python -m venv also works as expected (now... after some bug fixes from the initial release). > Is this doing a single-user or system wide install? It's an interesting hybrid. The files are installed once per system, and are locked down even from being changed by administrators. But each user has to install it themselves (which is very quick if another user has already got it) and they will have their own packages/settings/etc. > After install is IDLE available via start menu? Yes. Also at the command line as idle/idle3/idle3.7 > How does pip fit into this? Will pip be on PATH after install? Yes, as pip/pip3/pip3.7. Currently, installing a tool with its own entry point will not put it on PATH but will prompt you to put your Scripts directory there yourself. There's no good workaround for this, other than using -m or pipx. Cheers, Steve From steve.dower at python.org Wed May 22 16:52:37 2019 From: steve.dower at python.org (Steve Dower) Date: Wed, 22 May 2019 13:52:37 -0700 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: On 22May2019 1309, Ivan Pozdeev via Python-Dev wrote: > As someone whose job is to diagnose and fix problems with running software: > Are there patches in your release? Do you provide corresponding sources > and debug symbols for it? You can find the sources at https://github.com/python/cpython :) I'm working on getting debug symbols packaged for the next release, but once I do that they'll be exactly the same binaries as in the traditional installer on python.org. (Right now they are two separate builds of the same source.) The package on the Store is not a Microsoft build or release of Python - it's published by whoever the Windows build manager is at the time. Just to be confusing, it's me right now, but the actual install is not owned or managed by Microsoft - just endorsed and linked. Cheers, Steve From vano at mail.mipt.ru Wed May 22 18:14:23 2019 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 23 May 2019 01:14:23 +0300 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: <1adf8a66-d325-5f5d-9fca-5f50d232243b@mail.mipt.ru> On 22.05.2019 23:52, Steve Dower wrote: > On 22May2019 1309, Ivan Pozdeev via Python-Dev wrote: >> As someone whose job is to diagnose and fix problems with running software: >> Are there patches in your release? Do you provide corresponding sources and debug symbols for it? > > You can find the sources at https://github.com/python/cpython :) > For Anaconda, this is not so, they apply private patches. So I had to make sure. > I'm working on getting debug symbols packaged for the next release, but once I do that they'll be exactly the same binaries as in the > traditional installer on python.org. (Right now they are two separate builds of the same source.) > > The package on the Store is not a Microsoft build or release of Python - it's published by whoever the Windows build manager is at the > time. Just to be confusing, it's me right now, but the actual install is not owned or managed by Microsoft - just endorsed and linked. > > Cheers, > Steve -- Regards, Ivan From mingw.android at gmail.com Wed May 22 18:23:39 2019 From: mingw.android at gmail.com (Ray Donnelly) Date: Thu, 23 May 2019 00:23:39 +0200 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <1adf8a66-d325-5f5d-9fca-5f50d232243b@mail.mipt.ru> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> <1adf8a66-d325-5f5d-9fca-5f50d232243b@mail.mipt.ru> Message-ID: On Thu, May 23, 2019, 12:17 AM Ivan Pozdeev via Python-Dev < python-dev at python.org> wrote: > On 22.05.2019 23:52, Steve Dower wrote: > > On 22May2019 1309, Ivan Pozdeev via Python-Dev wrote: > >> As someone whose job is to diagnose and fix problems with running > software: > >> Are there patches in your release? Do you provide corresponding sources > and debug symbols for it? > > > > You can find the sources at https://github.com/python/cpython :) > > > For Anaconda, this is not so, they apply private patches. So I had to make > sure. > There's nothing 'private' about them. https://github.com/AnacondaRecipes/python-feedstock/tree/master/recipe > > > > I'm working on getting debug symbols packaged for the next release, but > once I do that they'll be exactly the same binaries as in the > > traditional installer on python.org. (Right now they are two separate > builds of the same source.) > > > > The package on the Store is not a Microsoft build or release of Python - > it's published by whoever the Windows build manager is at the > > time. Just to be confusing, it's me right now, but the actual install is > not owned or managed by Microsoft - just endorsed and linked. > > > > Cheers, > > Steve > > -- > Regards, > Ivan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed May 22 20:58:31 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 23 May 2019 10:58:31 +1000 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> <20190522101921.GM4221@ando.pearwood.info> Message-ID: <20190523005831.GP4221@ando.pearwood.info> On Wed, May 22, 2019 at 01:31:18PM +0200, Christian Heimes wrote: > On 22/05/2019 12.19, Steven D'Aprano wrote: > > I don't think this PEP should become a document about "Why you should > > use PAM". I appreciate that from your perspective as a Red Hat security > > guy, you want everyone to use best practices as you see them, but it > > isn't Python's position to convince Linux distros or users to use PAM. > > I think the PEP should make clear why spwd is bad and pining for The > Fjords. The document should point users to correct alternatives. There > is no correct and secure way to use the spwd module to verify user > accounts. Any use of spwd for logins introduces critical security > bugs. When you use absolute language about security without considering threat models, like "there is no ... way" and "Any use", you lose credibility in my eyes. I have a Linux desktop where I am the only user but not the only user account. If I use spwd, what vulnerabilty am I introducing? That's not a rhetorical question. If spwd does introduce a threat that isn't already there, then please educate me, I genuinely want to know. But if it doesn't, then you ought to tone down the absolutist language about "no way" and "any use" because that is factually untrue. Later, you wrote: > Steven, I feel like you are turning my own words and arguments against me. Yes? Is that supposed to be a bad thing? If you make an argument which is intended to support the PEP, but it actually undercuts the PEP, then you should fix the argument or acknowledge that the case for the PEP is not as strong as you hoped. We do you no favours by ignoring weak or incoherent arguments. PEPs are supposed to be fair, not partisan. Of course you have a position on this matter: you want to see spwd removed, and you want to put the best possible case for it. But to be a fair PEP, you also need to acknowledge counter-arguments and weaknesses in the argument, not just hope people won't notice them and then protest when they do. > By the way, all relevant BSD, Linux, and Darwin (macOS) distributions > come with PAM support. Almost all use PAM by default. AFAIK only the > minimal Alpine container does not have PAM installed by default. This > is not Red Hat trying to evangelize the world. PAM is *the* industry > standards on Unix-like OS. I appreciate that PAM is the standard, but Unix-like users are often an individualistic lot and "PAM is the default" doesn't mean "everyone uses PAM". See, for example, Arfrever's earlier post. This PEP isn't about using PAM. It's about removing spwd and crypt. It's okay to say that PAM is the industry standard, you don't have to justify that in depth on the PEP. Nor do you need to justify why most people should be using PAM, but you ought to acknowledge that there may be some people who aren't. For those who (rightly or wrongly) won't, can't or simply don't know how to use PAM, removing spwd is a functional regression. As PEP author, your job is to justify that functional regression and offer an alternative. We have a number of options: - tell them to re-implement spwd (but that undercuts your argument for removal); - tell them to use a third party module (but which?); - add a replacement module that Does The Right Thing; - or at least a recipe somewhere (but once the spwd module is removed, where will the recipe live?); - anything else? As PEP author, you get to choose which option the PEP suggests, but if you pick the one that undercuts the argument for removal, don't be surprised if people notice and consider the case for removal to be weak. -- Steven From v+python at g.nevcal.com Thu May 23 01:23:34 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Wed, 22 May 2019 22:23:34 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <8eba9ec6-5f24-ccad-1218-999173f9f252@python.org> References: <04b145f0-e402-9ded-0844-9a0acaca9b44@python.org> <7ab85a2e-d99f-f2c5-6aef-bb8a0b61ab3f@g.nevcal.com> <303943c0-0c48-ec35-cbd9-cbc357dd3e7e@g.nevcal.com> <8eba9ec6-5f24-ccad-1218-999173f9f252@python.org> Message-ID: <300940ca-cc63-5c78-bcba-0bbd37dbdb26@g.nevcal.com> On 5/22/2019 4:09 AM, Christian Heimes wrote: > On 22/05/2019 01.11, Glenn Linderman wrote: >> On 5/21/2019 2:00 PM, Nathaniel Smith wrote: >>> On Tue, May 21, 2019 at 10:43 AM Glenn Linderman wrote: >>>> After maintaining my own version of http.server to fix or workaround some of its deficiencies for some years, I discovered bottle.py. It has far more capability, is far better documented, and is just as quick to deploy. While I haven't yet converted all past projects to use bottle.py, it will likely happen in time, unless something even simpler to use is discovered, although I can hardly imagine that happening. >>> bottle.py uses http.server for its local development mode (the one you >>> see in their quickstart example at the top of their README). Same with >>> flask, django, and probably a bunch of other frameworks. It's *very* >>> widely used. >>> >>> -n >>> >> The source for bottle.py version 0.13-dev has an import for http.client, but not http.server. I hadn't tracked down every indirect dependency in the bottle.py source code, but it seems that if one uses the "default server" for bottle, that it is "wsgiref", imported from wsgiref.simple_server, and that in turn does import BaseHTTPRequestHandler and HTTPServer from http.server. >> >> It is the higher-level code in http.server that has significant deficiencies that have caused me problems over the years... a "SimpleHTTPRequestHandler" that is so simple it doesn't do POST, PUT or PASTE, a "CGIHTTPRequestHandler" that only implements part of the CGI protocol, only CGI support in POST, no support for PUT or PASTE, and no support for https, and not much bug fix activity in those areas. >> >> Maybe http.server should be split into the "basic parts" (used by bottle.py, and other frameworks), and the "higher-level parts", which could then be discarded by this PEP! At this point, though, I'd have to agree that the whole should not be discarded. Thanks for making me dig deeper. > The idea has merrit. However I feel its out of scope for the PEP. The http.server module and socketserver module are still widely used for debug and toy examples. > > Could you please open a bug to track your proposal? We may pursue it in a couple of years from now. issue 37018 -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Thu May 23 01:44:24 2019 From: christian at python.org (Christian Heimes) Date: Thu, 23 May 2019 07:44:24 +0200 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <20190523005831.GP4221@ando.pearwood.info> References: <20190520232749.442c45c7@fsol> <11d2680b-3def-08c0-0767-5edd227e9d98@python.org> <20190522101921.GM4221@ando.pearwood.info> <20190523005831.GP4221@ando.pearwood.info> Message-ID: <4971aab5-2391-9c8b-579d-b716dd7b0527@python.org> On 23/05/2019 02.58, Steven D'Aprano wrote: > On Wed, May 22, 2019 at 01:31:18PM +0200, Christian Heimes wrote: >> On 22/05/2019 12.19, Steven D'Aprano wrote: >>> I don't think this PEP should become a document about "Why you should >>> use PAM". I appreciate that from your perspective as a Red Hat security >>> guy, you want everyone to use best practices as you see them, but it >>> isn't Python's position to convince Linux distros or users to use PAM. >> > >> I think the PEP should make clear why spwd is bad and pining for The >> Fjords. The document should point users to correct alternatives. There >> is no correct and secure way to use the spwd module to verify user >> accounts. Any use of spwd for logins introduces critical security >> bugs. > > When you use absolute language about security without considering > threat models, like "there is no ... way" and "Any use", you lose > credibility in my eyes. > > I have a Linux desktop where I am the only user but not the only user > account. If I use spwd, what vulnerabilty am I introducing? That's not a > rhetorical question. If spwd does introduce a threat that isn't already > there, then please educate me, I genuinely want to know. I can give you more details once I have resolved some CVEs. The problem can result into full system compromise by a local or remote attacker without any trace in the system audit and security logs. Depending on other circumstances, the issue is CVSS HIGH to CRITICAL, perhaps up to CVSS score 9.9. Christian From tritium-list at sdamon.com Thu May 23 04:00:03 2019 From: tritium-list at sdamon.com (Alex Walters) Date: Thu, 23 May 2019 04:00:03 -0400 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: <016601d5113d$87bd8740$973895c0$@sdamon.com> I've watched the entire thread and its taken me a few days to put a finger on what bothers me about it. In my opinion, this shouldn't be a pep describing the list of modules that need to go as "dead batteries", but should be a process pep describing how dead batteries should be removed, and the individual modules should be given their own pep. I think reactions to individual module peps will give a better indication of if it's a used module or not. > -----Original Message----- > From: Python-Dev list=sdamon.com at python.org> On Behalf Of Christian Heimes > Sent: Monday, May 20, 2019 4:15 PM > To: Python Dev > Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard > library > > Hi, > > here is the first version of my PEP 594 to deprecate and eventually remove > modules from the standard library. The PEP started last year with talk during > the Python Language Summit 2018, https://lwn.net/Articles/755229/. > > The PEP can be confirmed in two stages. I'm not planning any code changes > for 3.8. Instead I only like to document a bunch of modules as deprecated. > Active deprecation is planned for 3.9 and removal for 3.10. The long > deprecation phase gives us 3 years to change our minds or handle edge > cases, too. > > Regards, > Christian > > > ---------------------------------------------------------------------------- > PEP: 594 > Title: Removing dead batteries from the standard library > Author: Christian Heimes > Status: Active > Type: Process > Content-Type: text/x-rst > Created: 20-May-2019 > Post-History: > > > Abstract > ======== > > This PEP proposed a list of standard library modules to be removed from the > standard library. The modules are mostly historic data formats and APIs that > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > Rationale > ========= > > Back in the early days of Python, the interpreter came with a large set of > useful modules. This was often refrained to as "batteries included" > philosophy and was one of the corner stones to Python's success story. > Users didn't have to figure out how to download and install separate > packages in order to write a simple web server or parse email. > > Times have changed. The introduction of the cheese shop (PyPI), setuptools, > and later pip, it became simple and straight forward to download and install > packages. Nowadays Python has a rich and vibrant ecosystem of third party > packages. It's pretty much standard to either install packages from PyPI or > use one of the many Python or Linux distributions. > > On the other hand, Python's standard library is piling up cruft, unnecessary > duplication of functionality, and dispensable features. This is undesirable > for several reasons. > > * Any additional module increases the maintenance cost for the Python core > development team. The team has limited resources, reduced maintenance > cost > frees development time for other improvements. > * Modules in the standard library are generally favored and seen as the > de-facto solution for a problem. A majority of users only pick 3rd party > modules to replace a stdlib module, when they have a compelling reason, > e.g. > lxml instead of `xml`. The removal of an unmaintained stdlib module > increases the chances of a community contributed module to become > widely > used. > * A lean and mean standard library benefits platforms with limited resources > like devices with just a few hundred kilobyte of storage (e.g. BBC > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > (e.g. pyodide) also benefit from reduced download size. > > The modules in the PEP have been selected for deprecation because their > removal is either least controversial or most beneficial. For example > least controversial are 30 years old multimedia formats like ``sunau`` > audio format, which was used on SPARC and NeXT workstations in the late > 1980ties. The ``crypt`` module has fundamental flaws that are better solved > outside the standard library. > > This PEP also designates some modules as not scheduled for removal. Some > modules have been deprecated for several releases or seem unnecessary at > first glance. However it is beneficial to keep the modules in the standard > library, mostly for environments where installing a package from PyPI is not > an option. This can be cooperate environments or class rooms where > external > code is not permitted without legal approval. > > * The usage of FTP is declining, but some files are still provided over > the FTP protocol or hosters offer FTP to upload content. Therefore > ``ftplib`` is going to stay. > * The ``optparse`` and ``getopt`` module are widely used. They are mature > modules with very low maintenance overhead. > * According to David Beazley [5]_ the ``wave`` module is easy to teach to > kids and can make crazy sounds. Making a computer generate crazy sounds > is > powerful and highly motivating exercise for a 9yo aspiring developer. It's > a fun battery to keep. > > > Deprecation schedule > ==================== > > 3.8 > --- > > This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be released > a few months before Python 2.7 will reach its end of lifetime. We expect that > Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and > 2020. To reduce churn and to allow a smooth transition from Python 2, > Python 3.8 will neither raise `DeprecationWarning` nor remove any > modules that have been scheduled for removal. Instead deprecated > modules will > just be *documented* as deprecated. Optionally modules may emit a > `PendingDeprecationWarning`. > > All deprecated modules will also undergo a feature freeze. No additional > features should be added. Bug should still be fixed. > > 3.9 > --- > > Starting with Python 3.9, deprecated modules will start issuing > `DeprecationWarning`. > > > 3.10 > ---- > > In 3.10 all deprecated modules will be removed from the CPython repository > together with tests, documentation, and autoconf rules. > > > PEP acceptance process > ====================== > > 3.8.0b1 is scheduled to be release shortly after the PEP is officially > submitted. Since it's improbable that the PEP will pass all stages of the > PEP process in time, I propose a two step acceptance process that is > analogous Python's two release deprecation process. > > The first *provisionally accepted* phase targets Python 3.8.0b1. In the first > phase no code is changes or removed. Modules are only documented as > deprecated. > > The final decision, which modules will be removed and how the removed > code > is preserved, can be delayed for another year. > > > Deprecated modules > ================== > > The modules are grouped as data encoding, multimedia, network, OS > interface, > and misc modules. The majority of modules are for old data formats or > old APIs. Some others are rarely useful and have better replacements on > PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with > audio processing. > > .. csv-table:: Table 1: Proposed modules deprecations > :header: "Module", "Deprecated in", "To be removed", "Replacement" > > aifc,3.8,3.10,\- > asynchat,3.8,3.10,asyncio > asyncore,3.8,3.10,asyncio > audioop,3.8,3.10,\- > binhex,3.8,3.10,\- > cgi,3.8,3.10,\- > cgitb,3.8,3.10,\- > chunk,3.8,3.10,\- > colorsys,**3.8?**,**3.10?**,\- > crypt,3.8,3.10,\- > fileinput,3.8,3.10,argparse > formatter,3.4,3.10,\- > fpectl,**3.7**,**3.7**,\- > getopt,**3.2**,**keep**,"argparse, optparse" > imghdr,3.8,3.10,\- > imp,**3.4**,3.10,importlib > lib2to3,\-,**keep**, > macpath,**3.7**,**3.8**,\- > msilib,3.8,3.10,\- > nntplib,3.8,3.10,\- > nis,3.8,3.10,\- > optparse,\-,**keep**,argparse > ossaudiodev,3.8,3.10,\- > pipes,3.8,3.10,subprocess > smtpd,**3.7**,3.10,aiosmtpd > sndhdr,3.8,3.10,\- > spwd,3.8,3.10,\- > sunau,3.8,3.10,\- > uu,3.8,3.10,\- > wave,\-,**keep**, > xdrlib,3.8,3.10,\- > > > Data encoding modules > --------------------- > > binhex > ~~~~~~ > > The `binhex `_ module > encodes > and decodes Apple Macintosh binhex4 data. It was originally developed for > TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to > encode binary email attachments. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > uu > ~~ > > The `uu `_ module provides > uuencode format, an old binary encoding format for email from 1980. The uu > format has been replaced by MIME. The uu codec is provided by the binascii > module. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > xdrlib > ~~~~~~ > > The `xdrlib `_ module > supports > the Sun External Data Representation Standard. XDR is an old binary > serialization format from 1987. These days it's rarely used outside > specialized domains like NFS. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Multimedia modules > ------------------ > > aifc > ~~~~ > > The `aifc `_ module provides > support for reading and writing AIFF and AIFF-C files. The Audio Interchange > File Format is an old audio format from 1988 based on Amiga IFF. It was most > commonly used on the Apple Macintosh. These days only few specialized > application use AIFF. > > Module type > pure Python (depends on `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > audioop > ~~~~~~~ > > The `audioop `_ module > contains helper functions to manipulate raw audio data and adaptive > differential pulse-code modulated audio data. The module is implemented in > C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ > module depend on `audioop`_ for some operations. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > colorsys > ~~~~~~~~ > > The `colorsys `_ module > defines color conversion functions between RGB, YIQ, HSL, and HSV > coordinate > systems. The Pillow library provides much faster conversation between > color systems. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `Pillow `_, > `colorspacious `_ > > chunk > ~~~~~ > > The `chunk `_ module > provides > support for reading and writing Electronic Arts' Interchange File Format. > IFF is an old audio file format originally introduced for Commodore and > Amiga. The format is no longer relevant. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > imghdr > ~~~~~~ > > The `imghdr `_ module is a > simple tool to guess the image file format from the first 32 bytes > of a file or buffer. It supports only a limited amount of formats and > neither returns resolution nor color depth. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > ossaudiodev > ~~~~~~~~~~~ > > The `ossaudiodev `_ > module provides support for Open Sound System, an interface to sound > playback and capture devices. OSS was initially free software, but later > support for newer sound devices and improvements were proprietary. Linux > community abandoned OSS in favor of ALSA [1]_. Some operation systems > like > OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > sndhdr > ~~~~~~ > > The `sndhdr `_ module is > similar to the `imghdr`_ module but for audio formats. It guesses file > format, channels, frame rate, and sample widths from the first 512 bytes of > a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and > other ancient formats. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > sunau > ~~~~~ > > The `sunau `_ module > provides > support for Sun AU sound format. It's yet another old, obsolete file format. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Networking modules > ------------------ > > asynchat > ~~~~~~~~ > > The `asynchat `_ module > is build on top of `asyncore`_ and has been deprecated since Python 3.6. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > asyncore > ~~~~~~~~ > > The `asyncore `_ module > was > the first module for asynchronous socket service clients and servers. It > has been replaced by asyncio and is deprecated since Python 3.6. > > The ``asyncore`` module is also used in stdlib tests. The tests for > ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly > based on ``asyncore``. These tests must be updated to use asyncio or > threading. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > > cgi > ~~~ > > The `cgi `_ module is a support > module for Common Gateway Interface (CGI) scripts. CGI is deemed as > inefficient because every incoming request is handled in a new process. PEP > 206 considers the module as *designed poorly and are now near-impossible > to fix*. > > Several people proposed to either keep the cgi module for features like > `cgi.parse_qs()` or move `cgi.escape()` to a different module. The > functions `cgi.parse_qs` and `cgi.parse_qsl` have been > deprecated for a while and are actually aliases for > `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The > function `cgi.quote` has been deprecated in favor of `html.quote` > with secure default values. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > cgitb > ~~~~~ > > The `cgitb `_ module is a > helper for the cgi module for configurable tracebacks. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > smtpd > ~~~~~ > > The `smtpd `_ module > provides > a simple implementation of a SMTP mail server. The module documentation > recommends ``aiosmtpd``. > > Module type > pure Python > Deprecated in > **3.7** > To be removed in > 3.10 > Substitute > aiosmtpd > > nntplib > ~~~~~~~ > > The `nntplib `_ module > implements the client side of the Network News Transfer Protocol (nntp). > News > groups used to be a dominant platform for online discussions. Over the last > two decades, news has been slowly but steadily replaced with mailing lists > and web-based discussion platforms. > > The ``nntplib`` tests have been the cause of additional work in the recent > past. Python only contains client side of NNTP. The test cases depend on > external news server. These servers were unstable in the past. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Operating system interface > -------------------------- > > crypt > ~~~~~ > > The `crypt `_ module > implements > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor > quality and insecure. Users are discouraged to use them. > > * The module is not available on Windows. Cross-platform application need > an alternative implementation any way. > * Only DES encryption is guarenteed to be available. DES has an extremely > limited key space of 2**56. > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > algorithm that is still secure. However it's in glibc and therefore not > commonly available on Linux. > * Depending on the platform, the ``crypt`` module is not thread safe. Only > implementations with ``crypt_r(3)`` are thread safe. > > Module type > C extension + Python module > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `bcrypt `_, > `passlib `_, > `argon2cffi `_, > hashlib module (PBKDF2, scrypt) > > macpath > ~~~~~~~ > > The `macpath `_ module > provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer > supported > > Module type > pure Python > Deprecated in > 3.7 > Removed in > 3.8 > Substitute > **none** > > nis > ~~~ > > The `nis `_ module provides > NIS/YP support. Network Information Service / Yellow Pages is an old and > deprecated directory service protocol developed by Sun Microsystems. It's > designed successor NIS+ from 1992 never took off. For a long time, libc's > Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more > powerful > and more secure replacement of NIS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > spwd > ~~~~ > > The `spwd `_ module > provides > direct access to Unix shadow password database using non-standard APIs. > In general it's a bad idea to use the spwd. The spwd circumvents system > security policies, it does not use the PAM stack, and is > only compatible with local user accounts. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > Misc modules > ------------ > > fileinput > ~~~~~~~~~ > > The `fileinput `_ module > implements a helpers to iterate over a list of files from ``sys.argv``. The > module predates the optparser and argparser module. The same > functionality > can be implemented with the argparser module. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > argparse > > formatter > ~~~~~~~~~ > > The `formatter `_ > module > is an old text formatting module which has been deprecated since Python > 3.4. > > Module type > pure Python > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > *n/a* > > imp > ~~~ > > The `imp `_ module is the > predecessor of the > `importlib `_ module. > Most > functions have been deprecated since Python 3.3 and the module since > Python 3.4. > > Module type > C extension > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > importlib > > msilib > ~~~~~~ > > The `msilib `_ package is a > Windows-only package. It supports the creation of Microsoft Installers (MSI). > The package also exposes additional APIs to create cabinet files (CAB). The > module is used to facilitate distutils to create MSI installers with > ``bdist_msi`` command. In the past it was used to create CPython's official > Windows installer, too. > > Microsoft is slowly moving away from MSI in favor of Windows 10 Apps > (AppX) > as new deployment model [3]_. > > Module type > C extension + Python code > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > pipes > ~~~~~ > > The `pipes `_ module > provides > helpers to pipe the input of one command into the output of another > command. > The module is built on top of ``os.popen``. Users are encouraged to use > the subprocess module instead. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > subprocess module > > Removed modules > =============== > > fpectl > ------ > > The `fpectl `_ module was > never built by default, its usage was discouraged and considered dangerous. > It also required a configure flag that caused an ABI incompatibility. The > module was removed in 3.7 by Nathaniel J. Smith in > `bpo-29137 `_. > > Module type > C extension + CAPI > Deprecated in > 3.7 > Removed in > 3.7 > Substitute > **none** > > > Modules to keep > =============== > > Some modules were originally proposed for deprecation. > > lib2to3 > ------- > > The `lib2to3 `_ package > provides > the ``2to3`` command to transpile Python 2 code to Python 3 code. > > The package is useful for other tasks besides porting code from Python 2 to > 3. For example `black`_ uses it for code reformatting. > > Module type > pure Python > > getopt > ------ > > The `getopt `_ module > mimics > C's getopt() option parser. Although users are encouraged to use argparse > instead, the getopt module is still widely used. > > Module type > pure Python > > optparse > -------- > > The `optparse `_ module > is > the predecessor of the argparse module. Although it has been deprecated > for > many years, it's still widely used. > > Module type > pure Python > Deprecated in > 3.2 > Substitute > argparse > > wave > ~~~~ > > The `wave `_ module > provides > support for the WAV sound format. The module uses one simple function > from the `audioop`_ module to perform byte swapping between little and > big > endian formats. Before 24 bit WAV support was added, byte swap used to be > implemented with the ``array`` module. To remove ``wave``'s dependency on > the > ``audioop``, the byte swap function could be either be moved to another > module (e.g. ``operator``) or the ``array`` module could gain support for > 24 bit (3 byte) arrays. > > Module type > pure Python (depends on *byteswap* from `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > > Future maintenance of removed modules > ===================================== > > The main goal of the PEP is to reduce the burden and workload on the > Python > core developer team. Therefore removed modules will not be maintained by > the core team as separate PyPI packages. However the removed code, tests > and > documentation may be moved into a new git repository, so community > members > have a place from which they can pick up and fork code. > > A first draft of a `legacylib `_ > repository is available on my private Github account. > > It's my hope that some of the deprecated modules will be picked up and > adopted by users that actually care about them. For example ``colorsys`` and > ``imghdr`` are useful modules, but have limited feature set. A fork of > ``imghdr`` can add new features and support for more image formats, > without > being constrained by Python's release cycle. > > Most of the modules are in pure Python and can be easily packaged. Some > depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ > does not depend on any external libraries, it can be shipped in as binary > wheels with some effort. Other C modules can be replaced with ctypes or > cffi. > For example I created `legacycrypt `_ > with ``_crypt`` extension reimplemented with a few lines of ctypes code. > > > Discussions > =========== > > * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. > * Berker Peksag proposed to deprecate and removed *msilib*. > * Brett Cannon recommended to delay active deprecation warnings and > removal > of modules like *imp* until Python 3.10. Version 3.8 will be released > shortly before Python 2 reaches end of lifetime. A delay reduced churn for > users that migrate from Python 2 to 3.8. > * Brett also came up with the idea to keep lib2to3. The package is useful > for other purposes, e.g. `black `_ uses > it to reformat Python code. > * At one point, distutils was mentioned in the same sentence as this PEP. > To avoid lengthy discussion and delay of the PEP, I decided against dealing > with distutils. Deprecation of the distutils package will be handled by > another PEP. > * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) > convinced me to keep the `wave`_ module. [4]_ > * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ > > > References > ========== > > .. [1] > https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_fre > e > .. [2] https://man.openbsd.org/ossaudio > .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of- > windows-installer-msi-in-the-light-of-windows-10-and-the-universal- > windows-platform/ > .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 > .. [5] https://twitter.com/dabeaz/status/1130278844479545351 > > > Copyright > ========= > > This document has been placed in the public domain. > > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > list%40sdamon.com From solipsis at pitrou.net Thu May 23 04:36:29 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 May 2019 10:36:29 +0200 Subject: [Python-Dev] Python in next Windows 10 update References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> <1adf8a66-d325-5f5d-9fca-5f50d232243b@mail.mipt.ru> Message-ID: <20190523103629.5e1dc8de@fsol> On Thu, 23 May 2019 00:23:39 +0200 Ray Donnelly wrote: > On Thu, May 23, 2019, 12:17 AM Ivan Pozdeev via Python-Dev < > python-dev at python.org> wrote: > > > On 22.05.2019 23:52, Steve Dower wrote: > > > On 22May2019 1309, Ivan Pozdeev via Python-Dev wrote: > > >> As someone whose job is to diagnose and fix problems with running > > software: > > >> Are there patches in your release? Do you provide corresponding sources > > and debug symbols for it? > > > > > > You can find the sources at https://github.com/python/cpython :) > > > > > For Anaconda, this is not so, they apply private patches. So I had to make > > sure. > > > > There's nothing 'private' about them. > https://github.com/AnacondaRecipes/python-feedstock/tree/master/recipe Note that some of those patches could probably be contributed back. Regards Antoine. From songofacandy at gmail.com Thu May 23 05:49:23 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Thu, 23 May 2019 18:49:23 +0900 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: I have only 32GB mem, but AWS has larger memory machine! Linux perf shows here is bottleneck: https://github.com/python/cpython/blob/master/Objects/obmalloc.c#L1784-L1819 obmalloc sorts arenas by number of free pools. If there are many arenas, and memory block is freed by random order, this sorting become O(N^2). That's too bad. I'm trying address order instead. Regards, -- Inada Naoki From j.orponen at 4teamwork.ch Thu May 23 08:35:21 2019 From: j.orponen at 4teamwork.ch (Joni Orponen) Date: Thu, 23 May 2019 14:35:21 +0200 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: On Thu, May 23, 2019 at 11:52 AM Inada Naoki wrote: > I have only 32GB mem, but AWS has larger memory machine! > Be aware that on larger cloud vendor instances, the backing vCPUs and memory you get allocated might or might not be spread opaquely across multiple sockets and/or NUMA nodes of the backing hardware. Some of them have options where you can allocate raw hardware as well so you can opt to lock your process to run within just one NUMA node and ensure hardware locality for your performance debugging. I'm pointing this out in case you experience any mystery jitters in your benchmark results. -- Joni Orponen -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Thu May 23 08:42:38 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Thu, 23 May 2019 21:42:38 +0900 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: 1. perf shows 95% of CPU time is eaten by _PyObject_Free, not kernel space. 2. This loop is cleary hot: https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 I can attach the process by gdb and I confirmed many arenas have same nfreepools. I believe this is not jitter caused from NUMA or something else in cloud. -- Inada Naoki From ajm at flonidan.dk Thu May 23 09:22:58 2019 From: ajm at flonidan.dk (Anders Munch) Date: Thu, 23 May 2019 13:22:58 +0000 Subject: [Python-Dev] PEP 594: update 1 Message-ID: Fra: Python-Dev [mailto:python-dev-bounces+ajm=flonidan.dk at python.org] P? vegne af Terry Reedy: >> Deprecation schedule >> ==================== > > I think it worth adding that some modules were deprecated years ago but kept until after 2.7 end-of-life to ease 3.x transition. Removing modules is not a sudden new idea. > > Also, the proposed schedule (with one exception) is a slower alternative than proposals to add a deprecation warning (if not one already) in 3.8 and remove in 3.9. About that schedule: I think the removal version should be called 4.0. These are working modules that to varying degrees are still in use that are being removed. It's perfectly legitimate for python-dev to decide to stop maintaining them, but it's a breaking change none the less. For me, it's a couple of CGI scripts that will break. If I hadn't read this thread, I would at some point be confronted with a ModuleNotFoundError exception, with no guidance on how to fix that. That's not a very good user experience, even if a fix can be found after some googling. A major version change serves as a heads up that something is going on and you need to check the consequences before upgrading. regards, Anders From p.f.moore at gmail.com Thu May 23 09:33:36 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 23 May 2019 14:33:36 +0100 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: Message-ID: On Thu, 23 May 2019 at 14:25, Anders Munch wrote: > A major version change serves as a heads up that something is going on and you need to check the consequences before upgrading. Python's backward compatibility policy allows breaking changes between versions X.Y and X.Y+1 (with a suitable deprecation period). This proposal is no different. Paul From ncoghlan at gmail.com Thu May 23 09:36:22 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 23 May 2019 23:36:22 +1000 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: On Wed, 22 May 2019 at 00:51, Nick Coghlan wrote: > P.S. I'm away this weekend, so I expect the reference implementation > to be done late next week, and I'd be submitting the PEP to Nathaniel > for formal pronouncement at that point. However, I'm posting this > thread now so that there's more time for discussion prior to the 3.8b1 > deadline. I found some time the other day to finish up the core of the reference implementation and resolve some bugs in the new test cases, so the essential changes can now be seen in https://github.com/python/cpython/pull/3640/files There are still some code level TODO items to cover off a few minor points in the PEP, as well as to make the fast locals proxy a better behaved mutable mapping. The proposed documentation changes also still need to be brought over from the PEP, and at least a quick scan done through the code comments and the rest of the documentation for now outdated references to the legacy trace function namespace handling. However, I think the PR does show that the proposed technique can be implemented without too much additional code complexity, and will hopefully be adaptable for all implementations that emulate the frame API at all. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve.dower at python.org Thu May 23 11:15:41 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 23 May 2019 08:15:41 -0700 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> On 23May2019 0542, Inada Naoki wrote: > 1. perf shows 95% of CPU time is eaten by _PyObject_Free, not kernel space. > 2. This loop is cleary hot: > https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 > > I can attach the process by gdb and I confirmed many arenas have > same nfreepools. It's relatively easy to test replacing our custom allocators with the system ones, yes? Can we try those to see whether they have the same characteristic? Given the relative amount of investment over the last 19 years [1], I wouldn't be surprised if most system ones are at least as good for our needs now. Certainly Windows HeapAlloc has had serious improvements in that time to help with fragmentation and small allocations. Cheers, Steve [1]: https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L769 From steve.dower at python.org Thu May 23 11:26:56 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 23 May 2019 08:26:56 -0700 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: On 23May2019 0636, Nick Coghlan wrote: > However, I think the PR does show that the proposed technique can be > implemented without too much additional code complexity, and will > hopefully be adaptable for all implementations that emulate the frame > API at all. Much excitement! One of the things I like best about this change is that it actually makes it *easier* for alternative implementations to use a simpler frame object without having to emulate CPython semantics (I'd love to get to a place where bug-for-bug compatibility wasn't required, but this is where we are right now so *shrug*). Cheers, Steve From songofacandy at gmail.com Thu May 23 11:59:08 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Fri, 24 May 2019 00:59:08 +0900 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> Message-ID: > > It's relatively easy to test replacing our custom allocators with the > system ones, yes? Can we try those to see whether they have the same > characteristic? > Yes. PYTHONMALLOC=malloc LD_PRELOAD=/path/to/jemalloc.so python script.py I will try it tomorrow. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeff at kintscher.cc Thu May 23 04:41:16 2019 From: jeff at kintscher.cc (Jeff Kintscher) Date: Thu, 23 May 2019 01:41:16 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <016601d5113d$87bd8740$973895c0$@sdamon.com> References: <016601d5113d$87bd8740$973895c0$@sdamon.com> Message-ID: +1 On 5/23/19 1:00 AM, Alex Walters wrote: > I've watched the entire thread and its taken me a few days to put a finger > on what bothers me about it. > > In my opinion, this shouldn't be a pep describing the list of modules that > need to go as "dead batteries", but should be a process pep describing how > dead batteries should be removed, and the individual modules should be given > their own pep. I think reactions to individual module peps will give a > better indication of if it's a used module or not. > >> -----Original Message----- >> From: Python-Dev > list=sdamon.com at python.org> On Behalf Of Christian Heimes >> Sent: Monday, May 20, 2019 4:15 PM >> To: Python Dev >> Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard >> library >> >> Hi, >> >> here is the first version of my PEP 594 to deprecate and eventually remove >> modules from the standard library. The PEP started last year with talk > during >> the Python Language Summit 2018, https://lwn.net/Articles/755229/. >> >> The PEP can be confirmed in two stages. I'm not planning any code changes >> for 3.8. Instead I only like to document a bunch of modules as deprecated. >> Active deprecation is planned for 3.9 and removal for 3.10. The long >> deprecation phase gives us 3 years to change our minds or handle edge >> cases, too. >> >> Regards, >> Christian >> >> >> > ---------------------------------------------------------------------------- >> PEP: 594 >> Title: Removing dead batteries from the standard library >> Author: Christian Heimes >> Status: Active >> Type: Process >> Content-Type: text/x-rst >> Created: 20-May-2019 >> Post-History: >> >> >> Abstract >> ======== >> >> This PEP proposed a list of standard library modules to be removed from > the >> standard library. The modules are mostly historic data formats and APIs > that >> have been superseded a long time ago, e.g. Mac OS 9 and Commodore. >> >> >> Rationale >> ========= >> >> Back in the early days of Python, the interpreter came with a large set of >> useful modules. This was often refrained to as "batteries included" >> philosophy and was one of the corner stones to Python's success story. >> Users didn't have to figure out how to download and install separate >> packages in order to write a simple web server or parse email. >> >> Times have changed. The introduction of the cheese shop (PyPI), > setuptools, >> and later pip, it became simple and straight forward to download and > install >> packages. Nowadays Python has a rich and vibrant ecosystem of third party >> packages. It's pretty much standard to either install packages from PyPI > or >> use one of the many Python or Linux distributions. >> >> On the other hand, Python's standard library is piling up cruft, > unnecessary >> duplication of functionality, and dispensable features. This is > undesirable >> for several reasons. >> >> * Any additional module increases the maintenance cost for the Python core >> development team. The team has limited resources, reduced maintenance >> cost >> frees development time for other improvements. >> * Modules in the standard library are generally favored and seen as the >> de-facto solution for a problem. A majority of users only pick 3rd party >> modules to replace a stdlib module, when they have a compelling reason, >> e.g. >> lxml instead of `xml`. The removal of an unmaintained stdlib module >> increases the chances of a community contributed module to become >> widely >> used. >> * A lean and mean standard library benefits platforms with limited > resources >> like devices with just a few hundred kilobyte of storage (e.g. BBC >> Micro:bit). Python on mobile platforms like BeeWare or WebAssembly >> (e.g. pyodide) also benefit from reduced download size. >> >> The modules in the PEP have been selected for deprecation because their >> removal is either least controversial or most beneficial. For example >> least controversial are 30 years old multimedia formats like ``sunau`` >> audio format, which was used on SPARC and NeXT workstations in the late >> 1980ties. The ``crypt`` module has fundamental flaws that are better > solved >> outside the standard library. >> >> This PEP also designates some modules as not scheduled for removal. Some >> modules have been deprecated for several releases or seem unnecessary at >> first glance. However it is beneficial to keep the modules in the standard >> library, mostly for environments where installing a package from PyPI is > not >> an option. This can be cooperate environments or class rooms where >> external >> code is not permitted without legal approval. >> >> * The usage of FTP is declining, but some files are still provided over >> the FTP protocol or hosters offer FTP to upload content. Therefore >> ``ftplib`` is going to stay. >> * The ``optparse`` and ``getopt`` module are widely used. They are mature >> modules with very low maintenance overhead. >> * According to David Beazley [5]_ the ``wave`` module is easy to teach to >> kids and can make crazy sounds. Making a computer generate crazy sounds >> is >> powerful and highly motivating exercise for a 9yo aspiring developer. > It's >> a fun battery to keep. >> >> >> Deprecation schedule >> ==================== >> >> 3.8 >> --- >> >> This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be > released >> a few months before Python 2.7 will reach its end of lifetime. We expect > that >> Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and >> 2020. To reduce churn and to allow a smooth transition from Python 2, >> Python 3.8 will neither raise `DeprecationWarning` nor remove any >> modules that have been scheduled for removal. Instead deprecated >> modules will >> just be *documented* as deprecated. Optionally modules may emit a >> `PendingDeprecationWarning`. >> >> All deprecated modules will also undergo a feature freeze. No additional >> features should be added. Bug should still be fixed. >> >> 3.9 >> --- >> >> Starting with Python 3.9, deprecated modules will start issuing >> `DeprecationWarning`. >> >> >> 3.10 >> ---- >> >> In 3.10 all deprecated modules will be removed from the CPython repository >> together with tests, documentation, and autoconf rules. >> >> >> PEP acceptance process >> ====================== >> >> 3.8.0b1 is scheduled to be release shortly after the PEP is officially >> submitted. Since it's improbable that the PEP will pass all stages of the >> PEP process in time, I propose a two step acceptance process that is >> analogous Python's two release deprecation process. >> >> The first *provisionally accepted* phase targets Python 3.8.0b1. In the > first >> phase no code is changes or removed. Modules are only documented as >> deprecated. >> >> The final decision, which modules will be removed and how the removed >> code >> is preserved, can be delayed for another year. >> >> >> Deprecated modules >> ================== >> >> The modules are grouped as data encoding, multimedia, network, OS >> interface, >> and misc modules. The majority of modules are for old data formats or >> old APIs. Some others are rarely useful and have better replacements on >> PyPI, e.g. Pillow for image processing or NumPy-based projects to deal > with >> audio processing. >> >> .. csv-table:: Table 1: Proposed modules deprecations >> :header: "Module", "Deprecated in", "To be removed", "Replacement" >> >> aifc,3.8,3.10,\- >> asynchat,3.8,3.10,asyncio >> asyncore,3.8,3.10,asyncio >> audioop,3.8,3.10,\- >> binhex,3.8,3.10,\- >> cgi,3.8,3.10,\- >> cgitb,3.8,3.10,\- >> chunk,3.8,3.10,\- >> colorsys,**3.8?**,**3.10?**,\- >> crypt,3.8,3.10,\- >> fileinput,3.8,3.10,argparse >> formatter,3.4,3.10,\- >> fpectl,**3.7**,**3.7**,\- >> getopt,**3.2**,**keep**,"argparse, optparse" >> imghdr,3.8,3.10,\- >> imp,**3.4**,3.10,importlib >> lib2to3,\-,**keep**, >> macpath,**3.7**,**3.8**,\- >> msilib,3.8,3.10,\- >> nntplib,3.8,3.10,\- >> nis,3.8,3.10,\- >> optparse,\-,**keep**,argparse >> ossaudiodev,3.8,3.10,\- >> pipes,3.8,3.10,subprocess >> smtpd,**3.7**,3.10,aiosmtpd >> sndhdr,3.8,3.10,\- >> spwd,3.8,3.10,\- >> sunau,3.8,3.10,\- >> uu,3.8,3.10,\- >> wave,\-,**keep**, >> xdrlib,3.8,3.10,\- >> >> >> Data encoding modules >> --------------------- >> >> binhex >> ~~~~~~ >> >> The `binhex `_ module >> encodes >> and decodes Apple Macintosh binhex4 data. It was originally developed for >> TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to >> encode binary email attachments. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> uu >> ~~ >> >> The `uu `_ module provides >> uuencode format, an old binary encoding format for email from 1980. The uu >> format has been replaced by MIME. The uu codec is provided by the binascii >> module. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> xdrlib >> ~~~~~~ >> >> The `xdrlib `_ module >> supports >> the Sun External Data Representation Standard. XDR is an old binary >> serialization format from 1987. These days it's rarely used outside >> specialized domains like NFS. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Multimedia modules >> ------------------ >> >> aifc >> ~~~~ >> >> The `aifc `_ module provides >> support for reading and writing AIFF and AIFF-C files. The Audio > Interchange >> File Format is an old audio format from 1988 based on Amiga IFF. It was > most >> commonly used on the Apple Macintosh. These days only few specialized >> application use AIFF. >> >> Module type >> pure Python (depends on `audioop`_ C extension) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> audioop >> ~~~~~~~ >> >> The `audioop `_ module >> contains helper functions to manipulate raw audio data and adaptive >> differential pulse-code modulated audio data. The module is implemented in >> C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ >> module depend on `audioop`_ for some operations. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> colorsys >> ~~~~~~~~ >> >> The `colorsys `_ module >> defines color conversion functions between RGB, YIQ, HSL, and HSV >> coordinate >> systems. The Pillow library provides much faster conversation between >> color systems. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `Pillow `_, >> `colorspacious `_ >> >> chunk >> ~~~~~ >> >> The `chunk `_ module >> provides >> support for reading and writing Electronic Arts' Interchange File Format. >> IFF is an old audio file format originally introduced for Commodore and >> Amiga. The format is no longer relevant. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> imghdr >> ~~~~~~ >> >> The `imghdr `_ module is a >> simple tool to guess the image file format from the first 32 bytes >> of a file or buffer. It supports only a limited amount of formats and >> neither returns resolution nor color depth. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> ossaudiodev >> ~~~~~~~~~~~ >> >> The `ossaudiodev `_ >> module provides support for Open Sound System, an interface to sound >> playback and capture devices. OSS was initially free software, but later >> support for newer sound devices and improvements were proprietary. Linux >> community abandoned OSS in favor of ALSA [1]_. Some operation systems >> like >> OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> sndhdr >> ~~~~~~ >> >> The `sndhdr `_ module is >> similar to the `imghdr`_ module but for audio formats. It guesses file >> format, channels, frame rate, and sample widths from the first 512 bytes > of >> a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and >> other ancient formats. >> >> Module type >> pure Python (depends on `audioop`_ C extension for some operations) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> sunau >> ~~~~~ >> >> The `sunau `_ module >> provides >> support for Sun AU sound format. It's yet another old, obsolete file > format. >> Module type >> pure Python (depends on `audioop`_ C extension for some operations) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Networking modules >> ------------------ >> >> asynchat >> ~~~~~~~~ >> >> The `asynchat `_ module >> is build on top of `asyncore`_ and has been deprecated since Python 3.6. >> >> Module type >> pure Python >> Deprecated in >> 3.6 >> Removed in >> 3.10 >> Substitute >> asyncio >> >> asyncore >> ~~~~~~~~ >> >> The `asyncore `_ module >> was >> the first module for asynchronous socket service clients and servers. It >> has been replaced by asyncio and is deprecated since Python 3.6. >> >> The ``asyncore`` module is also used in stdlib tests. The tests for >> ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly >> based on ``asyncore``. These tests must be updated to use asyncio or >> threading. >> >> Module type >> pure Python >> Deprecated in >> 3.6 >> Removed in >> 3.10 >> Substitute >> asyncio >> >> >> cgi >> ~~~ >> >> The `cgi `_ module is a > support >> module for Common Gateway Interface (CGI) scripts. CGI is deemed as >> inefficient because every incoming request is handled in a new process. > PEP >> 206 considers the module as *designed poorly and are now near-impossible >> to fix*. >> >> Several people proposed to either keep the cgi module for features like >> `cgi.parse_qs()` or move `cgi.escape()` to a different module. The >> functions `cgi.parse_qs` and `cgi.parse_qsl` have been >> deprecated for a while and are actually aliases for >> `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The >> function `cgi.quote` has been deprecated in favor of `html.quote` >> with secure default values. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> cgitb >> ~~~~~ >> >> The `cgitb `_ module is a >> helper for the cgi module for configurable tracebacks. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> smtpd >> ~~~~~ >> >> The `smtpd `_ module >> provides >> a simple implementation of a SMTP mail server. The module documentation >> recommends ``aiosmtpd``. >> >> Module type >> pure Python >> Deprecated in >> **3.7** >> To be removed in >> 3.10 >> Substitute >> aiosmtpd >> >> nntplib >> ~~~~~~~ >> >> The `nntplib `_ module >> implements the client side of the Network News Transfer Protocol (nntp). >> News >> groups used to be a dominant platform for online discussions. Over the > last >> two decades, news has been slowly but steadily replaced with mailing lists >> and web-based discussion platforms. >> >> The ``nntplib`` tests have been the cause of additional work in the recent >> past. Python only contains client side of NNTP. The test cases depend on >> external news server. These servers were unstable in the past. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Operating system interface >> -------------------------- >> >> crypt >> ~~~~~ >> >> The `crypt `_ module >> implements >> password hashing based on ``crypt(3)`` function from ``libcrypt`` or >> ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of > poor >> quality and insecure. Users are discouraged to use them. >> >> * The module is not available on Windows. Cross-platform application need >> an alternative implementation any way. >> * Only DES encryption is guarenteed to be available. DES has an extremely >> limited key space of 2**56. >> * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. >> SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only >> algorithm that is still secure. However it's in glibc and therefore not >> commonly available on Linux. >> * Depending on the platform, the ``crypt`` module is not thread safe. Only >> implementations with ``crypt_r(3)`` are thread safe. >> >> Module type >> C extension + Python module >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `bcrypt `_, >> `passlib `_, >> `argon2cffi `_, >> hashlib module (PBKDF2, scrypt) >> >> macpath >> ~~~~~~~ >> >> The `macpath `_ module >> provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no > longer >> supported >> >> Module type >> pure Python >> Deprecated in >> 3.7 >> Removed in >> 3.8 >> Substitute >> **none** >> >> nis >> ~~~ >> >> The `nis `_ module provides >> NIS/YP support. Network Information Service / Yellow Pages is an old and >> deprecated directory service protocol developed by Sun Microsystems. It's >> designed successor NIS+ from 1992 never took off. For a long time, libc's >> Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more >> powerful >> and more secure replacement of NIS. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> spwd >> ~~~~ >> >> The `spwd `_ module >> provides >> direct access to Unix shadow password database using non-standard APIs. >> In general it's a bad idea to use the spwd. The spwd circumvents system >> security policies, it does not use the PAM stack, and is >> only compatible with local user accounts. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> Misc modules >> ------------ >> >> fileinput >> ~~~~~~~~~ >> >> The `fileinput `_ module >> implements a helpers to iterate over a list of files from ``sys.argv``. > The >> module predates the optparser and argparser module. The same >> functionality >> can be implemented with the argparser module. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> argparse >> >> formatter >> ~~~~~~~~~ >> >> The `formatter `_ >> module >> is an old text formatting module which has been deprecated since Python >> 3.4. >> >> Module type >> pure Python >> Deprecated in >> 3.4 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> imp >> ~~~ >> >> The `imp `_ module is the >> predecessor of the >> `importlib `_ module. >> Most >> functions have been deprecated since Python 3.3 and the module since >> Python 3.4. >> >> Module type >> C extension >> Deprecated in >> 3.4 >> To be removed in >> 3.10 >> Substitute >> importlib >> >> msilib >> ~~~~~~ >> >> The `msilib `_ package is a >> Windows-only package. It supports the creation of Microsoft Installers > (MSI). >> The package also exposes additional APIs to create cabinet files (CAB). > The >> module is used to facilitate distutils to create MSI installers with >> ``bdist_msi`` command. In the past it was used to create CPython's > official >> Windows installer, too. >> >> Microsoft is slowly moving away from MSI in favor of Windows 10 Apps >> (AppX) >> as new deployment model [3]_. >> >> Module type >> C extension + Python code >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> pipes >> ~~~~~ >> >> The `pipes `_ module >> provides >> helpers to pipe the input of one command into the output of another >> command. >> The module is built on top of ``os.popen``. Users are encouraged to use >> the subprocess module instead. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> subprocess module >> >> Removed modules >> =============== >> >> fpectl >> ------ >> >> The `fpectl `_ module was >> never built by default, its usage was discouraged and considered > dangerous. >> It also required a configure flag that caused an ABI incompatibility. The >> module was removed in 3.7 by Nathaniel J. Smith in >> `bpo-29137 `_. >> >> Module type >> C extension + CAPI >> Deprecated in >> 3.7 >> Removed in >> 3.7 >> Substitute >> **none** >> >> >> Modules to keep >> =============== >> >> Some modules were originally proposed for deprecation. >> >> lib2to3 >> ------- >> >> The `lib2to3 `_ package >> provides >> the ``2to3`` command to transpile Python 2 code to Python 3 code. >> >> The package is useful for other tasks besides porting code from Python 2 > to >> 3. For example `black`_ uses it for code reformatting. >> >> Module type >> pure Python >> >> getopt >> ------ >> >> The `getopt `_ module >> mimics >> C's getopt() option parser. Although users are encouraged to use argparse >> instead, the getopt module is still widely used. >> >> Module type >> pure Python >> >> optparse >> -------- >> >> The `optparse `_ module >> is >> the predecessor of the argparse module. Although it has been deprecated >> for >> many years, it's still widely used. >> >> Module type >> pure Python >> Deprecated in >> 3.2 >> Substitute >> argparse >> >> wave >> ~~~~ >> >> The `wave `_ module >> provides >> support for the WAV sound format. The module uses one simple function >> from the `audioop`_ module to perform byte swapping between little and >> big >> endian formats. Before 24 bit WAV support was added, byte swap used to be >> implemented with the ``array`` module. To remove ``wave``'s dependency on >> the >> ``audioop``, the byte swap function could be either be moved to another >> module (e.g. ``operator``) or the ``array`` module could gain support for >> 24 bit (3 byte) arrays. >> >> Module type >> pure Python (depends on *byteswap* from `audioop`_ C extension) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> >> Future maintenance of removed modules >> ===================================== >> >> The main goal of the PEP is to reduce the burden and workload on the >> Python >> core developer team. Therefore removed modules will not be maintained by >> the core team as separate PyPI packages. However the removed code, tests >> and >> documentation may be moved into a new git repository, so community >> members >> have a place from which they can pick up and fork code. >> >> A first draft of a `legacylib `_ >> repository is available on my private Github account. >> >> It's my hope that some of the deprecated modules will be picked up and >> adopted by users that actually care about them. For example ``colorsys`` > and >> ``imghdr`` are useful modules, but have limited feature set. A fork of >> ``imghdr`` can add new features and support for more image formats, >> without >> being constrained by Python's release cycle. >> >> Most of the modules are in pure Python and can be easily packaged. Some >> depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since > `audioop`_ >> does not depend on any external libraries, it can be shipped in as binary >> wheels with some effort. Other C modules can be replaced with ctypes or >> cffi. >> For example I created `legacycrypt > `_ >> with ``_crypt`` extension reimplemented with a few lines of ctypes code. >> >> >> Discussions >> =========== >> >> * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. >> * Berker Peksag proposed to deprecate and removed *msilib*. >> * Brett Cannon recommended to delay active deprecation warnings and >> removal >> of modules like *imp* until Python 3.10. Version 3.8 will be released >> shortly before Python 2 reaches end of lifetime. A delay reduced churn > for >> users that migrate from Python 2 to 3.8. >> * Brett also came up with the idea to keep lib2to3. The package is useful >> for other purposes, e.g. `black `_ uses >> it to reformat Python code. >> * At one point, distutils was mentioned in the same sentence as this PEP. >> To avoid lengthy discussion and delay of the PEP, I decided against > dealing >> with distutils. Deprecation of the distutils package will be handled by >> another PEP. >> * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) >> convinced me to keep the `wave`_ module. [4]_ >> * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ >> >> >> References >> ========== >> >> .. [1] >> https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_fre >> e >> .. [2] https://man.openbsd.org/ossaudio >> .. [3] https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of- >> windows-installer-msi-in-the-light-of-windows-10-and-the-universal- >> windows-platform/ >> .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 >> .. [5] https://twitter.com/dabeaz/status/1130278844479545351 >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- >> list%40sdamon.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/websurfer%40surf2c.net -- ----------------------------------------- From there to here, from here to there, funny things are everywhere. -- Theodore Geisel -------------- next part -------------- An HTML attachment was scrubbed... URL: From ajm at flonidan.dk Thu May 23 12:47:58 2019 From: ajm at flonidan.dk (Anders Munch) Date: Thu, 23 May 2019 16:47:58 +0000 Subject: [Python-Dev] PEP 594: update 1 Message-ID: Fra: Paul Moore [mailto:p.f.moore at gmail.com]: > > A major version change serves as a heads up that something is going on and you need to check the consequences before upgrading. > Python's backward compatibility policy allows breaking changes between versions X.Y and X.Y+1 (with a suitable deprecation period). This proposal is no different. Except perhaps in scale. The same people that upgrade from 3.x to 3.x+1 without giving it a second thought, just to be on the latest version, will hesitate to go from 3.x to 4.y, because the major version change is a hint that they should be more careful. That means they're ready for it when they get the ModuleNotFoundError exception, instead of feeling ambushed. OK, it may be that this is not enough to warrant a 4.0 release, but I do think python-dev should get over its fear of major versions sometime. And that transitioning to a leaner standard library with some things moved to PyPI would not be a bad program statement for a Python 4.0. regards, Anders From solipsis at pitrou.net Thu May 23 12:33:31 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 23 May 2019 18:33:31 +0200 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> Message-ID: <20190523183331.2163dd60@fsol> On Fri, 24 May 2019 00:59:08 +0900 Inada Naoki wrote: > > > > It's relatively easy to test replacing our custom allocators with the > > system ones, yes? Can we try those to see whether they have the same > > characteristic? > > > > Yes. > > PYTHONMALLOC=malloc LD_PRELOAD=/path/to/jemalloc.so python script.py > > I will try it tomorrow. Can you simply test with the system allocator rather than jemalloc? Regards Antoine. From steve.dower at python.org Thu May 23 12:57:57 2019 From: steve.dower at python.org (Steve Dower) Date: Thu, 23 May 2019 09:57:57 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: Message-ID: On 23May2019 0947, Anders Munch wrote: > Fra: Paul Moore [mailto:p.f.moore at gmail.com]: >>> A major version change serves as a heads up that something is going on and you need to check the consequences before upgrading. >> Python's backward compatibility policy allows breaking changes between versions X.Y and X.Y+1 (with a suitable deprecation period). This proposal is no different. > > Except perhaps in scale. The same people that upgrade from 3.x to 3.x+1 without giving it a second thought, just to be on the latest version, will hesitate to go from 3.x to 4.y, because the major version change is a hint that they should be more careful. That means they're ready for it when they get the ModuleNotFoundError exception, instead of feeling ambushed. > > OK, it may be that this is not enough to warrant a 4.0 release, but I do think python-dev should get over its fear of major versions sometime. And that transitioning to a leaner standard library with some things moved to PyPI would not be a bad program statement for a Python 4.0. We need to make it more clear somehow that Python uses series.major.minor.micro versioning, not SemVer. I see this confusion happen all the time. Think of it as Python3 7.x moving to Python3 8.x, and evaluate how much effort you should put into migration :) That said, I'm totally in favour of Python 4.0 being the point where we change expectations about what will be in the standard library (and also C API, language/implementation semantics, etc.). But until we have a clear vision statement that everyone (or enough) is behind, there's nothing gained by prematurely causing that much disruption. Cheers, Steve From tim.peters at gmail.com Thu May 23 13:13:11 2019 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 23 May 2019 12:13:11 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: [Inada Naoki ] > ... > 2. This loop is cleary hot: > https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 Which is 3 lines of code plus a closing brace. The OP didn't build their own Python, and the source from which it was compiled wasn't available to them. But they did say that when they got into gdb and single-stepped, it was looping through the same three lines of code, in obmalloc.c, over & over. Which is 100% consistent with the loop you identified as being "the hot spot". From barry at python.org Thu May 23 14:02:01 2019 From: barry at python.org (Barry Warsaw) Date: Thu, 23 May 2019 11:02:01 -0700 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: On May 20, 2019, at 13:15, Christian Heimes wrote: > > here is the first version of my PEP 594 to deprecate and eventually remove modules from the standard library. The PEP started last year with talk during the Python Language Summit 2018, https://lwn.net/Articles/755229/. > > The PEP can be confirmed in two stages. I'm not planning any code changes for 3.8. Instead I only like to document a bunch of modules as deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The long deprecation phase gives us 3 years to change our minds or handle edge cases, too. Hi Christian, First, I appreciate the work you?ve done on this, and the general sentiment. But I do feel like this is a partial solution to a problem the Python distribution has had for a very long time. Maybe it?s a good step forward, but in a sense it is incomplete and only chips away at the problem. We have two competing pressures, one to provide a rich standard library with lots of useful features that come right out of the box. Let?s not underestimate the value that this has for our users, and the contribution such a stdlib has made to making Python as popular as it is. But it?s also true that lots of the stdlib don?t get the love they need to stay relevant, and a curated path to keeping only the most useful and modern libraries. I wonder how much the long development cycle and relatively big overhead for contributing to stdlib maintenance causes a big part of our headaches with the stdlib. Current stdlib development processes also incur burden for alternative implementations. We?ve had many ideas over the years, such as stripping the CPython repo stdlib to its bare minimum and providing some way of *distributing* a sumo tarball. But none have made it far enough to be adopted. I don?t have any new bright ideas for how to make this work, but I think finding a holistic approach to these competing pressures is in the best long term interest of Python. That all said, if PEP 594 can be viewed as part of the bigger picture, then maybe it?s a good idea. Perhaps the approaches for maintaining such deprecated libraries can be used as an experiment for finding a more modern, useful, and vibrant approach to stdlib maintenance. Cheers, -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From greg at krypto.org Thu May 23 14:49:43 2019 From: greg at krypto.org (Gregory P. Smith) Date: Thu, 23 May 2019 11:49:43 -0700 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: I suggest filing a bug to track this... On Thu, May 23, 2019 at 10:13 AM Tim Peters wrote: > [Inada Naoki ] > > ... > > 2. This loop is cleary hot: > > > https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 > > Which is 3 lines of code plus a closing brace. The OP didn't build > their own Python, and the source from which it was compiled wasn't > available to them. But they did say that when they got into gdb and > single-stepped, it was looping through the same three lines of code, > in obmalloc.c, over & over. Which is 100% consistent with the loop > you identified as being "the hot spot". > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Thu May 23 15:25:13 2019 From: steve at holdenweb.com (Steve Holden) Date: Thu, 23 May 2019 20:25:13 +0100 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: Message-ID: Besides which, it would be lovely to have a major release that didn't involve any pain at all for the majority of users! Our erstwhile BDFL always eschewed two-digit version identifiers- due to the possibility for confusion about collating sequence, I beleive.. We should honour his preferences by going from 3.9 to 4.0. On Thu, May 23, 2019 at 5:49 PM Anders Munch wrote: > Fra: Paul Moore [mailto:p.f.moore at gmail.com]: > > > A major version change serves as a heads up that something is going on > and you need to check the consequences before upgrading. > > Python's backward compatibility policy allows breaking changes between > versions X.Y and X.Y+1 (with a suitable deprecation period). This proposal > is no different. > > Except perhaps in scale. The same people that upgrade from 3.x to 3.x+1 > without giving it a second thought, just to be on the latest version, will > hesitate to go from 3.x to 4.y, because the major version change is a hint > that they should be more careful. That means they're ready for it when > they get the ModuleNotFoundError exception, instead of feeling ambushed. > > OK, it may be that this is not enough to warrant a 4.0 release, but I do > think python-dev should get over its fear of major versions sometime. And > that transitioning to a leaner standard library with some things moved to > PyPI would not be a bad program statement for a Python 4.0. > > regards, Anders > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Thu May 23 15:28:54 2019 From: steve at holdenweb.com (Steve Holden) Date: Thu, 23 May 2019 20:28:54 +0100 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: <016601d5113d$87bd8740$973895c0$@sdamon.com> References: <016601d5113d$87bd8740$973895c0$@sdamon.com> Message-ID: It might also serve to identify those with an interest in maintaining the non-core packages, which might even be given some special status on PyPI. On Thu, May 23, 2019 at 9:01 AM Alex Walters wrote: > I've watched the entire thread and its taken me a few days to put a finger > on what bothers me about it. > > In my opinion, this shouldn't be a pep describing the list of modules that > need to go as "dead batteries", but should be a process pep describing how > dead batteries should be removed, and the individual modules should be > given > their own pep. I think reactions to individual module peps will give a > better indication of if it's a used module or not. > > > -----Original Message----- > > From: Python-Dev > list=sdamon.com at python.org> On Behalf Of Christian Heimes > > Sent: Monday, May 20, 2019 4:15 PM > > To: Python Dev > > Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard > > library > > > > Hi, > > > > here is the first version of my PEP 594 to deprecate and eventually > remove > > modules from the standard library. The PEP started last year with talk > during > > the Python Language Summit 2018, https://lwn.net/Articles/755229/. > > > > The PEP can be confirmed in two stages. I'm not planning any code changes > > for 3.8. Instead I only like to document a bunch of modules as > deprecated. > > Active deprecation is planned for 3.9 and removal for 3.10. The long > > deprecation phase gives us 3 years to change our minds or handle edge > > cases, too. > > > > Regards, > > Christian > > > > > > > > ---------------------------------------------------------------------------- > > PEP: 594 > > Title: Removing dead batteries from the standard library > > Author: Christian Heimes > > Status: Active > > Type: Process > > Content-Type: text/x-rst > > Created: 20-May-2019 > > Post-History: > > > > > > Abstract > > ======== > > > > This PEP proposed a list of standard library modules to be removed from > the > > standard library. The modules are mostly historic data formats and APIs > that > > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > > > > Rationale > > ========= > > > > Back in the early days of Python, the interpreter came with a large set > of > > useful modules. This was often refrained to as "batteries included" > > philosophy and was one of the corner stones to Python's success story. > > Users didn't have to figure out how to download and install separate > > packages in order to write a simple web server or parse email. > > > > Times have changed. The introduction of the cheese shop (PyPI), > setuptools, > > and later pip, it became simple and straight forward to download and > install > > packages. Nowadays Python has a rich and vibrant ecosystem of third party > > packages. It's pretty much standard to either install packages from PyPI > or > > use one of the many Python or Linux distributions. > > > > On the other hand, Python's standard library is piling up cruft, > unnecessary > > duplication of functionality, and dispensable features. This is > undesirable > > for several reasons. > > > > * Any additional module increases the maintenance cost for the Python > core > > development team. The team has limited resources, reduced maintenance > > cost > > frees development time for other improvements. > > * Modules in the standard library are generally favored and seen as the > > de-facto solution for a problem. A majority of users only pick 3rd > party > > modules to replace a stdlib module, when they have a compelling reason, > > e.g. > > lxml instead of `xml`. The removal of an unmaintained stdlib module > > increases the chances of a community contributed module to become > > widely > > used. > > * A lean and mean standard library benefits platforms with limited > resources > > like devices with just a few hundred kilobyte of storage (e.g. BBC > > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > > (e.g. pyodide) also benefit from reduced download size. > > > > The modules in the PEP have been selected for deprecation because their > > removal is either least controversial or most beneficial. For example > > least controversial are 30 years old multimedia formats like ``sunau`` > > audio format, which was used on SPARC and NeXT workstations in the late > > 1980ties. The ``crypt`` module has fundamental flaws that are better > solved > > outside the standard library. > > > > This PEP also designates some modules as not scheduled for removal. Some > > modules have been deprecated for several releases or seem unnecessary at > > first glance. However it is beneficial to keep the modules in the > standard > > library, mostly for environments where installing a package from PyPI is > not > > an option. This can be cooperate environments or class rooms where > > external > > code is not permitted without legal approval. > > > > * The usage of FTP is declining, but some files are still provided over > > the FTP protocol or hosters offer FTP to upload content. Therefore > > ``ftplib`` is going to stay. > > * The ``optparse`` and ``getopt`` module are widely used. They are mature > > modules with very low maintenance overhead. > > * According to David Beazley [5]_ the ``wave`` module is easy to teach to > > kids and can make crazy sounds. Making a computer generate crazy sounds > > is > > powerful and highly motivating exercise for a 9yo aspiring developer. > It's > > a fun battery to keep. > > > > > > Deprecation schedule > > ==================== > > > > 3.8 > > --- > > > > This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be > released > > a few months before Python 2.7 will reach its end of lifetime. We expect > that > > Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and > > 2020. To reduce churn and to allow a smooth transition from Python 2, > > Python 3.8 will neither raise `DeprecationWarning` nor remove any > > modules that have been scheduled for removal. Instead deprecated > > modules will > > just be *documented* as deprecated. Optionally modules may emit a > > `PendingDeprecationWarning`. > > > > All deprecated modules will also undergo a feature freeze. No additional > > features should be added. Bug should still be fixed. > > > > 3.9 > > --- > > > > Starting with Python 3.9, deprecated modules will start issuing > > `DeprecationWarning`. > > > > > > 3.10 > > ---- > > > > In 3.10 all deprecated modules will be removed from the CPython > repository > > together with tests, documentation, and autoconf rules. > > > > > > PEP acceptance process > > ====================== > > > > 3.8.0b1 is scheduled to be release shortly after the PEP is officially > > submitted. Since it's improbable that the PEP will pass all stages of the > > PEP process in time, I propose a two step acceptance process that is > > analogous Python's two release deprecation process. > > > > The first *provisionally accepted* phase targets Python 3.8.0b1. In the > first > > phase no code is changes or removed. Modules are only documented as > > deprecated. > > > > The final decision, which modules will be removed and how the removed > > code > > is preserved, can be delayed for another year. > > > > > > Deprecated modules > > ================== > > > > The modules are grouped as data encoding, multimedia, network, OS > > interface, > > and misc modules. The majority of modules are for old data formats or > > old APIs. Some others are rarely useful and have better replacements on > > PyPI, e.g. Pillow for image processing or NumPy-based projects to deal > with > > audio processing. > > > > .. csv-table:: Table 1: Proposed modules deprecations > > :header: "Module", "Deprecated in", "To be removed", "Replacement" > > > > aifc,3.8,3.10,\- > > asynchat,3.8,3.10,asyncio > > asyncore,3.8,3.10,asyncio > > audioop,3.8,3.10,\- > > binhex,3.8,3.10,\- > > cgi,3.8,3.10,\- > > cgitb,3.8,3.10,\- > > chunk,3.8,3.10,\- > > colorsys,**3.8?**,**3.10?**,\- > > crypt,3.8,3.10,\- > > fileinput,3.8,3.10,argparse > > formatter,3.4,3.10,\- > > fpectl,**3.7**,**3.7**,\- > > getopt,**3.2**,**keep**,"argparse, optparse" > > imghdr,3.8,3.10,\- > > imp,**3.4**,3.10,importlib > > lib2to3,\-,**keep**, > > macpath,**3.7**,**3.8**,\- > > msilib,3.8,3.10,\- > > nntplib,3.8,3.10,\- > > nis,3.8,3.10,\- > > optparse,\-,**keep**,argparse > > ossaudiodev,3.8,3.10,\- > > pipes,3.8,3.10,subprocess > > smtpd,**3.7**,3.10,aiosmtpd > > sndhdr,3.8,3.10,\- > > spwd,3.8,3.10,\- > > sunau,3.8,3.10,\- > > uu,3.8,3.10,\- > > wave,\-,**keep**, > > xdrlib,3.8,3.10,\- > > > > > > Data encoding modules > > --------------------- > > > > binhex > > ~~~~~~ > > > > The `binhex `_ module > > encodes > > and decodes Apple Macintosh binhex4 data. It was originally developed for > > TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to > > encode binary email attachments. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > uu > > ~~ > > > > The `uu `_ module provides > > uuencode format, an old binary encoding format for email from 1980. The > uu > > format has been replaced by MIME. The uu codec is provided by the > binascii > > module. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > xdrlib > > ~~~~~~ > > > > The `xdrlib `_ module > > supports > > the Sun External Data Representation Standard. XDR is an old binary > > serialization format from 1987. These days it's rarely used outside > > specialized domains like NFS. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > > > Multimedia modules > > ------------------ > > > > aifc > > ~~~~ > > > > The `aifc `_ module > provides > > support for reading and writing AIFF and AIFF-C files. The Audio > Interchange > > File Format is an old audio format from 1988 based on Amiga IFF. It was > most > > commonly used on the Apple Macintosh. These days only few specialized > > application use AIFF. > > > > Module type > > pure Python (depends on `audioop`_ C extension) > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > audioop > > ~~~~~~~ > > > > The `audioop `_ module > > contains helper functions to manipulate raw audio data and adaptive > > differential pulse-code modulated audio data. The module is implemented > in > > C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ > > module depend on `audioop`_ for some operations. > > > > Module type > > C extension > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > colorsys > > ~~~~~~~~ > > > > The `colorsys `_ module > > defines color conversion functions between RGB, YIQ, HSL, and HSV > > coordinate > > systems. The Pillow library provides much faster conversation between > > color systems. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > `Pillow `_, > > `colorspacious `_ > > > > chunk > > ~~~~~ > > > > The `chunk `_ module > > provides > > support for reading and writing Electronic Arts' Interchange File Format. > > IFF is an old audio file format originally introduced for Commodore and > > Amiga. The format is no longer relevant. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > imghdr > > ~~~~~~ > > > > The `imghdr `_ module is > a > > simple tool to guess the image file format from the first 32 bytes > > of a file or buffer. It supports only a limited amount of formats and > > neither returns resolution nor color depth. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > *n/a* > > > > ossaudiodev > > ~~~~~~~~~~~ > > > > The `ossaudiodev `_ > > module provides support for Open Sound System, an interface to sound > > playback and capture devices. OSS was initially free software, but later > > support for newer sound devices and improvements were proprietary. Linux > > community abandoned OSS in favor of ALSA [1]_. Some operation systems > > like > > OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. > > > > Module type > > C extension > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > sndhdr > > ~~~~~~ > > > > The `sndhdr `_ module is > > similar to the `imghdr`_ module but for audio formats. It guesses file > > format, channels, frame rate, and sample widths from the first 512 bytes > of > > a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and > > other ancient formats. > > > > Module type > > pure Python (depends on `audioop`_ C extension for some operations) > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > *n/a* > > > > sunau > > ~~~~~ > > > > The `sunau `_ module > > provides > > support for Sun AU sound format. It's yet another old, obsolete file > format. > > > > Module type > > pure Python (depends on `audioop`_ C extension for some operations) > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > > > Networking modules > > ------------------ > > > > asynchat > > ~~~~~~~~ > > > > The `asynchat `_ module > > is build on top of `asyncore`_ and has been deprecated since Python 3.6. > > > > Module type > > pure Python > > Deprecated in > > 3.6 > > Removed in > > 3.10 > > Substitute > > asyncio > > > > asyncore > > ~~~~~~~~ > > > > The `asyncore `_ module > > was > > the first module for asynchronous socket service clients and servers. It > > has been replaced by asyncio and is deprecated since Python 3.6. > > > > The ``asyncore`` module is also used in stdlib tests. The tests for > > ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly > > based on ``asyncore``. These tests must be updated to use asyncio or > > threading. > > > > Module type > > pure Python > > Deprecated in > > 3.6 > > Removed in > > 3.10 > > Substitute > > asyncio > > > > > > cgi > > ~~~ > > > > The `cgi `_ module is a > support > > module for Common Gateway Interface (CGI) scripts. CGI is deemed as > > inefficient because every incoming request is handled in a new process. > PEP > > 206 considers the module as *designed poorly and are now near-impossible > > to fix*. > > > > Several people proposed to either keep the cgi module for features like > > `cgi.parse_qs()` or move `cgi.escape()` to a different module. The > > functions `cgi.parse_qs` and `cgi.parse_qsl` have been > > deprecated for a while and are actually aliases for > > `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The > > function `cgi.quote` has been deprecated in favor of `html.quote` > > with secure default values. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > > > cgitb > > ~~~~~ > > > > The `cgitb `_ module is a > > helper for the cgi module for configurable tracebacks. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > smtpd > > ~~~~~ > > > > The `smtpd `_ module > > provides > > a simple implementation of a SMTP mail server. The module documentation > > recommends ``aiosmtpd``. > > > > Module type > > pure Python > > Deprecated in > > **3.7** > > To be removed in > > 3.10 > > Substitute > > aiosmtpd > > > > nntplib > > ~~~~~~~ > > > > The `nntplib `_ module > > implements the client side of the Network News Transfer Protocol (nntp). > > News > > groups used to be a dominant platform for online discussions. Over the > last > > two decades, news has been slowly but steadily replaced with mailing > lists > > and web-based discussion platforms. > > > > The ``nntplib`` tests have been the cause of additional work in the > recent > > past. Python only contains client side of NNTP. The test cases depend on > > external news server. These servers were unstable in the past. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > > > Operating system interface > > -------------------------- > > > > crypt > > ~~~~~ > > > > The `crypt `_ module > > implements > > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of > poor > > quality and insecure. Users are discouraged to use them. > > > > * The module is not available on Windows. Cross-platform application need > > an alternative implementation any way. > > * Only DES encryption is guarenteed to be available. DES has an extremely > > limited key space of 2**56. > > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > > algorithm that is still secure. However it's in glibc and therefore not > > commonly available on Linux. > > * Depending on the platform, the ``crypt`` module is not thread safe. > Only > > implementations with ``crypt_r(3)`` are thread safe. > > > > Module type > > C extension + Python module > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > `bcrypt `_, > > `passlib `_, > > `argon2cffi `_, > > hashlib module (PBKDF2, scrypt) > > > > macpath > > ~~~~~~~ > > > > The `macpath `_ module > > provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no > longer > > supported > > > > Module type > > pure Python > > Deprecated in > > 3.7 > > Removed in > > 3.8 > > Substitute > > **none** > > > > nis > > ~~~ > > > > The `nis `_ module provides > > NIS/YP support. Network Information Service / Yellow Pages is an old and > > deprecated directory service protocol developed by Sun Microsystems. It's > > designed successor NIS+ from 1992 never took off. For a long time, libc's > > Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more > > powerful > > and more secure replacement of NIS. > > > > Module type > > C extension > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > spwd > > ~~~~ > > > > The `spwd `_ module > > provides > > direct access to Unix shadow password database using non-standard APIs. > > In general it's a bad idea to use the spwd. The spwd circumvents system > > security policies, it does not use the PAM stack, and is > > only compatible with local user accounts. > > > > Module type > > C extension > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > Misc modules > > ------------ > > > > fileinput > > ~~~~~~~~~ > > > > The `fileinput `_ > module > > implements a helpers to iterate over a list of files from ``sys.argv``. > The > > module predates the optparser and argparser module. The same > > functionality > > can be implemented with the argparser module. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > argparse > > > > formatter > > ~~~~~~~~~ > > > > The `formatter `_ > > module > > is an old text formatting module which has been deprecated since Python > > 3.4. > > > > Module type > > pure Python > > Deprecated in > > 3.4 > > To be removed in > > 3.10 > > Substitute > > *n/a* > > > > imp > > ~~~ > > > > The `imp `_ module is the > > predecessor of the > > `importlib `_ module. > > Most > > functions have been deprecated since Python 3.3 and the module since > > Python 3.4. > > > > Module type > > C extension > > Deprecated in > > 3.4 > > To be removed in > > 3.10 > > Substitute > > importlib > > > > msilib > > ~~~~~~ > > > > The `msilib `_ package > is a > > Windows-only package. It supports the creation of Microsoft Installers > (MSI). > > The package also exposes additional APIs to create cabinet files (CAB). > The > > module is used to facilitate distutils to create MSI installers with > > ``bdist_msi`` command. In the past it was used to create CPython's > official > > Windows installer, too. > > > > Microsoft is slowly moving away from MSI in favor of Windows 10 Apps > > (AppX) > > as new deployment model [3]_. > > > > Module type > > C extension + Python code > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > **none** > > > > pipes > > ~~~~~ > > > > The `pipes `_ module > > provides > > helpers to pipe the input of one command into the output of another > > command. > > The module is built on top of ``os.popen``. Users are encouraged to use > > the subprocess module instead. > > > > Module type > > pure Python > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > subprocess module > > > > Removed modules > > =============== > > > > fpectl > > ------ > > > > The `fpectl `_ module > was > > never built by default, its usage was discouraged and considered > dangerous. > > It also required a configure flag that caused an ABI incompatibility. The > > module was removed in 3.7 by Nathaniel J. Smith in > > `bpo-29137 `_. > > > > Module type > > C extension + CAPI > > Deprecated in > > 3.7 > > Removed in > > 3.7 > > Substitute > > **none** > > > > > > Modules to keep > > =============== > > > > Some modules were originally proposed for deprecation. > > > > lib2to3 > > ------- > > > > The `lib2to3 `_ package > > provides > > the ``2to3`` command to transpile Python 2 code to Python 3 code. > > > > The package is useful for other tasks besides porting code from Python 2 > to > > 3. For example `black`_ uses it for code reformatting. > > > > Module type > > pure Python > > > > getopt > > ------ > > > > The `getopt `_ module > > mimics > > C's getopt() option parser. Although users are encouraged to use argparse > > instead, the getopt module is still widely used. > > > > Module type > > pure Python > > > > optparse > > -------- > > > > The `optparse `_ module > > is > > the predecessor of the argparse module. Although it has been deprecated > > for > > many years, it's still widely used. > > > > Module type > > pure Python > > Deprecated in > > 3.2 > > Substitute > > argparse > > > > wave > > ~~~~ > > > > The `wave `_ module > > provides > > support for the WAV sound format. The module uses one simple function > > from the `audioop`_ module to perform byte swapping between little and > > big > > endian formats. Before 24 bit WAV support was added, byte swap used to be > > implemented with the ``array`` module. To remove ``wave``'s dependency on > > the > > ``audioop``, the byte swap function could be either be moved to another > > module (e.g. ``operator``) or the ``array`` module could gain support for > > 24 bit (3 byte) arrays. > > > > Module type > > pure Python (depends on *byteswap* from `audioop`_ C extension) > > Deprecated in > > 3.8 > > To be removed in > > 3.10 > > Substitute > > *n/a* > > > > > > Future maintenance of removed modules > > ===================================== > > > > The main goal of the PEP is to reduce the burden and workload on the > > Python > > core developer team. Therefore removed modules will not be maintained by > > the core team as separate PyPI packages. However the removed code, tests > > and > > documentation may be moved into a new git repository, so community > > members > > have a place from which they can pick up and fork code. > > > > A first draft of a `legacylib `_ > > repository is available on my private Github account. > > > > It's my hope that some of the deprecated modules will be picked up and > > adopted by users that actually care about them. For example ``colorsys`` > and > > ``imghdr`` are useful modules, but have limited feature set. A fork of > > ``imghdr`` can add new features and support for more image formats, > > without > > being constrained by Python's release cycle. > > > > Most of the modules are in pure Python and can be easily packaged. Some > > depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since > `audioop`_ > > does not depend on any external libraries, it can be shipped in as binary > > wheels with some effort. Other C modules can be replaced with ctypes or > > cffi. > > For example I created `legacycrypt > `_ > > with ``_crypt`` extension reimplemented with a few lines of ctypes code. > > > > > > Discussions > > =========== > > > > * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. > > * Berker Peksag proposed to deprecate and removed *msilib*. > > * Brett Cannon recommended to delay active deprecation warnings and > > removal > > of modules like *imp* until Python 3.10. Version 3.8 will be released > > shortly before Python 2 reaches end of lifetime. A delay reduced churn > for > > users that migrate from Python 2 to 3.8. > > * Brett also came up with the idea to keep lib2to3. The package is useful > > for other purposes, e.g. `black `_ > uses > > it to reformat Python code. > > * At one point, distutils was mentioned in the same sentence as this PEP. > > To avoid lengthy discussion and delay of the PEP, I decided against > dealing > > with distutils. Deprecation of the distutils package will be handled by > > another PEP. > > * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) > > convinced me to keep the `wave`_ module. [4]_ > > * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ > > > > > > References > > ========== > > > > .. [1] > > https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_fre > > e > > .. [2] https://man.openbsd.org/ossaudio > > .. [3] > https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of- > > windows-installer-msi-in-the-light-of-windows-10-and-the-universal- > > windows-platform/ > > .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 > > .. [5] https://twitter.com/dabeaz/status/1130278844479545351 > > > > > > Copyright > > ========= > > > > This document has been placed in the public domain. > > > > > > > > > > .. > > Local Variables: > > mode: indented-text > > indent-tabs-mode: nil > > sentence-end-double-space: t > > fill-column: 70 > > coding: utf-8 > > End: > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > > list%40sdamon.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Thu May 23 15:32:08 2019 From: steve at holdenweb.com (Steve Holden) Date: Thu, 23 May 2019 20:32:08 +0100 Subject: [Python-Dev] PEP 594: Removing dead batteries from the standard library In-Reply-To: References: Message-ID: What would _really_ help is getting the groups that maintain each dead parrot to collaborate on a "Python Legacy release" that adds back anything with a maintainer to the stdlib of the current Python version. Even that will demand large resources, and if it's to be organised in a way that doesn't involve lots of dev time then perhaps the PSF could be involved? I presume the Steering Committee are the people to consider directions like this. Steve Holden On Thu, May 23, 2019 at 7:03 PM Barry Warsaw wrote: > On May 20, 2019, at 13:15, Christian Heimes wrote: > > > > here is the first version of my PEP 594 to deprecate and eventually > remove modules from the standard library. The PEP started last year with > talk during the Python Language Summit 2018, > https://lwn.net/Articles/755229/. > > > > The PEP can be confirmed in two stages. I'm not planning any code > changes for 3.8. Instead I only like to document a bunch of modules as > deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The > long deprecation phase gives us 3 years to change our minds or handle edge > cases, too. > > Hi Christian, > > First, I appreciate the work you?ve done on this, and the general > sentiment. But I do feel like this is a partial solution to a problem the > Python distribution has had for a very long time. Maybe it?s a good step > forward, but in a sense it is incomplete and only chips away at the problem. > > We have two competing pressures, one to provide a rich standard library > with lots of useful features that come right out of the box. Let?s not > underestimate the value that this has for our users, and the contribution > such a stdlib has made to making Python as popular as it is. > > But it?s also true that lots of the stdlib don?t get the love they need to > stay relevant, and a curated path to keeping only the most useful and > modern libraries. I wonder how much the long development cycle and > relatively big overhead for contributing to stdlib maintenance causes a big > part of our headaches with the stdlib. Current stdlib development > processes also incur burden for alternative implementations. > > We?ve had many ideas over the years, such as stripping the CPython repo > stdlib to its bare minimum and providing some way of *distributing* a sumo > tarball. But none have made it far enough to be adopted. I don?t have any > new bright ideas for how to make this work, but I think finding a holistic > approach to these competing pressures is in the best long term interest of > Python. > > That all said, if PEP 594 can be viewed as part of the bigger picture, > then maybe it?s a good idea. Perhaps the approaches for maintaining such > deprecated libraries can be used as an experiment for finding a more > modern, useful, and vibrant approach to stdlib maintenance. > > Cheers, > -Barry > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ezio.melotti at gmail.com Thu May 23 16:17:24 2019 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Thu, 23 May 2019 22:17:24 +0200 Subject: [Python-Dev] PEP 595: Improving bugs.python.org Message-ID: Hello, Berker and I have been working on a PEP that suggests we keep using and improving bugs.python.org and Roundup instead of switching to GitHub Issues as proposed by PEP 581. The PEP covers: * What are the advantages of Roundup over GitHub issues; * What features are missing in Roundup and how can we add them; * Issues with PEP 581; * Issues with the migration plan proposed by PEP 588; The rendered version of PEP 595 is available at https://www.python.org/dev/peps/pep-0595/ For reference, you can consult PEP 581 and 588 at https://www.python.org/dev/peps/pep-0581/ and https://www.python.org/dev/peps/pep-0588/ The full text of the PEP is include below. We are planning to update the PEP to include the feedback we receive and to update the status of features as we implement them (we also have a Google Summer of Code students working on it). Best Regards, Ezio Melotti ================ PEP: 595 Title: Improving bugs.python.org Author: Ezio Melotti , Berker Peksag Status: Draft Type: Process Content-Type: text/x-rst Created: 12-May-2019 Abstract ======== This PEP proposes a list of improvements to make bugs.python.org more usable for contributors and core developers. This PEP also discusses why remaining on Roundup should be preferred over switching to GitHub Issues, as proposed by :pep:`581`. Motivation ========== On May 14th, 2019 :pep:`581` has been accepted [#]_ without much public discussion and without a clear consensus [#]_. The PEP contains factual errors and doesn't address some of the issues that the migration to GitHub Issues might present. Given the scope of the migration, the amount of work required, and how it will negatively affect the workflow during the transition phase, this decision should be re-evaluated. Roundup advantages over GitHub Issues ===================================== This section discusses reasons why Roundup should be preferred over GitHub Issues and Roundup features that are not available on GitHub Issues. * **Roundup is the status quo.** Roundup has been an integral part of the CPython workflow for years. It is a stable product that has been tested and customized to adapt to our needs as the workflow evolved. It is possible to gradually improve it and avoid the disruption that a switch to a different system would inevitabily bring to the workflow. * **Open-source and Python powered.** Roundup is an open-source project and is written in Python. By using it and supporting it, we also support the Python ecosystem. Several features developed for bpo have also been ported to upstream Roundup over the years. * **Fully customizable.** Roundup can be (and has been) fully customized to fit our needs. * **Finer-grained access control.** Roundup allows the creation of different roles with different permissions (e.g. create, view, edit, etc.) for each individual property, and users can have multiple roles. * **Flexible UI.** While Roundup UI might look dated, it is convenient and flexible. For example, on the issue page, each field (e.g. title, type, versions, status, linked files and PRs, etc.) have appropriate UI elements (input boxes, dropdowns, tables, etc.) that are easy to set and also provide a convenient way to get info about the issue at a glance. The number of fields, their values, and the UI element they use is also fully customizable. GitHub only provides labels. The issue list page presents the issues in a compact and easy to read table with separate columns for different fields. For comparison, Roundup lists 50 issues in a screen, whereas GitHub takes two screens to shows 25 issues. * **Advanced search.** Roundup provides an accurate way to search and filter by using any combination of issue fields. It is also possible to customize the number of results and the fields displayed in the table, and the sorting and grouping (up to two levels). bpo also provides predefined summaries (e.g. "Created by you", "Assigned to you", etc.) and allows the creation of custom search queries that can be conveniently accessed from the sidebar. * **Nosy list autocomplete.** The nosy list has an autocomplete feature that suggests maintainers and experts. The suggestions are automatically updated when the experts index [#]_ changes. * **Dependencies and Superseders.** Roundup allows to specify dependencies that must be addressed before the current issues can be closed and a superseder issue to easily mark duplicates [#]_. The list of dependencies can also be used to create meta-issues that references several other sub-issues [#]_. Improving Roundup ================= This section lists some of the issues mentioned by :pep:`581` and other desired features and discusses how they can be implemented by improving Roundup and/or our instance. * **REST API support.** A REST API will make integration with other services and the development of new tools and applications easiers. Upstream Roundup now supports a REST API. Updating the tracker will make the REST API available. * **GitHub login support.** This will allow users to login to bugs.python.org (bpo) without having to create a new account. It will also solve issues with confirmation emails being marked as spam, and provide two-factor authentication. A patch to add this functionality is already available and is being integrated at the time of writing [#]_. * **Markdown support and message preview and editing.** This feature will allow the use of Markdown in messages and the ability to preview the message before the submission and edit it afterward. This can be done, but it will take some work. Possible solutions have been proposed on the roundup-devel mailing list [#]_. * **"Remove me from nosy list" button.** Add a button on issue pages to remove self from the nosy list. This feature will be added during GSoC 2019. * **Mobile friendly theme.** Current theme of bugs.python.org looks dated and it doesn't work well with mobile browsers. A mobile-friendly theme that is more modern but still familiar will be added. * **Add PR link to BPO emails.** Currently bpo emails don't include links to the corresponding PRs. A patch [#]_ is available to change the content of the bpo emails from:: components: +Tkinter versions: +Python 3.4 pull_requests: +42 to:: components: +Tkinter versions: +Python 3.4 pull_request: https://github.com/python/cpython/pull/341 * **Python 3 support.** Using Python 3 will make maintenance easier. Upstream Roundup now supports Python 3. Updating the tracker will allow us to switch to Python 3. The instances will need to be updated as well. * **Use upstream Roundup.** We currently use a fork of Roundup with a few modifications, most notably the GitHub integration. If this is ported upstream, we can start using upstream Roundup without having to maintain our fork. PEP 581 issues ============== This section addresses some errors and inaccuracies found in :pep:`581`. The "Why GitHub?" section of PEP 581 lists features currently available on GitHub Issues but not on Roundup. Some of this features are currently supported: * "Ability to reply to issue and pull request conversations via email." * Being able to reply by email has been one of the core features of Roundup since the beginning. It is also possible to create new issues or close existing ones, set or modify fields, and add attachments. * "Email notifications containing metadata, integrated with Gmail, allowing systematic filtering of emails." * Emails sent by Roundup contains metadata that can be used for filtering. * "Additional privacy, such as offering the user a choice to hide an email address, while still allowing communication with the user through @-mentions." * Email addresses are hidden by default to users that are not registered. Registered users can see other users' addresses because we configured the tracker to show them. It can easily be changed if desired. Users can still be added to the nosy list by using their username even if their address is hidden. * "Ability to automatically close issues when a PR has been merged." * The GitHub integration of Roundup automatically closes issues when a commit that contains "fixes issue " is merged. (Alternative spellings such as "closes" or "bug" are also supported.) See [#]_ for a recent example of this feature. * "Support for permalinks, allowing easy quoting and copying & pasting of source code." * Roundup has permalinks for issues, messages, attachments, etc. In addition, Roundup allows to easily rewrite broken URLs in messages (e.g. if the code hosting changes). * "Core developers, volunteers, and the PSF don't have to maintain the issue infrastructure/site, giving us more time and resources to focus on the development of Python." * While this is partially true, additional resources are required to write and maintain bots. In some cases, bots are required to workaround GitHub's lack of features rather than expanding. [#]_ was written specifically to workaround GitHub's email integration. Updating our bots to stay up-to-date with changes in the GitHub API has also maintenance cost. [#]_ took two days to be fixed. In addition, we will still need to maintain Roundup for bpo (even if it becomes read-only) and for the other trackers we currently host/maintain (Jython [#]_ and Roundup [#]_). The "Issues with Roundup / bpo" section of :pep:`581` lists some issues that have already been fixed: * "The upstream Roundup code is in Mercurial. Without any CI available, it puts heavy burden on the few existing maintainers in terms of reviewing, testing, and applying patches." * While Roundup uses Mercurial by default, there is a git clone available on GitHub [#]_. Roundup also has CI available [#]_ [#]_. * "There is no REST API available. There is an open issue in Roundup for adding REST API. Last activity was in 2016." * The REST API has been integrated and it's now available in Roundup. * "Users email addresses are exposed. There is no option to mask it." * Exposing addresses to registered and logged in users was a decision taken when our instance was set up. This has now been changed to make the email addresses hidden for regular users too (Developers and Coordinators can still see them). The "Email address"" column from the user listing page [#]_ has been removed too. * "It sends a number of unnecessary emails and notifications, and it is difficult, if not impossible, to configure." * This can be configured. * "Creating an account has been a hassle. There have been reports of people having trouble creating accounts or logging in." * The main issue is confirmation emails being marked as spam. Work has been done to resolve the issue. Migration considerations ======================== This section describes issues with the migrations that might not have been addressed by :pep:`581` and :pep:`588`. :pep:`588` suggests to add a button to migrate issues to GitHub only when someone wants to keep working on them. This approach has several issues: * bpo will need to be updated in order to add a button that, once pressed, creates a new issue on GitHub, copies over all the messages, attachments, and creates/adds label for the existing fields. Permissions will also need to be tweaked to make individual issues read-only once they are migrated, and to prevent users to create new accounts. * The issues will be split between two trackers; searching issues will take significant more effort. * The conversion from Roundup to GitHub is lossy, unless all the bpo fields are converted into labels or preserved somewhere else. * bpo converts a number of references into links, including issue, message, and PR IDs, changeset numbers, legacy SVN revision numbers, paths to files in the repo, files in tracebacks (detecting the correct branch), links to devguide pages and sections [#]_. This happens when messages are requested so it is possible to create the correct link (e.g. all the file links used to point to hg.python.org and now point to GitHub). If the links are hardcoded during the migration, it will be difficult (if not impossible) to change them later. If they aren't, they will either be lost, or a tool to generate the links and updating them will need to be written. * GitHub doesn't provide a way to set and preserve issue IDs if they are migrated automatically with the use of a button. (Some projects managed to preserve the IDs by contaacting the GitHub staff and migrating the issues *en masse*.) * On top of the work and changes required to migrate to GitHub issues, we will still need to keep running and maintaining Roundup, for both our instance (read-only) and for the Jython and Roundup trackers (read-write). In addition to the issues listed in the "Open issues" section of :pep:`588`, this issues will need to be addressed: * GitHub is properietary and there is risk of vendor lock-in. Their business model might change and they could shut down altogether. * Switching to GitHub Issues will likely increase the number of invalid reports and increase the triaging effort. This concern has been raised in the past in a Zulip topic [#]_. There have been already cases where people posted comments on PRs that required moderators to mark them as off-topic or disruptive, delete them altogether, and even lock the conversation [#]_. * Roundup sends weekly reports to python-dev with a summary that includes new issues, recent issues with no replies, recent issues waiting for review, most discussed issues, closed issues, and deltas for open/closed/total issue counts [#]_. The report provides an easy way to keep track of the tracker activity and to make sure that issues that require attention are noticed. The data collect by the weekly report is also use to generate statistics and graphs that can be used to gain new insights [#]_. * There are currently two mailing lists where Roundup posts new tracker issues and all messages respectively: new-bugs-announce [#]_ and python-bugs-list [#]_. A new system will need to be developed to preserve this functionality. These MLs offer additional ways to keep track of the tracker activity. References ========== .. [#] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted https://mail.python.org/pipermail/python-dev/2019-May/157399.html .. [#] [python-committers] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted https://mail.python.org/pipermail/python-committers/2019-May/006755.html .. [#] Experts Index -- Python Devguide https://devguide.python.org/experts/ .. [#] An example of superseded issues: "re.sub() replaces only several matches" https://bugs.python.org/issue12078 .. [#] An example of meta issue using dependencies to track sub-issues: "Meta-issue: support of the android platform"" https://bugs.python.org/issue26865 .. [#] Support logging in with GitHub https://github.com/python/bugs.python.org/issues/7 .. [#] Re: [Roundup-devel] PEP 581 and Google Summer of Code https://sourceforge.net/p/roundup/mailman/message/36667828/ .. [#] [Tracker-discuss] [issue624] bpo emails contain useless non-github pull_request number - users want a link to actual github PR https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html .. [#] The commit reported in msg342882 closes the issue (see the history below) https://bugs.python.org/issue36951#msg342882 .. [#] The cpython-emailer-webhook project https://github.com/berkerpeksag/cpython-emailer-webhook .. [#] A recent incident caused by GitHub https://github.com/python/bedevere/pull/163 .. [#] Jython issue tracker https://bugs.jython.org/ .. [#] Roundup issue tracker https://issues.roundup-tracker.org/ .. [#] GitHub clone of Roundup https://github.com/roundup-tracker/roundup .. [#] Travis-CI for Roundup https://travis-ci.org/roundup-tracker/roundup) and codecov .. [#] Codecov for Roundup https://codecov.io/gh/roundup-tracker/roundup/commits .. [#] User listing -- Python tracker https://bugs.python.org/user?@sort=username .. [#] Generating Special Links in a Comment -- Python Devguide https://devguide.python.org/triaging/#generating-special-links-in-a-comment .. [#] The New-bugs-announce mailing list https://mail.python.org/mailman/listinfo/new-bugs-announce .. [#] The Python-bugs-list mailing list https://mail.python.org/mailman/listinfo/python-bugs-list .. [#] An example of [Python-Dev] Summary of Python tracker Issues https://mail.python.org/pipermail/python-dev/2019-May/157483.html .. [#] Issues stats -- Python tracker https://bugs.python.org/issue?@template=stats .. [#] s/n ratio -- Python -- Zulip https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20ratio .. [#] For example this and other related PRs: https://github.com/python/cpython/pull/9099 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From tim.peters at gmail.com Thu May 23 16:39:25 2019 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 23 May 2019 15:39:25 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: It seems pretty clear now that the primary cause is keeping arenas sorted by number of free pools. As deallocation goes on, the number of distinct "# of free pools" values decreases, leaving large numbers of arenas sharing the same value. Then, e.g., if there are 10,000 arenas with 30 free pools remaining, and another pool is freed in one of them, its free pool count jumps to 31, and on average we have to crawl over 5,000 of its former peers to locate its new position in the list. Which is also consistent with what the OP saw, and I did too but in a much smaller case: the longer deallocation goes on, the slower it gets (fewer and fewer Nodes freed per unit time). Which suggests a different - albeit related - approach: it's not really the order of arenas that matters, but the partitioning of arenas by number of free pools. So instead of a singly doubly linked list of arenas, have a vector of doubly linked lists, one list per possible number of free pools. That's a fixed and relatively small number. For example, even if all arena bytes could be used for pools (they can't be - there's bookkeeping info at the start of an arena), the maximum number of free pools in an arena would be 2**18 / 2**12 == 2**6 == 64. When a pool is freed, no big deal: unlink the arena from its current list, and stick it in the list for the next higher number of free pools. This, of course, is constant-time. Of course there are edge cases (e.g., if the area is entirely free, it should be given back to C instead), but they seem pretty obvious, and I haven't thought of a real problem. Tedious, though ;-) From brett at python.org Thu May 23 17:06:13 2019 From: brett at python.org (Brett Cannon) Date: Thu, 23 May 2019 14:06:13 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> <20190521173112.503f4c91@fsol> <27ba695b-3219-a628-acef-4e0af649e2de@python.org> <20190522121237.021cbec4@fsol> Message-ID: On Wed, May 22, 2019 at 1:23 PM Sean Wallitsch < sean.wallitsch at dreamworks.com> wrote: > My apologies for that oversight. My understanding is that many of the > methods present in aifc depend heavily on audioop for reading and writing. > But are people using audioop directly? This shifts whether we still publicly deprecate auidoop and make it an implementation detail of aifc versus a free-standing module whose API we need to continue to support. -Brett > > On Wed, May 22, 2019 at 12:35 PM Nathaniel Smith wrote: > >> On Wed, May 22, 2019, 12:14 Sean Wallitsch >> wrote: >> >>> Dear python-dev, >>> >>> I'm writing to provide some feedback on PEP-594, primarily the proposed >>> deprecation and reason for the removal of the aifc and audioop libraries. >>> >>> The post production film industry continues to make heavy use of AIFFs, >>> as completely uncompressed audio is preferred. Support for the consumer >>> alternatives (ALAC, FLAC) is virtually non-existent, with no movement >>> towards adoption of those formats. Even Apple's own professional editing >>> tool Final Cut Pro does not support ALAC. Many of the applications also >>> support WAV, but not all. >>> >>> Removal of this module from the standard library is complicated by the >>> fact that a large number of film industry facilities have extremely limited >>> internet access for security reasons. This does not make it impossible to >>> get a library from pypi, but speaking to those devs has made me aware of >>> what a painful process that is for them. They have benefited greatly from >>> aifc's inclusion in the standard library. >>> >> >> That's really helpful data, thank you! >> >> Is audioop also used? You mention both aifc and audioop at the beginning >> and end of your message, but all the details in the middle focus on just >> aifc. >> >> -n >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu May 23 17:17:28 2019 From: brett at python.org (Brett Cannon) Date: Thu, 23 May 2019 14:17:28 -0700 Subject: [Python-Dev] we will probably be having an difficult discussion about the stdlib after PEP 594 is done (was: PEP 594: Removing dead batteries from the standard library) In-Reply-To: References: Message-ID: On Thu, May 23, 2019 at 11:03 AM Barry Warsaw wrote: > On May 20, 2019, at 13:15, Christian Heimes wrote: > > > > here is the first version of my PEP 594 to deprecate and eventually > remove modules from the standard library. The PEP started last year with > talk during the Python Language Summit 2018, > https://lwn.net/Articles/755229/. > > > > The PEP can be confirmed in two stages. I'm not planning any code > changes for 3.8. Instead I only like to document a bunch of modules as > deprecated. Active deprecation is planned for 3.9 and removal for 3.10. The > long deprecation phase gives us 3 years to change our minds or handle edge > cases, too. > > Hi Christian, > > First, I appreciate the work you?ve done on this, and the general > sentiment. But I do feel like this is a partial solution to a problem the > Python distribution has had for a very long time. Maybe it?s a good step > forward, but in a sense it is incomplete and only chips away at the problem. > Yep, but I will take this chipping away over nothing. :) > > We have two competing pressures, one to provide a rich standard library > with lots of useful features that come right out of the box. Let?s not > underestimate the value that this has for our users, and the contribution > such a stdlib has made to making Python as popular as it is. > > But it?s also true that lots of the stdlib don?t get the love they need to > stay relevant, and a curated path to keeping only the most useful and > modern libraries. I wonder how much the long development cycle and > relatively big overhead for contributing to stdlib maintenance causes a big > part of our headaches with the stdlib. Current stdlib development > processes also incur burden for alternative implementations. > > We?ve had many ideas over the years, such as stripping the CPython repo > stdlib to its bare minimum and providing some way of *distributing* a sumo > tarball. But none have made it far enough to be adopted. I don?t have any > new bright ideas for how to make this work, but I think finding a holistic > approach to these competing pressures is in the best long term interest of > Python. > > That all said, if PEP 594 can be viewed as part of the bigger picture, > then maybe it?s a good idea. Perhaps the approaches for maintaining such > deprecated libraries can be used as an experiment for finding a more > modern, useful, and vibrant approach to stdlib maintenance. > I'm personally viewing it as a first step in addressing the maintenance burden we have with such a large stdlib. Christian started this work over a year ago and I think it's worth seeing through. After that we should probably have a discussion as a team about how we view the stdlib long-term and how that ties into maintaining it so that people's opinion of the stdlib's quality goes up rather than viewing the quality of it as varying module-to-module. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu May 23 20:44:22 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 May 2019 10:44:22 +1000 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <27ba695b-3219-a628-acef-4e0af649e2de@python.org> <20190522121237.021cbec4@fsol> Message-ID: <20190524004422.GR4221@ando.pearwood.info> On Thu, May 23, 2019 at 02:06:13PM -0700, Brett Cannon wrote: > On Wed, May 22, 2019 at 1:23 PM Sean Wallitsch < > sean.wallitsch at dreamworks.com> wrote: > > > My apologies for that oversight. My understanding is that many of the > > methods present in aifc depend heavily on audioop for reading and writing. > > > > But are people using audioop directly? Yes they are. https://old.reddit.com/r/Python/comments/brgl8v/pep_594_removing_dead_batteries_from_the_standard/eodvexl/ Shouldn't the PEP be responsable for establishing (as well as any negative can be proven) that these modules aren't used, rather than merely assuming that they aren't? Of course it is hard to establish that a module isn't being used. Not all code can be searched for on the Internet, there's huge amounts of non-open source code written by users that never leaves their computer, or behind corporate firewalls. The difficulty of proving this negative requires that we be especially conservative about removal. At the very least, we should demonstrate that the package is *an active burden*. Its not enough to say "nobody has touched this module for ages" since stable, working code doesn't rot unless we change the language (or the std lib!) and break it. So far, the PEPs record hasn't been too good: out of the 31 modules listed in the original draft, the PEP marks four as "keep", leaving 27 to be deprecated. So far, we've found: - the sound libraries are in heavy use by hobbyists and the professional audio-visual industry; aifc in particular is a mature, stable library that isn't break and doesn't need much attention; - cgi and cgitb are used by people who don't need a heavyweight HTML solution (see Reddit); - colorsys, fileinput and nntplib (at least) have been added to the "keep" list; - the removal of spwd (and crypt?) has been controversial. So far, nearly 30% of the so-called "dead batteries" turn out to be not so dead after all. I may have missed some. Nor have I even talked much about modules which I personally use occasionally, like binhex, because it's not about *me* its about the community. As much as I would prefer binhex to remain, if it is removed I will be disappointed but I'll make do. Speaking of binhex, I tried to get a sense of how much of a burden it actually is. There was a comment by Guido in *2007*: https://github.com/python/cpython/commit/34a042d301d6ab88645046a6dfa6c38265ca4b39 "This is the last time I fix binhex. If it breaks again it goes in the dustbin" which of course is Guido's perogative to say Won't Fix. Its now 2019 and it hasn't broken again, or at least it hasn't broken again sufficiently to make any difference. Serhey found one issue: https://bugs.python.org/issue29566 but I'm not sure if that's actually a bug or not. There was a documentation bug (in the source code) in 2017: https://bugs.python.org/issue29557 and one enhancement request to clean up the code: https://bugs.python.org/issue34063 Other than that, Serhey touched binhex.py as part of a mass patch to almost the whole stdlib to modernise file handling to use "with". So by my count, in 12 years since Guido's frustrated comment about binhex, it has seen: - removal of an obsolete and ambiguous comment from the source code; - addition of a few with blocks to modernise file handling; - one enhancement request still outstanding; - one possibly a bug (or maybe not) still outstanding. I may have missed some, but we're talking about one issue per three years. How is this a maintenance burden? -- Steven From barry at python.org Thu May 23 21:02:20 2019 From: barry at python.org (Barry Warsaw) Date: Thu, 23 May 2019 18:02:20 -0700 Subject: [Python-Dev] we will probably be having an difficult discussion about the stdlib after PEP 594 is done (was: PEP 594: Removing dead batteries from the standard library) In-Reply-To: References: Message-ID: <61B1753E-F700-46DA-9992-C67D09B8268B@python.org> On May 23, 2019, at 14:17, Brett Cannon wrote: > I'm personally viewing it as a first step in addressing the maintenance burden we have with such a large stdlib. Christian started this work over a year ago and I think it's worth seeing through. After that we should probably have a discussion as a team about how we view the stdlib long-term and how that ties into maintaining it so that people's opinion of the stdlib's quality goes up rather than viewing the quality of it as varying module-to-module. +1 :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From greg.ewing at canterbury.ac.nz Thu May 23 22:25:33 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 24 May 2019 14:25:33 +1200 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: Message-ID: <5CE7561D.2060806@canterbury.ac.nz> Steve Dower wrote: > We need to make it more clear somehow that Python uses > series.major.minor.micro versioning, not SemVer. We could also do with deciding what the "series" part is really supposed to be for. At the moment it seems to be "we change it when the major number gets up to 9, or we feel like it, whichever comes first." -- Greg From songofacandy at gmail.com Fri May 24 01:35:27 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Fri, 24 May 2019 14:35:27 +0900 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: On Fri, May 24, 2019 at 3:49 AM Gregory P. Smith wrote: > > I suggest filing a bug to track this... > I created the issue: https://bugs.python.org/issue37029 -- Inada Naoki From steve at pearwood.info Fri May 24 02:55:44 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 May 2019 16:55:44 +1000 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules Message-ID: <20190524065544.GS4221@ando.pearwood.info> I don't know if this is a good idea or a terrible idea or somewhere in between, so I'm throwing it out to see if anyone likes it. Let's add a third option to PEP 594 between "keep" and "remove": explicitly flagging a module as unmaintained. Unmaintained modules: - will raise a warning when imported that they are unmaintained; - will have their tests disabled; - possibly we move them into a seperate namespace: ``from unmaintained import aardvark`` - bug reports without patches will be closed Will Not Fix; - bug reports with patches *may* be accepted if some core dev is willing to review and check it in, but there is no obligation to do so; - should it turn out that someone is willing to maintain the module, it can be returned to regular status. Plus side: - reduce the maintenance burden (if any) from the module; - while still distributing the module and allowing users to use it: "no promises, but here you go"; - other implementations are under no obligation to distribute unmaintained modules. Minus side: - this becomes a self-fulfilling prophesy: with tests turned off, bit-rot will eventually set in and break modules that currently aren't broken. Thoughts? -- Steven From songofacandy at gmail.com Fri May 24 03:17:20 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Fri, 24 May 2019 16:17:20 +0900 Subject: [Python-Dev] PEP 594 -- Bundling libraries? Message-ID: When removing libraries from stdlib, can we bundle removed libraries and install it like ensurepip does? Ruby does similar thing, called "Gemification". See https://rubykaigi.org/2017/presentations/hsbt.html When people don't use venv, scripts importing nntplib or aifc runs correctly. When people use venv, they need to `pip install nntplib` (we may be able to use PyPI's namespace beta feature here). But I assume `pip install` is not a problem for people using venv. Regards, -- Inada Naoki From songofacandy at gmail.com Fri May 24 03:37:06 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Fri, 24 May 2019 16:37:06 +0900 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: <20190523183331.2163dd60@fsol> References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> <20190523183331.2163dd60@fsol> Message-ID: > > > > > > It's relatively easy to test replacing our custom allocators with the > > > system ones, yes? Can we try those to see whether they have the same > > > characteristic? > > > > > > > Yes. > > > > PYTHONMALLOC=malloc LD_PRELOAD=/path/to/jemalloc.so python script.py > > > > I will try it tomorrow. > > Can you simply test with the system allocator rather than jemalloc? > For the record, result for 10M nodes, Ubuntu 18.04 on AWS r5a.4xlarge: $ local/bin/python3 t1.py # default 1138.1123778309993 -- end train, start del 688.7927911250008 -- end $ arena-1m/bin/python3 t1.py # Changed ARENA_SIZE to 1MiB 1085.3363994170013 -- end train, start del 84.57135540099989 -- end $ PYTHONMALLOC=malloc local/bin/python3 t1.py 1157.4882792789995 -- end train, start del 27.919834706000074 -- end $ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 PYTHONMALLOC=malloc local/bin/python3 t1.py 1098.4383037820007 -- end train, start del 117.93938426599925 -- end In this case, glibc malloc is the fastest. glibc is know to weak about fragmentation. But algorithm to avoid fragmentation is just an overhead in this script. Regards, -- Inada Naoki From vstinner at redhat.com Fri May 24 07:21:22 2019 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 24 May 2019 13:21:22 +0200 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> <20190523183331.2163dd60@fsol> Message-ID: Le ven. 24 mai 2019 ? 09:41, Inada Naoki a ?crit : > For the record, result for 10M nodes, Ubuntu 18.04 on AWS r5a.4xlarge: > > $ local/bin/python3 t1.py # default > 1138.1123778309993 -- end train, start del > 688.7927911250008 -- end > > $ arena-1m/bin/python3 t1.py # Changed ARENA_SIZE to 1MiB > 1085.3363994170013 -- end train, start del > 84.57135540099989 -- end 688 => 84 looks like an interesting speedup. Would it be technically possible to make ARENA_SIZE configurable at runtime? Using the new PEP 587 preinitialization, it shouldn't be too hard to hard a new command line option and/or an environment variable to tune the memory allocator: https://www.python.org/dev/peps/pep-0587/#preinitialization-with-pypreconfig Victor -- Night gathers, and now my watch begins. It shall not end until my death. From thomas at python.org Fri May 24 08:23:21 2019 From: thomas at python.org (Thomas Wouters) Date: Fri, 24 May 2019 14:23:21 +0200 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> Message-ID: On Thu, May 23, 2019 at 5:15 PM Steve Dower wrote: > On 23May2019 0542, Inada Naoki wrote: > > 1. perf shows 95% of CPU time is eaten by _PyObject_Free, not kernel > space. > > 2. This loop is cleary hot: > > > https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 > > > > I can attach the process by gdb and I confirmed many arenas have > > same nfreepools. > > It's relatively easy to test replacing our custom allocators with the > system ones, yes? Can we try those to see whether they have the same > characteristic? > > Given the relative amount of investment over the last 19 years [1], I > wouldn't be surprised if most system ones are at least as good for our > needs now. Certainly Windows HeapAlloc has had serious improvements in > that time to help with fragmentation and small allocations. > FYI, and I've mentioned this at PyCon to a few people (might've been you, Steve, I don't remember) -- but at Google we've experimented with disabling obmalloc when using TCMalloc (a faster and thread-aware malloc, which makes a huge difference within Google when dealing with multi-threaded C++ libraries), both using the usual Python benchmarks and real-world code with real-world workloads (a core part of YouTube, for example), all on Linux. There's still a significant benefit to using obmalloc when using glibc's malloc, and also a noticeable benefit when using TCMalloc. There are certainly cases where it doesn't matter much, and there may even be cases where the overhead of obmalloc isn't worth it, but I do believe it's still a firm net benefit. > > Cheers, > Steve > > [1]: > > https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L769 > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/thomas%40python.org > -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From devel2019 at baptiste-carvello.net Fri May 24 05:20:57 2019 From: devel2019 at baptiste-carvello.net (Baptiste Carvello) Date: Fri, 24 May 2019 11:20:57 +0200 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: <010000d6-1450-9953-6258-0c93b56d98f2@baptiste-carvello.net> Hello, Le 21/05/2019 ? 22:30, Steve Dower a ?crit?: > > [...] > > * the Python 3.7 installed from the store will not auto-update to 3.8, > but when 3.8 is released we (Microsoft) will update the redirect to > point at it > * if you pass arguments to the redirect command, it just exits with an > error code - you only get the Store page if you run it without arguments I was wondering how those 2 bullet points combine. Say a user installs 3.7 from the store, then uses "python.exe" with arguments, in a shebang line or a batch script. Does it mean the script might break unexpectedly when 3.8 is released? Then, would it make sense for the redirect command to proxy to the 3.7 install when arguments are passed? Cheers, Baptiste From benjamin_cail at brown.edu Fri May 24 10:49:36 2019 From: benjamin_cail at brown.edu (Ben Cail) Date: Fri, 24 May 2019 10:49:36 -0400 Subject: [Python-Dev] we will probably be having an difficult discussion about the stdlib after PEP 594 is done In-Reply-To: <61B1753E-F700-46DA-9992-C67D09B8268B@python.org> References: <61B1753E-F700-46DA-9992-C67D09B8268B@python.org> Message-ID: First, thanks to all the python core devs (and others) who work on Python - many, many people (myself included) benefit from your work. Suggestion below... On 5/23/19 9:02 PM, Barry Warsaw wrote: > On May 23, 2019, at 14:17, Brett Cannon wrote: > >> I'm personally viewing it as a first step in addressing the maintenance burden we have with such a large stdlib. Christian started this work over a year ago and I think it's worth seeing through. After that we should probably have a discussion as a team about how we view the stdlib long-term and how that ties into maintaining it so that people's opinion of the stdlib's quality goes up rather than viewing the quality of it as varying module-to-module. > +1 :) Why not have the PSF hire someone (or multiple people) to be paid to work on the maintenance burden? This could be similar to the Django fellows: https://www.djangoproject.com/fundraising/#who-is-the-django-fellow. It seems like a good thing for Django, and Python is used by many more people than Django. Why not pay someone to do the work that others don't want to do? The person in this position could be guided by the PSF and/or the Steering Council, to do the work most necessary for the good of the language as a whole (like maintaining old modules that other core devs don't want to work on). You could market it together with the maintenance burden: "you want to use all these old modules, but we don't want to maintain them. So pay us some money, and we'll hire someone to maintain them." If this idea has already been discussed in this context and rejected, could it be added to the PEP, with the reasons why it won't work? Thanks again for your work. -Ben From steve.dower at python.org Fri May 24 12:11:33 2019 From: steve.dower at python.org (Steve Dower) Date: Fri, 24 May 2019 09:11:33 -0700 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <010000d6-1450-9953-6258-0c93b56d98f2@baptiste-carvello.net> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> <010000d6-1450-9953-6258-0c93b56d98f2@baptiste-carvello.net> Message-ID: <8514e16f-0068-6cbb-b8f9-5ee8d2c15406@python.org> On 24May2019 0220, Baptiste Carvello wrote: > Hello, > > Le 21/05/2019 ? 22:30, Steve Dower a ?crit?: >> >> [...] >> >> * the Python 3.7 installed from the store will not auto-update to 3.8, >> but when 3.8 is released we (Microsoft) will update the redirect to >> point at it >> * if you pass arguments to the redirect command, it just exits with an >> error code - you only get the Store page if you run it without arguments > > I was wondering how those 2 bullet points combine. Say a user installs > 3.7 from the store, then uses "python.exe" with arguments, in a shebang > line or a batch script. > > Does it mean the script might break unexpectedly when 3.8 is released? > Then, would it make sense for the redirect command to proxy to the 3.7 > install when arguments are passed? This is a very big complicated situation (just like on Linux), but the answer is that if you are worried about your script breaking, then don't use "python.exe" in the shebang line or batch file. You can use "python3.7" just fine (well, no shebang lines on Windows). As for the redirect command, if you install Python from the store, it will be replaced completely. Uninstalling Python doesn't even bring it back (another thing we considered and decided wasn't worth the effort - if you've installed Python once then you probably know how to find it again). And if you replace it with something higher in PATH then it's just like shadowing any other command, including the case where one day Microsoft adds a new version of a command you were using (e.g. ssh or bash :) ). So there's really no change to your scenario due to the redirect, just the same old "if you don't specify the version of Python you want then you'll get what you get". (Refer to the linux-sig list for detailed discussion.) Cheers, Steve From steve.dower at python.org Fri May 24 12:54:05 2019 From: steve.dower at python.org (Steve Dower) Date: Fri, 24 May 2019 09:54:05 -0700 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules In-Reply-To: <20190524065544.GS4221@ando.pearwood.info> References: <20190524065544.GS4221@ando.pearwood.info> Message-ID: <25c05229-7e8a-c8b7-8445-ccbb5836aecb@python.org> On 23May2019 2355, Steven D'Aprano wrote: > I don't know if this is a good idea or a terrible idea or somewhere in > between, so I'm throwing it out to see if anyone likes it. > > Let's add a third option to PEP 594 between "keep" and "remove": > explicitly flagging a module as unmaintained. Unmaintained modules: > > - will raise a warning when imported that they are unmaintained; Nobody reads warnings. > - will have their tests disabled; I'm fine with this. Reducing time to run the test suite is the biggest immediate win from removing unmaintained modules. > - possibly we move them into a seperate namespace: > ``from unmaintained import aardvark`` May as well move them all the way to PyPI and leave the import name the same. > - bug reports without patches will be closed Will Not Fix; By whom? > - bug reports with patches *may* be accepted if some core dev is > willing to review and check it in, but there is no obligation > to do so; Historically, nobody has been willing to review and merge these modules for years. How long do we have to wait? > - should it turn out that someone is willing to maintain the module, > it can be returned to regular status. Or we can give them the keys to the PyPI package. Or help pip implement the "Obsoleted By" header and redirect to a fork. > Plus side: > > - reduce the maintenance burden (if any) from the module; Apart from having to read, review, and decide on bug reports, CVEs, and documentation. > - while still distributing the module and allowing users to use > it: "no promises, but here you go"; a.k.a. PyPI? > - other implementations are under no obligation to distribute > unmaintained modules. a.k.a. PyPI? > Minus side: > > - this becomes a self-fulfilling prophesy: with tests turned off, > bit-rot will eventually set in and break modules that currently > aren't broken. True. And then we have the added maintenance burden of explaining repeatedly that we don't care that it's broken, but we want to put it on your machine anyway. All in all, this is basically where we are today, with the exception that we haven't officially said that we no longer support these modules. PEP 594 is this official statement, and our usual process for things we don't support is to remove them in two versions time. It doesn't have to be so controversial - either the people who are saying "we rely on this" are also willing to help us maintain them, or they're not. And if they're not, they clearly don't rely on it (or realize the cost of relying on volunteer-maintained software). Cheers, Steve From tim.peters at gmail.com Fri May 24 13:48:30 2019 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 24 May 2019 12:48:30 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> <20190523183331.2163dd60@fsol> Message-ID: [Inada Naoki ] > For the record, result for 10M nodes, Ubuntu 18.04 on AWS r5a.4xlarge: I'm unclear on what "nodes" means. If you mean you changed 27M to 10M in this line: for token in random_strings(27_000_000): that's fine, but there are about 40 times more than that `Node` objects created by the program. So if you did change that to 10_000_000, you'd have about 400M `Node` objects, consuming about 80x that many bytes of RAM (or about 32GB). > $ local/bin/python3 t1.py # default > 1138.1123778309993 -- end train, start del > 688.7927911250008 -- end > > $ arena-1m/bin/python3 t1.py # Changed ARENA_SIZE to 1MiB > 1085.3363994170013 -- end train, start del > 84.57135540099989 -- end Nice! I assume these are seconds. On Stackoverflow, the OP reported that boosting ARENA_SIZE the same way cut deallocation time in his original program to about 13 minutes. > $ PYTHONMALLOC=malloc local/bin/python3 t1.py > 1157.4882792789995 -- end train, start del > 27.919834706000074 -- end While the OP reported, for the original program: """ With PYTHONMALLOC=malloc, insertion takes nearly twice as long, but deallocation only takes 2 minutes! """ The "nearly twice as long" for building the tree is in sharp contrast to what you saw, but there's about 3x as much of everything in the original program, and, e.;g., it's certainly possible malloc is tripping over fragmentation then that obmalloc avoids. > $ LD_PRELOAD=/usr/lib/x86_64-linux-gnu/libjemalloc.so.1 > PYTHONMALLOC=malloc local/bin/python3 t1.py > 1098.4383037820007 -- end train, start del > 117.93938426599925 -- end > > In this case, glibc malloc is the fastest. > glibc is know to weak about fragmentation. > But algorithm to avoid fragmentation is just an overhead in this script. I can't say. I'll note that the approach I very briefly sketched before (restructure the list of arenas to partition it into multiple lists partitioned by number of free pools) "should make" obmalloc competitive with malloc here (it would eliminate all searches, except perhaps (depending on how gonzo the code is) a rare need to search for "the first" non-empty list). From g.rodola at gmail.com Fri May 24 13:56:00 2019 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 25 May 2019 00:56:00 +0700 Subject: [Python-Dev] Python in next Windows 10 update In-Reply-To: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> References: <768e3895-2d7e-4150-f043-3c59d04e9de3@python.org> Message-ID: On Wed, 22 May 2019 at 03:30, Steve Dower wrote: > Hi all > > Just sharing this here because I think it's important for us to be aware > of it - I'm not trying to promote or sell anything here :) (Those who > were at the language summit have seen this already.) > > In the next Windows 10 update that starts rolling out today, we > (Microsoft) have added "python.exe" and "python3.exe" commands that are > installed on PATH *by default* and will open the Microsoft Store at the > page where we (Python core team) publish our build. > > This makes it a 1-2 click process to get from a clean machine to having > a usable Python install ("python.exe" -> opens Store -> "Get it Free" -> > "python.exe" now works!) > > The associated blog post: > > > https://devblogs.microsoft.com/python/python-in-the-windows-10-may-2019-update/ > > Here are answers to a few questions that I assume will come up, at least > from this audience that understands the issues better than most: > > * if someone had installed Python and put it on PATH with our installer, > this new command *does not* interfere > * if someone had manually modified their own PATH, they *may* see some > interference (but we [Microsoft] decided this was an acceptable risk) > * the Python 3.7 installed from the store will not auto-update to 3.8, > but when 3.8 is released we (Microsoft) will update the redirect to > point at it > * if you pass arguments to the redirect command, it just exits with an > error code - you only get the Store page if you run it without arguments > * once the Store package is installed, the redirect command is replaced > (this required a new feature in the OS). If you install with the regular > installer and update PATH, or active a venv, it will add it *before* the > redirect. So these scenarios should be all good. > > I'm happy to answer other questions here. The long-term contact for this > integration is python (at) microsoft.com, which right now will come to me. > > And on a personal note, I'm very excited that we (Microsoft) got the > approval to do this. Getting *anything* added to Windows is a big task, > so it's a reflection of the popularity and support for Python that's > growing within Microsoft that we were able to make this happen. That's > due to every contributor, both to the core runtime and the ecosystem. I > hope this will only help us improve the availability of Python for users > and make it an easier choice for dev tasks in the future. > > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/g.rodola%40gmail.com > I am really glad this happened. I think that in a sense this could be considered sort of historical. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri May 24 14:06:03 2019 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 24 May 2019 11:06:03 -0700 Subject: [Python-Dev] we will probably be having an difficult discussion about the stdlib after PEP 594 is done In-Reply-To: References: <61B1753E-F700-46DA-9992-C67D09B8268B@python.org> Message-ID: On Fri, May 24, 2019, 08:08 Ben Cail wrote: > > Why not have the PSF hire someone (or multiple people) to be paid to > work on the maintenance burden? This could be similar to the Django > fellows: > https://www.djangoproject.com/fundraising/#who-is-the-django-fellow. It > seems like a good thing for Django, and Python is used by many more > people than Django. Why not pay someone to do the work that others don't > want to do? The person in this position could be guided by the PSF > and/or the Steering Council, to do the work most necessary for the good > of the language as a whole (like maintaining old modules that other core > devs don't want to work on). > > You could market it together with the maintenance burden: "you want to > use all these old modules, but we don't want to maintain them. So pay us > some money, and we'll hire someone to maintain them." > I think the basic idea here is a good one, but: - transitioning from an all-volunteer project to a mixed paid-staff+volunteers project is a big step, and we'll need to take time to figure out what that would look like before people are comfortable with it. - even if we have folks paid to help with maintenance, it won't mean we suddenly have infinite resources and can do everything. We'll still need to pick which things to prioritize. And I think if you asked 100 people to name the most critical issues facing Python today, most of them would not say "maintaining xdrlib". -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri May 24 14:08:23 2019 From: status at bugs.python.org (Python tracker) Date: Fri, 24 May 2019 18:08:23 +0000 (UTC) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20190524180823.DADF652B26B@bugs.ams1.psf.io> ACTIVITY SUMMARY (2019-05-17 - 2019-05-24) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7067 ( +0) closed 41721 (+88) total 48788 (+88) Open issues with patches: 2849 Issues opened (59) ================== #34651: Disallow fork in a subinterpreter. https://bugs.python.org/issue34651 reopened by vstinner #36950: test.support: add an helper to wait for an event with a timeou https://bugs.python.org/issue36950 opened by vstinner #36953: Remove collections ABCs? https://bugs.python.org/issue36953 opened by p-ganssle #36954: test_recursive_repr breaks tracing in test_xml_etree https://bugs.python.org/issue36954 opened by gphemsley #36956: Calling "functions" used to implement generators/comps easily https://bugs.python.org/issue36956 opened by bup #36959: ISO date errors in _strptime are jumbled https://bugs.python.org/issue36959 opened by gphemsley #36960: Make datetime docs more user-friendly https://bugs.python.org/issue36960 opened by bsolomon1124 #36967: Eliminate unnecessary check in _strptime when determining AM/P https://bugs.python.org/issue36967 opened by gphemsley #36968: Top level transient modal windows stopping deiconify on window https://bugs.python.org/issue36968 opened by Daniel Law #36969: pdb.do_args: display keyword-only and positional only argument https://bugs.python.org/issue36969 opened by blueyed #36971: Add subsections in C API "Common Object Structures" page https://bugs.python.org/issue36971 opened by jdemeyer #36973: test_json.test_recursion.TestPyRecursion.test_endless_recursio https://bugs.python.org/issue36973 opened by xtreak #36974: Implement PEP 590 https://bugs.python.org/issue36974 opened by jdemeyer #36976: email: AttributeError https://bugs.python.org/issue36976 opened by alter-bug-tracer #36977: SharedMemoryManager should relase its resources when its paren https://bugs.python.org/issue36977 opened by pierreglaser #36979: ncurses extension uses wrong include path https://bugs.python.org/issue36979 opened by chargr #36982: Add support for extended color functions in ncurses 6.1 https://bugs.python.org/issue36982 opened by websurfer5 #36983: typing.__all__ has drifted from actual contents https://bugs.python.org/issue36983 opened by Anthony Sottile #36984: typing docs "versionadded" is inaccurate for many attributes https://bugs.python.org/issue36984 opened by Anthony Sottile #36985: typing.ForwardRef is undocumented https://bugs.python.org/issue36985 opened by Anthony Sottile #36986: tarfile: unexpected IsADirectoryError on extraction https://bugs.python.org/issue36986 opened by alter-bug-tracer #36988: zipfile: string IndexError on extract https://bugs.python.org/issue36988 opened by alter-bug-tracer #36989: test_thread fails because symbol is (no longer) exported https://bugs.python.org/issue36989 opened by Michael.Felt #36990: test_asyncio.test_create_connection_ipv6_scope fails(in mock t https://bugs.python.org/issue36990 opened by Michael.Felt #36996: unittest.mock.patch decorator doesn't work with async function https://bugs.python.org/issue36996 opened by xtreak #36997: Document that spwd is considered harmful https://bugs.python.org/issue36997 opened by christian.heimes #36998: distutils sdist command fails to create MANIFEST if any filena https://bugs.python.org/issue36998 opened by a.badger #36999: Expose the coroutine object in asyncio task objects https://bugs.python.org/issue36999 opened by alex.gronholm #37001: symtable.symtable doesn't accept bytes which leads to a mismat https://bugs.python.org/issue37001 opened by dino.viehland #37002: PyType_FromSpec can't create immutable types https://bugs.python.org/issue37002 opened by dino.viehland #37003: ast unparse does not support f-string new debug format. https://bugs.python.org/issue37003 opened by mbussonn #37004: SequenceMatcher.ratio() noncommutativity not well-documented https://bugs.python.org/issue37004 opened by Dennis Sweeney #37005: bz2 module doesn't write end-of-stream marker https://bugs.python.org/issue37005 opened by Dobatymo #37006: Add top level await statement support for doctest https://bugs.python.org/issue37006 opened by xtreak #37007: Implement socket.if_{nametoindex,indextoname} for Windows https://bugs.python.org/issue37007 opened by dtantsur #37009: Threading and THREAD_SAFE for AIX https://bugs.python.org/issue37009 opened by Michael.Felt #37010: Review performance of inspect.getfullargspec https://bugs.python.org/issue37010 opened by ncoghlan #37011: pdb: restore original tracing function instead of sys.settrace https://bugs.python.org/issue37011 opened by blueyed #37012: Possible crash due to PyType_FromSpecWithBases() https://bugs.python.org/issue37012 opened by petr.viktorin #37013: Fatal Python error in socket.if_indextoname() https://bugs.python.org/issue37013 opened by ZackerySpytz #37014: [First easy issue] fileinput module should document that openh https://bugs.python.org/issue37014 opened by Grant Wu2 #37015: Fix asyncio mock warnings https://bugs.python.org/issue37015 opened by asvetlov #37017: Use LOAD_METHOD optimization in CallMethod C API functions https://bugs.python.org/issue37017 opened by msullivan #37018: put http.server on a diet https://bugs.python.org/issue37018 opened by v+python #37021: Can _random.getrandbits() be converted to METH_FASTCALL? https://bugs.python.org/issue37021 opened by rhettinger #37022: pdb: do_p and do_pp swallow exceptions from __repr__ https://bugs.python.org/issue37022 opened by blueyed #37023: test_gdb failed on AMD64 Debian PGO 3.x https://bugs.python.org/issue37023 opened by vstinner #37024: SQLite flag in configure due to homebrew not linking sqlite https://bugs.python.org/issue37024 opened by lys.nikolaou #37025: Misleading error message "Python failed to load the default ac https://bugs.python.org/issue37025 opened by rhabacker #37026: socketserver: BaseServer.handle_request() infinite loop https://bugs.python.org/issue37026 opened by jmberg #37027: Return a safe proxy over a socket from get_extra_info('socket' https://bugs.python.org/issue37027 opened by yselivanov #37028: Implement asyncio repl https://bugs.python.org/issue37028 opened by yselivanov #37029: PyObject_Free is O(N) where N = # of arenas https://bugs.python.org/issue37029 opened by inada.naoki #37030: Lib/cmd.py: Hide undocumented commands in help and completenam https://bugs.python.org/issue37030 opened by aldwinaldwin #37032: Add CodeType.replace() method https://bugs.python.org/issue37032 opened by vstinner #37034: Argument Clinic omits name of keyword-only parameter on _PyArg https://bugs.python.org/issue37034 opened by vstinner #37035: Don't log OSError exceptions in asyncio transports https://bugs.python.org/issue37035 opened by asvetlov #37036: Iterating a text file by line should not implicitly disable te https://bugs.python.org/issue37036 opened by josh.r #37037: Enable rpath remapping in makefile https://bugs.python.org/issue37037 opened by steve.dower Most recent 15 issues with no replies (15) ========================================== #37037: Enable rpath remapping in makefile https://bugs.python.org/issue37037 #37034: Argument Clinic omits name of keyword-only parameter on _PyArg https://bugs.python.org/issue37034 #37029: PyObject_Free is O(N) where N = # of arenas https://bugs.python.org/issue37029 #37028: Implement asyncio repl https://bugs.python.org/issue37028 #37027: Return a safe proxy over a socket from get_extra_info('socket' https://bugs.python.org/issue37027 #37026: socketserver: BaseServer.handle_request() infinite loop https://bugs.python.org/issue37026 #37025: Misleading error message "Python failed to load the default ac https://bugs.python.org/issue37025 #37024: SQLite flag in configure due to homebrew not linking sqlite https://bugs.python.org/issue37024 #37018: put http.server on a diet https://bugs.python.org/issue37018 #37013: Fatal Python error in socket.if_indextoname() https://bugs.python.org/issue37013 #37012: Possible crash due to PyType_FromSpecWithBases() https://bugs.python.org/issue37012 #37009: Threading and THREAD_SAFE for AIX https://bugs.python.org/issue37009 #37004: SequenceMatcher.ratio() noncommutativity not well-documented https://bugs.python.org/issue37004 #37002: PyType_FromSpec can't create immutable types https://bugs.python.org/issue37002 #37001: symtable.symtable doesn't accept bytes which leads to a mismat https://bugs.python.org/issue37001 Most recent 15 issues waiting for review (15) ============================================= #37035: Don't log OSError exceptions in asyncio transports https://bugs.python.org/issue37035 #37032: Add CodeType.replace() method https://bugs.python.org/issue37032 #37030: Lib/cmd.py: Hide undocumented commands in help and completenam https://bugs.python.org/issue37030 #37028: Implement asyncio repl https://bugs.python.org/issue37028 #37027: Return a safe proxy over a socket from get_extra_info('socket' https://bugs.python.org/issue37027 #37023: test_gdb failed on AMD64 Debian PGO 3.x https://bugs.python.org/issue37023 #37021: Can _random.getrandbits() be converted to METH_FASTCALL? https://bugs.python.org/issue37021 #37017: Use LOAD_METHOD optimization in CallMethod C API functions https://bugs.python.org/issue37017 #37014: [First easy issue] fileinput module should document that openh https://bugs.python.org/issue37014 #37013: Fatal Python error in socket.if_indextoname() https://bugs.python.org/issue37013 #37012: Possible crash due to PyType_FromSpecWithBases() https://bugs.python.org/issue37012 #37011: pdb: restore original tracing function instead of sys.settrace https://bugs.python.org/issue37011 #37007: Implement socket.if_{nametoindex,indextoname} for Windows https://bugs.python.org/issue37007 #37004: SequenceMatcher.ratio() noncommutativity not well-documented https://bugs.python.org/issue37004 #37003: ast unparse does not support f-string new debug format. https://bugs.python.org/issue37003 Top 10 most discussed issues (10) ================================= #36906: Compile time textwrap.dedent() equivalent for str or bytes lit https://bugs.python.org/issue36906 14 msgs #36763: Implementation of the PEP 587 https://bugs.python.org/issue36763 11 msgs #36829: Add sys.unraisablehook() to custom how "unraisable exceptions" https://bugs.python.org/issue36829 11 msgs #36998: distutils sdist command fails to create MANIFEST if any filena https://bugs.python.org/issue36998 10 msgs #28971: nntplib is broken when responses are longer than _MAXLINE https://bugs.python.org/issue28971 9 msgs #36721: Add pkg-config python-3.8-embed and --embed to python3.8-confi https://bugs.python.org/issue36721 9 msgs #37003: ast unparse does not support f-string new debug format. https://bugs.python.org/issue37003 8 msgs #37032: Add CodeType.replace() method https://bugs.python.org/issue37032 8 msgs #1230540: sys.excepthook doesn't work in threads https://bugs.python.org/issue1230540 7 msgs #21315: email._header_value_parser does not recognise in-line encoding https://bugs.python.org/issue21315 6 msgs Issues closed (84) ================== #8138: wsgiref.simple_server.SimpleServer claims to be multithreaded https://bugs.python.org/issue8138 closed by berker.peksag #20285: Improve object.__doc__ and help(object) output https://bugs.python.org/issue20285 closed by cheryl.sabella #22865: Document how to make pty.spawn not copy data https://bugs.python.org/issue22865 closed by vstinner #23378: argparse.add_argument action parameter should allow value exte https://bugs.python.org/issue23378 closed by gvanrossum #23395: _thread.interrupt_main() errors if SIGINT handler in SIG_DFL, https://bugs.python.org/issue23395 closed by pitrou #23896: lib2to3 doesn't provide a grammar where exec is a function https://bugs.python.org/issue23896 closed by gvanrossum #24882: ThreadPoolExecutor doesn't reuse threads until #threads == max https://bugs.python.org/issue24882 closed by pitrou #25329: test_json crashes with stack overflow on Windows https://bugs.python.org/issue25329 closed by vstinner #26467: Add async magic method support to unittest.mock.Mock https://bugs.python.org/issue26467 closed by lisroach #27268: Incorrect error message on float('') https://bugs.python.org/issue27268 closed by cheryl.sabella #27737: email.header.Header.encode() crashes with IndexError on spaces https://bugs.python.org/issue27737 closed by r.david.murray #29183: Unintuitive error handling in wsgiref when a crash happens in https://bugs.python.org/issue29183 closed by berker.peksag #31850: test_nntplib failed with "nntplib.NNTPDataError: line too long https://bugs.python.org/issue31850 closed by vstinner #33110: Adding a done callback to a concurrent.futures Future once it https://bugs.python.org/issue33110 closed by pitrou #33164: Blake 2 module update https://bugs.python.org/issue33164 closed by inada.naoki #33482: codecs.StreamRecoder.writelines is broken https://bugs.python.org/issue33482 closed by gvanrossum #33519: Should MutableSequence provide .copy()? https://bugs.python.org/issue33519 closed by cheryl.sabella #33524: non-ascii characters in headers causes TypeError on email.poli https://bugs.python.org/issue33524 closed by cheryl.sabella #33563: fileinput input's and Fileinput's bufsize=0 do not remit depre https://bugs.python.org/issue33563 closed by mbussonn #34580: sqlite doc: clarify the scope of the context manager https://bugs.python.org/issue34580 closed by berker.peksag #34616: implement "Async exec" https://bugs.python.org/issue34616 closed by yselivanov #35091: Objects/listobject.c: gallop functions rely on signed integer https://bugs.python.org/issue35091 closed by cheryl.sabella #35252: test_functools dead code after FIXME https://bugs.python.org/issue35252 closed by lys.nikolaou #35563: Doc: warnings.rst - add links to references https://bugs.python.org/issue35563 closed by cheryl.sabella #35721: _UnixSubprocessTransport leaks socket pair if Popen fails https://bugs.python.org/issue35721 closed by asvetlov #35810: Object Initialization does not incref Heap-allocated Types https://bugs.python.org/issue35810 closed by petr.viktorin #35924: curses segfault resizing window https://bugs.python.org/issue35924 closed by Josiah Ulfers #35926: Need openssl 1.1.1 support on Windows for ARM and ARM64 https://bugs.python.org/issue35926 closed by steve.dower #35933: python doc does not say that the state kwarg in Pickler.save_r https://bugs.python.org/issue35933 closed by pierreglaser #35990: ipaddress.IPv4Interface won't accept 2-tuple (address, mask) https://bugs.python.org/issue35990 closed by SilentGhost #36035: pathlib.Path().rglob() breaks with broken symlinks https://bugs.python.org/issue36035 closed by pitrou #36045: builtins.help function is not much help with async functions https://bugs.python.org/issue36045 closed by asvetlov #36368: server process of shared_memory shuts down if KeyboardInterrup https://bugs.python.org/issue36368 closed by pierreglaser #36567: DOC: manpage directive doesn't create hyperlink https://bugs.python.org/issue36567 closed by berker.peksag #36648: MAP_SHARED isn't proper for anonymous mappings for VxWorks https://bugs.python.org/issue36648 closed by vstinner #36713: duplicate method definition in Lib/ctypes/test/test_unicode.py https://bugs.python.org/issue36713 closed by ezio.melotti #36755: [2.7] test_httplib leaked [8, 8, 8] references with OpenSSL 1. https://bugs.python.org/issue36755 closed by benjamin.peterson #36797: Cull more oudated distutils information https://bugs.python.org/issue36797 closed by ncoghlan #36810: Recursive type annotations do not work in documentation tests https://bugs.python.org/issue36810 closed by levkivskyi #36826: ast_unparser.c doesn't handle := expressions https://bugs.python.org/issue36826 closed by pablogsal #36867: Make semaphore_tracker track other system resources https://bugs.python.org/issue36867 closed by vstinner #36878: ast.parse with type_comments=True should allow extra text afte https://bugs.python.org/issue36878 closed by gvanrossum #36887: Add integer square root, math.isqrt https://bugs.python.org/issue36887 closed by mark.dickinson #36888: Create a way to check that the parent process is alive for dea https://bugs.python.org/issue36888 closed by pitrou #36907: Crash due to borrowed references in _PyStack_UnpackDict() https://bugs.python.org/issue36907 closed by petr.viktorin #36908: "This module is always available" is confusing https://bugs.python.org/issue36908 closed by Mariatta #36921: Deprecate yield from and @coroutine in asyncio https://bugs.python.org/issue36921 closed by asvetlov #36929: Other Python _io implementations may not expose _io in their t https://bugs.python.org/issue36929 closed by vstinner #36932: asyncio-task.rst could use proper deprecated-removed directive https://bugs.python.org/issue36932 closed by asvetlov #36936: CALL_FUNCTION_KW opcode: keyword names must be non-empty https://bugs.python.org/issue36936 closed by petr.viktorin #36937: New _PyObject_MakeTpCall() function https://bugs.python.org/issue36937 closed by jdemeyer #36945: Add _PyPreConfig.configure_locale: allow to leave LC_CTYPE unc https://bugs.python.org/issue36945 closed by vstinner #36948: NameError in urllib.request.URLopener.retrieve https://bugs.python.org/issue36948 closed by berker.peksag #36949: WeakSet.__repr__ and __str__ do not show contents of the set https://bugs.python.org/issue36949 closed by steve.dower #36951: Wrong types for PyMemberDefs in Objects/typeobject.c https://bugs.python.org/issue36951 closed by benjamin.peterson #36952: fileinput input's and Fileinput's bufsize=0 marked for removal https://bugs.python.org/issue36952 closed by inada.naoki #36955: Python3 - mulltiprocessing https://bugs.python.org/issue36955 closed by SilentGhost #36957: Speed up math.isqrt https://bugs.python.org/issue36957 closed by mark.dickinson #36958: IDLE should print exit message or status if one is provided https://bugs.python.org/issue36958 closed by terry.reedy #36961: ast_unparser.c doesn't handle PEP570 https://bugs.python.org/issue36961 closed by pablogsal #36962: Cant sort ModuleInfo instances https://bugs.python.org/issue36962 closed by eric.smith #36963: PyDict_GetItem SegFaults on simple dictionary lookup when usin https://bugs.python.org/issue36963 closed by larry #36964: `python3 -m venv NAME`: virtualenv is not portable https://bugs.python.org/issue36964 closed by brett.cannon #36965: use of STATUS_CONTROL_C_EXIT in Modules/main.c breaks compilat https://bugs.python.org/issue36965 closed by vstinner #36966: Export __VENV_PROMPT__ as environment variable https://bugs.python.org/issue36966 closed by lys.nikolaou #36970: Rename _PyObject_FastCallKeywords https://bugs.python.org/issue36970 closed by jdemeyer #36972: Add SupportsIndex https://bugs.python.org/issue36972 closed by levkivskyi #36975: csv: undocumented UnicodeDecodeError on malformed file https://bugs.python.org/issue36975 closed by brett.cannon #36978: `python3 -m pip install` has no `--requirement` option on Wind https://bugs.python.org/issue36978 closed by eric.smith #36980: pass-by-reference clues https://bugs.python.org/issue36980 closed by steven.daprano #36981: asyncio transport.write() memory out https://bugs.python.org/issue36981 closed by viocal #36987: Dictionary: why is the value not used up? https://bugs.python.org/issue36987 closed by matrixise #36991: zipfile: AttributeError on extract https://bugs.python.org/issue36991 closed by matrixise #36992: zipfile: AttributeError on extract (LZMA) https://bugs.python.org/issue36992 closed by matrixise #36993: zipfile: tuple IndexError on extract https://bugs.python.org/issue36993 closed by matrixise #36994: Missing tests for CALL_FUNCTION_EX opcode profiling method_des https://bugs.python.org/issue36994 closed by jdemeyer #36995: tp_print -> tp_vectorcall_offset https://bugs.python.org/issue36995 closed by jdemeyer #37000: _randbelow_with_getrandbits function inefficient with powers o https://bugs.python.org/issue37000 closed by rhettinger #37008: make unittest.mock.mock_open honor next() https://bugs.python.org/issue37008 closed by asvetlov #37016: Python embedded in C++ cannot open a file https://bugs.python.org/issue37016 closed by matrixise #37019: Create symlinks relative to cwd https://bugs.python.org/issue37019 closed by serhiy.storchaka #37020: Invalid floating point multiplication result https://bugs.python.org/issue37020 closed by SilentGhost #37031: signalmodule.c: reuse runtime->main_thread https://bugs.python.org/issue37031 closed by vstinner #37033: Dictionary defaults .get(), .pop(), .setdefault() https://bugs.python.org/issue37033 closed by SilentGhost From greg at krypto.org Fri May 24 14:23:43 2019 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 24 May 2019 11:23:43 -0700 Subject: [Python-Dev] [python-committers] PEP 595: Improving bugs.python.org In-Reply-To: References: Message-ID: -cc: committers to avoid crossposting. I have feedback for roundup as experienced on BPO that should be represented within PEP 595 if we are going to have a summary of "improving roundup for BPO" captured in a PEP (presumably already rejected given 581? But good to have documented regardless so _thank you_ for doing this writeup even though I realize our plan of record may be demoralizing for you). > **Flexible UI.** While Roundup UI might look dated, it is convenient and flexible. I wholly disagree with this statement. The BPO roundup UI drives me nuts. every. single. time. I have to use it. It is not optimized for common workflows users actually need to accomplish when using a bug tracker. Two example usability issues (of many): Users can't read the latest update to a bug of length because it is hidden within the *middle* of the scrolling region, they must hunt for it. After reading, the text box to add to the discussion is oddly located near the *top* of the scrolling region so that a user cannot see context of a bug discussion they are responding to as they type. I file things like this under "User experience design is needed for the common workflows of all classes of users". Roundup needs a modern responsive web interface, not GET/POST request based interface seen on BPO. As a result of that, roundup *feels* like is belongs in the Pre-2004 era interface wise by being web form and full page reload server for everything. A responsive modern "async javascript requests happen in the background of the UI" system that we all expect of any web UI is needed. Not just tweaking the existing thing to have a mobile friendly version of the web form. This includes persistent connections so that updates to an issue show up live as they happen instead of getting an error message "someone/something else has updated this issue since you started typing, the action you wanted to take such as submitting that comment or editing that field is now invalid and cannot be completed without a lot of manual work figuring out what happened, cut and pasting, and fixing things up on the you the users part". I'm not going to try proposing a PR to this PEP encapsulating that, I'll leave that up to anyone willing to wrangle such a PEP. The list archive has it regardless now. :) -Greg On Thu, May 23, 2019 at 1:17 PM Ezio Melotti wrote: > Hello, > Berker and I have been working on a PEP that suggests we keep using > and improving bugs.python.org and Roundup instead of switching to > GitHub Issues as proposed by PEP 581. > > The PEP covers: > * What are the advantages of Roundup over GitHub issues; > * What features are missing in Roundup and how can we add them; > * Issues with PEP 581; > * Issues with the migration plan proposed by PEP 588; > > The rendered version of PEP 595 is available at > https://www.python.org/dev/peps/pep-0595/ > > For reference, you can consult PEP 581 and 588 at > https://www.python.org/dev/peps/pep-0581/ and > https://www.python.org/dev/peps/pep-0588/ > > The full text of the PEP is include below. We are planning to update > the PEP to include the feedback we receive and to update the status of > features as we implement them (we also have a Google Summer of Code > students working on it). > > Best Regards, > Ezio Melotti > > > ================ > PEP: 595 > Title: Improving bugs.python.org > Author: Ezio Melotti , Berker Peksag > > Status: Draft > Type: Process > Content-Type: text/x-rst > Created: 12-May-2019 > > > Abstract > ======== > > This PEP proposes a list of improvements to make bugs.python.org > more usable for contributors and core developers. This PEP also > discusses why remaining on Roundup should be preferred over > switching to GitHub Issues, as proposed by :pep:`581`. > > > Motivation > ========== > > On May 14th, 2019 :pep:`581` has been accepted [#]_ without much > public discussion and without a clear consensus [#]_. The PEP > contains factual errors and doesn't address some of the > issues that the migration to GitHub Issues might present. > > Given the scope of the migration, the amount of work required, > and how it will negatively affect the workflow during the > transition phase, this decision should be re-evaluated. > > > Roundup advantages over GitHub Issues > ===================================== > > This section discusses reasons why Roundup should be preferred > over GitHub Issues and Roundup features that are not available > on GitHub Issues. > > * **Roundup is the status quo.** Roundup has been an integral > part of the CPython workflow for years. It is a stable product > that has been tested and customized to adapt to our needs as the > workflow evolved. > > It is possible to gradually improve it and avoid the disruption > that a switch to a different system would inevitabily bring to > the workflow. > > * **Open-source and Python powered.** Roundup is an open-source > project and is written in Python. By using it and supporting > it, we also support the Python ecosystem. Several features > developed for bpo have also been ported to upstream Roundup > over the years. > > * **Fully customizable.** Roundup can be (and has been) fully > customized to fit our needs. > > * **Finer-grained access control.** Roundup allows the creation > of different roles with different permissions (e.g. create, > view, edit, etc.) for each individual property, and users can > have multiple roles. > > * **Flexible UI.** While Roundup UI might look dated, it is > convenient and flexible. > > For example, on the issue page, each field (e.g. title, type, > versions, status, linked files and PRs, etc.) have appropriate > UI elements (input boxes, dropdowns, tables, etc.) that are > easy to set and also provide a convenient way to get info about > the issue at a glance. The number of fields, their values, and > the UI element they use is also fully customizable. > GitHub only provides labels. > > The issue list page presents the issues in a compact and easy > to read table with separate columns for different fields. For > comparison, Roundup lists 50 issues in a screen, whereas GitHub > takes two screens to shows 25 issues. > > * **Advanced search.** Roundup provides an accurate way to search > and filter by using any combination of issue fields. > It is also possible to customize the number of results and the > fields displayed in the table, and the sorting and grouping > (up to two levels). > > bpo also provides predefined summaries (e.g. "Created by you", > "Assigned to you", etc.) and allows the creation of custom > search queries that can be conveniently accessed from the sidebar. > > * **Nosy list autocomplete.** The nosy list has an autocomplete > feature that suggests maintainers and experts. The suggestions > are automatically updated when the experts index [#]_ changes. > > * **Dependencies and Superseders.** Roundup allows to specify > dependencies that must be addressed before the current issues > can be closed and a superseder issue to easily mark duplicates > [#]_. The list of dependencies can also be used to create > meta-issues that references several other sub-issues [#]_. > > > Improving Roundup > ================= > > This section lists some of the issues mentioned by :pep:`581` > and other desired features and discusses how they can be implemented > by improving Roundup and/or our instance. > > * **REST API support.** A REST API will make integration with other > services and the development of new tools and applications easiers. > > Upstream Roundup now supports a REST API. Updating the tracker will > make the REST API available. > > * **GitHub login support.** This will allow users to login > to bugs.python.org (bpo) without having to create a new account. > It will also solve issues with confirmation emails being marked > as spam, and provide two-factor authentication. > > A patch to add this functionality is already available and is > being integrated at the time of writing [#]_. > > * **Markdown support and message preview and editing.** This feature > will allow the use of Markdown in messages and the ability to > preview the message before the submission and edit it afterward. > > This can be done, but it will take some work. Possible solutions > have been proposed on the roundup-devel mailing list [#]_. > > * **"Remove me from nosy list" button.** Add a button on issue pages > to remove self from the nosy list. > > This feature will be added during GSoC 2019. > > * **Mobile friendly theme.** Current theme of bugs.python.org looks > dated and it doesn't work well with mobile browsers. > > A mobile-friendly theme that is more modern but still familiar > will be added. > > * **Add PR link to BPO emails.** Currently bpo emails don't include > links to the corresponding PRs. > > A patch [#]_ is available to change the content of the bpo emails > from:: > > components: +Tkinter > versions: +Python 3.4 > pull_requests: +42 > > to:: > > components: +Tkinter > versions: +Python 3.4 > pull_request: https://github.com/python/cpython/pull/341 > > * **Python 3 support.** Using Python 3 will make maintenance easier. > > Upstream Roundup now supports Python 3. Updating the tracker will > allow us to switch to Python 3. The instances will need to be > updated as well. > > * **Use upstream Roundup.** We currently use a fork of Roundup with > a few modifications, most notably the GitHub integration. If this > is ported upstream, we can start using upstream Roundup without > having to maintain our fork. > > > PEP 581 issues > ============== > > This section addresses some errors and inaccuracies found in :pep:`581`. > > The "Why GitHub?" section of PEP 581 lists features currently > available on GitHub Issues but not on Roundup. Some of this features > are currently supported: > > * "Ability to reply to issue and pull request conversations via email." > > * Being able to reply by email has been one of the core features of > Roundup since the beginning. It is also possible to create new > issues or close existing ones, set or modify fields, and add > attachments. > > * "Email notifications containing metadata, integrated with Gmail, > allowing systematic filtering of emails." > > * Emails sent by Roundup contains metadata that can be used for > filtering. > > * "Additional privacy, such as offering the user a choice to hide an > email address, while still allowing communication with the user > through @-mentions." > > * Email addresses are hidden by default to users that are not > registered. Registered users can see other users' addresses > because we configured the tracker to show them. It can easily > be changed if desired. Users can still be added to the nosy > list by using their username even if their address is hidden. > > * "Ability to automatically close issues when a PR has been merged." > > * The GitHub integration of Roundup automatically closes issues > when a commit that contains "fixes issue " is merged. > (Alternative spellings such as "closes" or "bug" are also supported.) > See [#]_ for a recent example of this feature. > > * "Support for permalinks, allowing easy quoting and copying & > pasting of source code." > > * Roundup has permalinks for issues, messages, attachments, etc. > In addition, Roundup allows to easily rewrite broken URLs in > messages (e.g. if the code hosting changes). > > * "Core developers, volunteers, and the PSF don't have to maintain the > issue infrastructure/site, giving us more time and resources to focus > on the development of Python." > > * While this is partially true, additional resources are required to > write and maintain bots. > > In some cases, bots are required to workaround GitHub's lack of > features rather than expanding. [#]_ was written > specifically to workaround GitHub's email integration. > > Updating our bots to stay up-to-date with changes in the GitHub API > has also maintenance cost. [#]_ took two days to be fixed. > > In addition, we will still need to maintain Roundup for bpo (even > if it becomes read-only) and for the other trackers > we currently host/maintain (Jython [#]_ and Roundup [#]_). > > The "Issues with Roundup / bpo" section of :pep:`581` lists some issues > that have already been fixed: > > * "The upstream Roundup code is in Mercurial. Without any CI available, > it puts heavy burden on the few existing maintainers in terms of > reviewing, testing, and applying patches." > > * While Roundup uses Mercurial by default, there is a git clone > available on GitHub [#]_. Roundup also has CI available [#]_ [#]_. > > * "There is no REST API available. There is an open issue in Roundup for > adding REST API. Last activity was in 2016." > > * The REST API has been integrated and it's now available in Roundup. > > * "Users email addresses are exposed. There is no option to mask it." > > * Exposing addresses to registered and logged in users was a decision > taken when our instance was set up. > > This has now been changed to make the email addresses hidden for > regular users too (Developers and Coordinators can still see them). > The "Email address"" column from the user listing page [#]_ has been > removed too. > > * "It sends a number of unnecessary emails and notifications, and it is > difficult, if not impossible, to configure." > > * This can be configured. > > * "Creating an account has been a hassle. There have been reports of people > having trouble creating accounts or logging in." > > * The main issue is confirmation emails being marked as spam. Work has > been done to resolve the issue. > > Migration considerations > ======================== > > This section describes issues with the migrations that might not > have been addressed by :pep:`581` and :pep:`588`. > > :pep:`588` suggests to add a button to migrate issues to GitHub > only when someone wants to keep working on them. This approach > has several issues: > > * bpo will need to be updated in order to add a button that, > once pressed, creates a new issue on GitHub, copies over all > the messages, attachments, and creates/adds label for the > existing fields. Permissions will also need to be tweaked > to make individual issues read-only once they are migrated, > and to prevent users to create new accounts. > > * The issues will be split between two trackers; searching issues > will take significant more effort. > > * The conversion from Roundup to GitHub is lossy, unless all > the bpo fields are converted into labels or preserved somewhere > else. > > * bpo converts a number of references into links, including > issue, message, and PR IDs, changeset numbers, legacy SVN > revision numbers, paths to files in the repo, files in > tracebacks (detecting the correct branch), links to devguide > pages and sections [#]_. This happens when messages are > requested so it is possible to create the correct link (e.g. > all the file links used to point to hg.python.org and now > point to GitHub). > > If the links are hardcoded during the migration, it will be > difficult (if not impossible) to change them later. If they > aren't, they will either be lost, or a tool to generate the > links and updating them will need to be written. > > * GitHub doesn't provide a way to set and preserve issue IDs > if they are migrated automatically with the use of a button. > (Some projects managed to preserve the IDs by contaacting > the GitHub staff and migrating the issues *en masse*.) > > * On top of the work and changes required to migrate to GitHub > issues, we will still need to keep running and maintaining > Roundup, for both our instance (read-only) and for the Jython > and Roundup trackers (read-write). > > > In addition to the issues listed in the "Open issues" section of > :pep:`588`, this issues will need to be addressed: > > * GitHub is properietary and there is risk of vendor lock-in. > Their business model might change and they could shut down > altogether. > > * Switching to GitHub Issues will likely increase the number of > invalid reports and increase the triaging effort. This concern > has been raised in the past in a Zulip topic [#]_. > > There have been already cases where people posted comments on > PRs that required moderators to mark them as off-topic or > disruptive, delete them altogether, and even lock the > conversation [#]_. > > * Roundup sends weekly reports to python-dev with a summary that > includes new issues, recent issues with no replies, recent > issues waiting for review, most discussed issues, closed issues, > and deltas for open/closed/total issue counts [#]_. The report > provides an easy way to keep track of the tracker activity and > to make sure that issues that require attention are noticed. > > The data collect by the weekly report is also use to generate > statistics and graphs that can be used to gain new insights [#]_. > > * There are currently two mailing lists where Roundup posts new > tracker issues and all messages respectively: new-bugs-announce > [#]_ and python-bugs-list [#]_. A new system will need to be > developed to preserve this functionality. These MLs offer > additional ways to keep track of the tracker activity. > > > References > ========== > > .. [#] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted > > https://mail.python.org/pipermail/python-dev/2019-May/157399.html > > .. [#] [python-committers] [Python-Dev] PEP 581 (Using GitHub issues > for CPython) is accepted > > > https://mail.python.org/pipermail/python-committers/2019-May/006755.html > > .. [#] Experts Index -- Python Devguide > > https://devguide.python.org/experts/ > > .. [#] An example of superseded issues: > "re.sub() replaces only several matches" > > https://bugs.python.org/issue12078 > > .. [#] An example of meta issue using dependencies to track sub-issues: > "Meta-issue: support of the android platform"" > > https://bugs.python.org/issue26865 > > .. [#] Support logging in with GitHub > > https://github.com/python/bugs.python.org/issues/7 > > .. [#] Re: [Roundup-devel] PEP 581 and Google Summer of Code > > https://sourceforge.net/p/roundup/mailman/message/36667828/ > > .. [#] [Tracker-discuss] [issue624] bpo emails contain useless non-github > pull_request number - users want a link to actual github PR > > https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html > > .. [#] The commit reported in msg342882 closes the issue (see the history > below) > > https://bugs.python.org/issue36951#msg342882 > > .. [#] The cpython-emailer-webhook project > > https://github.com/berkerpeksag/cpython-emailer-webhook > > .. [#] A recent incident caused by GitHub > > https://github.com/python/bedevere/pull/163 > > .. [#] Jython issue tracker > > https://bugs.jython.org/ > > .. [#] Roundup issue tracker > > https://issues.roundup-tracker.org/ > > .. [#] GitHub clone of Roundup > > https://github.com/roundup-tracker/roundup > > .. [#] Travis-CI for Roundup > > https://travis-ci.org/roundup-tracker/roundup) and codecov > > .. [#] Codecov for Roundup > > https://codecov.io/gh/roundup-tracker/roundup/commits > > .. [#] User listing -- Python tracker > > https://bugs.python.org/user?@sort=username > > .. [#] Generating Special Links in a Comment -- Python Devguide > > > https://devguide.python.org/triaging/#generating-special-links-in-a-comment > > .. [#] The New-bugs-announce mailing list > > https://mail.python.org/mailman/listinfo/new-bugs-announce > > .. [#] The Python-bugs-list mailing list > > https://mail.python.org/mailman/listinfo/python-bugs-list > > .. [#] An example of [Python-Dev] Summary of Python tracker Issues > > https://mail.python.org/pipermail/python-dev/2019-May/157483.html > > .. [#] Issues stats -- Python tracker > > https://bugs.python.org/issue?@template=stats > > .. [#] s/n ratio -- Python -- Zulip > > > https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20ratio > > .. [#] For example this and other related PRs: > > https://github.com/python/cpython/pull/9099 > > > Copyright > ========= > > This document has been placed in the public domain. > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > python-committers mailing list > python-committers at python.org > https://mail.python.org/mailman/listinfo/python-committers > Code of Conduct: https://www.python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 24 15:12:17 2019 From: brett at python.org (Brett Cannon) Date: Fri, 24 May 2019 12:12:17 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <20190524004422.GR4221@ando.pearwood.info> References: <27ba695b-3219-a628-acef-4e0af649e2de@python.org> <20190522121237.021cbec4@fsol> <20190524004422.GR4221@ando.pearwood.info> Message-ID: On Thu, May 23, 2019 at 5:45 PM Steven D'Aprano wrote: > On Thu, May 23, 2019 at 02:06:13PM -0700, Brett Cannon wrote: > > On Wed, May 22, 2019 at 1:23 PM Sean Wallitsch < > > sean.wallitsch at dreamworks.com> wrote: > > > > > My apologies for that oversight. My understanding is that many of the > > > methods present in aifc depend heavily on audioop for reading and > writing. > > > > > > > But are people using audioop directly? > > Yes they are. > > > https://old.reddit.com/r/Python/comments/brgl8v/pep_594_removing_dead_batteries_from_the_standard/eodvexl/ > > Shouldn't the PEP be responsable for establishing (as well as any > negative can be proven) that these modules aren't used, rather than > merely assuming that they aren't? > > Of course it is hard to establish that a module isn't being used. Not > all code can be searched for on the Internet, there's huge amounts of > non-open source code written by users that never leaves their computer, > or behind corporate firewalls. > Exactly, hence why we are going through the PEP process on this and not simply deprecating everything outright without public discussion. > > The difficulty of proving this negative requires that we be especially > conservative about removal. At the very least, we should demonstrate > that the package is *an active burden*. > > Its not enough to say "nobody has touched this module for ages" since > stable, working code doesn't rot unless we change the language (or the > std lib!) and break it. > > So far, the PEPs record hasn't been too good: out of the 31 modules > listed in the original draft, the PEP marks four as "keep", leaving 27 > to be deprecated. So far, we've found: > I don't think it's fair to be saying Christian isn't doing "too good" simply because he took a stab at trying to figure out some way to know which modules would make sense to remove and he got it wrong for some of them from some people's perspective. As you pointed out, there's no scientific way to do this ahead of time due to closed source code (e.g. the VFX industry is not publishing all of their asset pipeline tools so there was no way to know ahead of time without asking like we are with the PEP), so he had to start somewhere. And this initial list was discussed at the PyCon US 2018 language summit as a good one to start from so he isn't entirely guessing without some initial support to try this list out. > > - the sound libraries are in heavy use by hobbyists and the professional > audio-visual industry; aifc in particular is a mature, stable library > that isn't break and doesn't need much attention; > > - cgi and cgitb are used by people who don't need a heavyweight HTML > solution (see Reddit); > > - colorsys, fileinput and nntplib (at least) have been added to the > "keep" list; > > - the removal of spwd (and crypt?) has been controversial. > > So far, nearly 30% of the so-called "dead batteries" turn out to be not > so dead after all. > > I may have missed some. Nor have I even talked much about modules which > I personally use occasionally, like binhex, because it's not about *me* > its about the community. As much as I would prefer binhex to remain, if > it is removed I will be disappointed but I'll make do. > I personally think it's about both us and the community. The community can and does ask for stuff all the time, but we have to balance that with what we as a team are capable of handling and in my opinion the stdlib is too big right now for us to maintain appropriately. Plus there's an asymmetric ask here when the community says they want something while we have to keep it going. > > Speaking of binhex, I tried to get a sense of how much of a burden it > actually is. There was a comment by Guido in *2007*: > > > https://github.com/python/cpython/commit/34a042d301d6ab88645046a6dfa6c38265ca4b39 > > "This is the last time I fix binhex. If it breaks again it goes in the > dustbin" > > which of course is Guido's perogative to say Won't Fix. Its now 2019 and > it hasn't broken again, or at least it hasn't broken again sufficiently > to make any difference. Serhey found one issue: > > https://bugs.python.org/issue29566 > > but I'm not sure if that's actually a bug or not. There was a > documentation bug (in the source code) in 2017: > > https://bugs.python.org/issue29557 > > and one enhancement request to clean up the code: > > https://bugs.python.org/issue34063 > > Other than that, Serhey touched binhex.py as part of a mass patch to > almost the whole stdlib to modernise file handling to use "with". > > So by my count, in 12 years since Guido's frustrated comment about > binhex, it has seen: > > - removal of an obsolete and ambiguous comment from the source code; > - addition of a few with blocks to modernise file handling; > - one enhancement request still outstanding; > - one possibly a bug (or maybe not) still outstanding. > > > I may have missed some, but we're talking about one issue per three > years. How is this a maintenance burden? > Then multiply that by however many modules stay in the PEP -- and which we know don't all have such small stats compared to binhex -- and it starts to become death by a thousand paper cuts. Numerically all of that seems small, but all of that required time that we all have a limited amount of (both in the open source contribution sense and in the living-and-breathing sense). My point is none of this is free no matter how minuscule we think the cost is. You can also flip that question around and ask why are there *any* enhancement requests or open issues if maintenance is so easy for binhex? Why has no one signed up for the module in the experts index if it's such a minor load to maintain? My guess is we all have other priorities that we value more, which makes total sense, but that also says to me that other things are taking higher priority. And for me, even ignoring a module still takes effort. For me personally, a module not being in this PEP means we consider it critical to a large portion of users of Python where being "in the box" is an important aspect as well as having us directly maintain the module. If a module is in this PEP it says to me that we consider it a nice-to-have for the community, but not at the cost of the modules not listed, or the CPython interpreter, or the language itself which is the other stuff we are planning to keep maintaining after this PEP. Anyway, in the end we are never all going to agree to what the definition is of what is or is not enough effort in maintenance to warrant keeping something in the stdlib. The best I think we can do is try to reach consensus and then try to figure out what guidelines we want to use for what does belong in the stdlib going forward along with trying to figure out how to maintain what we do choose to keep. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 24 15:19:29 2019 From: brett at python.org (Brett Cannon) Date: Fri, 24 May 2019 12:19:29 -0700 Subject: [Python-Dev] PEP 594 -- Bundling libraries? In-Reply-To: References: Message-ID: On Fri, May 24, 2019 at 12:20 AM Inada Naoki wrote: > When removing libraries from stdlib, can we bundle > removed libraries and install it like ensurepip does? > I think that would require people picking those modules up and maintaining them. But even then I don't know how easy it would be to communicate that those modules are no longer under our care and so please report bugs and feature requests to that module's own repository before they open an issue on bpo. -Brett > > Ruby does similar thing, called "Gemification". > See https://rubykaigi.org/2017/presentations/hsbt.html > > When people don't use venv, scripts importing nntplib > or aifc runs correctly. > > When people use venv, they need to `pip install nntplib` > (we may be able to use PyPI's namespace beta feature here). > But I assume `pip install` is not a problem for people using venv. > > Regards, > -- > Inada Naoki > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 24 15:30:54 2019 From: brett at python.org (Brett Cannon) Date: Fri, 24 May 2019 12:30:54 -0700 Subject: [Python-Dev] we will probably be having an difficult discussion about the stdlib after PEP 594 is done In-Reply-To: References: <61B1753E-F700-46DA-9992-C67D09B8268B@python.org> Message-ID: On Fri, May 24, 2019 at 11:07 AM Nathaniel Smith wrote: > On Fri, May 24, 2019, 08:08 Ben Cail wrote: > >> >> Why not have the PSF hire someone (or multiple people) to be paid to >> work on the maintenance burden? This could be similar to the Django >> fellows: >> https://www.djangoproject.com/fundraising/#who-is-the-django-fellow. It >> seems like a good thing for Django, and Python is used by many more >> people than Django. Why not pay someone to do the work that others don't >> want to do? The person in this position could be guided by the PSF >> and/or the Steering Council, to do the work most necessary for the good >> of the language as a whole (like maintaining old modules that other core >> devs don't want to work on). >> >> You could market it together with the maintenance burden: "you want to >> use all these old modules, but we don't want to maintain them. So pay us >> some money, and we'll hire someone to maintain them." >> > > I think the basic idea here is a good one, but: > > - transitioning from an all-volunteer project to a mixed > paid-staff+volunteers project is a big step, and we'll need to take time to > figure out what that would look like before people are comfortable with it. > > - even if we have folks paid to help with maintenance, it won't mean we > suddenly have infinite resources and can do everything. We'll still need to > pick which things to prioritize. And I think if you asked 100 people to > name the most critical issues facing Python today, most of them would not > say "maintaining xdrlib". > I'll just quickly say that the steering council has gotten out from under the PEP backlog enough to start a conversation with the PSF about staffing. It's initially around PM-type things (e.g. Python 2 sunset, PEP 588), but if that works out then I would assume we would expand the conversation to other types of staffing. IOW it's too early to start talking about this topic but just know the steering council is aware of what the DSF does and the general idea of potentially hiring folks to help on the dev-side of things (and Nathaniel's point about setting expectations I think is spot-on :) . -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Fri May 24 16:43:21 2019 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 24 May 2019 13:43:21 -0700 Subject: [Python-Dev] we will probably be having an difficult discussion about the stdlib after PEP 594 is done (was: PEP 594: Removing dead batteries from the standard library) In-Reply-To: References: Message-ID: On Thu, May 23, 2019 at 2:18 PM Brett Cannon wrote: > I'm personally viewing it as a first step in addressing the maintenance burden we have with such a large stdlib. Christian started this work over a year ago and I think it's worth seeing through. After that we should probably have a discussion as a team about how we view the stdlib long-term and how that ties into maintaining it so that people's opinion of the stdlib's quality goes up rather than viewing the quality of it as varying module-to-module. I started a thread on discourse to discuss some "what if" scenarios here, in the hopes it will help us make more informed decisions: https://discuss.python.org/t/1738 -n -- Nathaniel J. Smith -- https://vorpus.org From ezio.melotti at gmail.com Fri May 24 16:48:33 2019 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Fri, 24 May 2019 22:48:33 +0200 Subject: [Python-Dev] [python-committers] PEP 595: Improving bugs.python.org In-Reply-To: References: Message-ID: On Fri, May 24, 2019, 20:23 Gregory P. Smith wrote: > -cc: committers to avoid crossposting. > +1 (I wanted to include committers, since the announcement about PEP 581 was posted there too, but it's better to keep the discussion here) > I have feedback for roundup as experienced on BPO that should be > represented within PEP 595 if we are going to have a summary of "improving > roundup for BPO" captured in a PEP (presumably already rejected given 581? > But good to have documented regardless so _thank you_ for doing this > writeup even though I realize our plan of record may be demoralizing for > you). > We would like people to re-evaluate the decision, but if that doesn't happen I think the PEP is still useful, since it provides a fair view of Roundup capabilities and discusses things that we will have to take into account if we proceed with the migration -- that's why we decided to go ahead and write the PEP. > > **Flexible UI.** While Roundup UI might look dated, it is convenient > and flexible. > > I wholly disagree with this statement. > > The BPO roundup UI drives me nuts. every. single. time. I have to use it. > It is not optimized for common workflows users actually need to accomplish > when using a bug tracker. Two example usability issues (of many): Users > can't read the latest update to a bug of length because it is hidden within > the *middle* of the scrolling region, they must hunt for it. > This came up in the past, and two solutions have been proposed already: 1) keyboard shortcuts have been added in the issue page to quickly jump to the first/previous/next/last message and to the response box [0]. They support both mnemonic (f/p/n/l/r) and vim-style (h/k/j/l/i) combinations. You can find a summary table in the left sidebar of the issue page, under help -> keyboard shortcuts. 2) a patch to collapse the history by default (so that the last message was at the end of the page) was proposed and merged [1], but it was reverted after a few days because some devs wanted direct access to the history without having to do an extra click every time to expand it. After reading, the text box to add to the discussion is oddly located near > the *top* of the scrolling region so that a user cannot see context of a > bug discussion they are responding to as they type. > This has also been discussed and different people had different opinion. Some suggested to reverse the order of the messages so that the last message is at the top near the reply box (like Twitter does), but other said it's unnatural to read. Some suggested to put the reply box at the bottom; however if the other fields are left at the top you would have to go back up to set them, and if they are moved down you won't easily see them at the top when you open an existing issue. Another solution is duplicating the fields and response box at the top and bottom. What I mean with *flexible* when I wrote the PEP, is that, unlike GitHub (where they decide), we can customize bpo however we want (as long as we agree on what we want -- we could even have per-user settings if we really want to :) I think I last heard discussion on the position of the response box in 2011 (when shortcuts and history collapsing were discussed). Maybe people didn't care enough about it so they didn't bother bringing it up or they didn't know it could be changed. If people do speak up, we can change bpo/Roundup. I file things like this under "User experience design is needed for the > common workflows of all classes of users". > > Roundup needs a modern responsive web interface, not GET/POST request > based interface seen on BPO. As a result of that, roundup *feels* like > is belongs in the Pre-2004 era interface wise by being web form and full > page reload server for everything. A responsive modern "async javascript > requests happen in the background of the UI" system that we all expect of > any web UI is needed. Not just tweaking the existing thing to have a mobile > friendly version of the web form. This includes persistent connections so > that updates to an issue show up live as they happen instead of getting an > error message "someone/something else has updated this issue since you > started typing, the action you wanted to take such as submitting that > comment or editing that field is now invalid and cannot be completed > without a lot of manual work figuring out what happened, cut and pasting, > and fixing things up on the you the users part". > This is a good point and I think it can be done now that Roundup has a REST API. > I'm not going to try proposing a PR to this PEP encapsulating that, I'll > leave that up to anyone willing to wrangle such a PEP. The list archive > has it regardless now. :) > Thanks a lot for the feedback, I'll update the PEP once I get back to a PC (using mobile now). Best Regards, Ezio Melotti [0]: https://bitbucket.org/python/tracker-cpython/commits/dbe3912f93895cd1c44fd353a0fd5db1b467a075 [1]: https://bitbucket.org/python/tracker-cpython/commits/525d3e387a65c4ada41eaa19608e6e20ecc16b86 > -Greg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Fri May 24 17:14:45 2019 From: greg at krypto.org (Gregory P. Smith) Date: Fri, 24 May 2019 14:14:45 -0700 Subject: [Python-Dev] [python-committers] PEP 595: Improving bugs.python.org In-Reply-To: References: Message-ID: On Fri, May 24, 2019 at 1:48 PM Ezio Melotti wrote: > > On Fri, May 24, 2019, 20:23 Gregory P. Smith wrote: > >> -cc: committers to avoid crossposting. >> > > +1 (I wanted to include committers, since the announcement about PEP 581 > was posted there too, but it's better to keep the discussion here) > > >> I have feedback for roundup as experienced on BPO that should be >> represented within PEP 595 if we are going to have a summary of "improving >> roundup for BPO" captured in a PEP (presumably already rejected given 581? >> But good to have documented regardless so _thank you_ for doing this >> writeup even though I realize our plan of record may be demoralizing for >> you). >> > > We would like people to re-evaluate the decision, but if that doesn't > happen I think the PEP is still useful, since it provides a fair view of > Roundup capabilities and discusses things that we will have to take into > account if we proceed with the migration -- that's why we decided to go > ahead and write the PEP. > > >> > **Flexible UI.** While Roundup UI might look dated, it is convenient >> and flexible. >> >> I wholly disagree with this statement. >> > >> The BPO roundup UI drives me nuts. every. single. time. I have to use >> it. It is not optimized for common workflows users actually need to >> accomplish when using a bug tracker. Two example usability issues (of >> many): Users can't read the latest update to a bug of length because it is >> hidden within the *middle* of the scrolling region, they must hunt for it. >> > > This came up in the past, and two solutions have been proposed already: > 1) keyboard shortcuts have been added in the issue page to quickly jump to > the first/previous/next/last message and to the response box [0]. They > support both mnemonic (f/p/n/l/r) and vim-style (h/k/j/l/i) combinations. > You can find a summary table in the left sidebar of the issue page, under > help -> keyboard shortcuts. > 2) a patch to collapse the history by default (so that the last message > was at the end of the page) was proposed and merged [1], but it was > reverted after a few days because some devs wanted direct access to the > history without having to do an extra click every time to expand it. > > After reading, the text box to add to the discussion is oddly located near >> the *top* of the scrolling region so that a user cannot see context of a >> bug discussion they are responding to as they type. >> > > This has also been discussed and different people had different opinion. > Some suggested to reverse the order of the messages so that the last > message is at the top near the reply box (like Twitter does), but other > said it's unnatural to read. Some suggested to put the reply box at the > bottom; however if the other fields are left at the top you would have to > go back up to set them, and if they are moved down you won't easily see > them at the top when you open an existing issue. Another solution is > duplicating the fields and response box at the top and bottom. > The two other modern bugtracker UIs I use on a regular basis do this by having all such non-conversation metainfo in a persistent top and side bar boxes such that it is always present. Keeping the conversation and metainfo changes listed in order (latest message at the bottom, metainfo deltas interspersed in between messages all ordered/grouped by timestamp, and reply text entry box below that. These two are typically big-screen engineering UIs (github being one of them), if mobile is desired i expect these would effectively wind up as a multi-pane UI. There's a third ticket that I use for non-engineering stuff which does things in reverse order with the text entry up top and messages in reverse chronological order below that. This reversal always annoys me; probably because I spend most of my time in the others so it forces me to do headstands. I don't know if there is a *right* answer between the two approaches, I just know what I prefer. But the common theme is keeping new the update UI elements right next to the most recent entries which is what BPO lacks today. What I mean with *flexible* when I wrote the PEP, is that, unlike GitHub > (where they decide), we can customize bpo however we want (as long as we > agree on what we want -- we could even have per-user settings if we really > want to :) > Definitely true in that sense. Custom things, no matter where, are custom code. diverge from an upstream project in our own fork and maintenance and updates become harder. Author a pile of bots to automate tasks within a standard API or UI (as is working great for our Github PR workflow) and the equivalent maintenance burden falls on keeping the bots working. I think I last heard discussion on the position of the response box in 2011 > (when shortcuts and history collapsing were discussed). Maybe people > didn't care enough about it so they didn't bother bringing it up or they > didn't know it could be changed. If people do speak up, we can change > bpo/Roundup. > > > I file things like this under "User experience design is needed for the >> common workflows of all classes of users". >> >> Roundup needs a modern responsive web interface, not GET/POST request >> based interface seen on BPO. As a result of that, roundup *feels* like >> is belongs in the Pre-2004 era interface wise by being web form and full >> page reload server for everything. A responsive modern "async javascript >> requests happen in the background of the UI" system that we all expect of >> any web UI is needed. Not just tweaking the existing thing to have a mobile >> friendly version of the web form. This includes persistent connections so >> that updates to an issue show up live as they happen instead of getting an >> error message "someone/something else has updated this issue since you >> started typing, the action you wanted to take such as submitting that >> comment or editing that field is now invalid and cannot be completed >> without a lot of manual work figuring out what happened, cut and pasting, >> and fixing things up on the you the users part". >> > > This is a good point and I think it can be done now that Roundup has a > REST API. > Agreed, that seemed likely to be a pre-requisite. I don't know how many other non-bpo roundup users are out there, but even as CPython moves away from roundup, the other users would presumably appreciate such a UI. It'd also be a good educational project for anyone wanting to learn how to construct such things on top of an existing system. -gps > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ezio.melotti at gmail.com Fri May 24 17:59:37 2019 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Fri, 24 May 2019 23:59:37 +0200 Subject: [Python-Dev] [python-committers] PEP 595: Improving bugs.python.org In-Reply-To: References: Message-ID: On Fri, May 24, 2019, 23:14 Gregory P. Smith wrote: > > > On Fri, May 24, 2019 at 1:48 PM Ezio Melotti > wrote: > >> >> On Fri, May 24, 2019, 20:23 Gregory P. Smith wrote: >> >>> -cc: committers to avoid crossposting. >>> >> >> +1 (I wanted to include committers, since the announcement about PEP 581 >> was posted there too, but it's better to keep the discussion here) >> >> >>> I have feedback for roundup as experienced on BPO that should be >>> represented within PEP 595 if we are going to have a summary of "improving >>> roundup for BPO" captured in a PEP (presumably already rejected given 581? >>> But good to have documented regardless so _thank you_ for doing this >>> writeup even though I realize our plan of record may be demoralizing for >>> you). >>> >> >> We would like people to re-evaluate the decision, but if that doesn't >> happen I think the PEP is still useful, since it provides a fair view of >> Roundup capabilities and discusses things that we will have to take into >> account if we proceed with the migration -- that's why we decided to go >> ahead and write the PEP. >> >> >>> > **Flexible UI.** While Roundup UI might look dated, it is convenient >>> and flexible. >>> >>> I wholly disagree with this statement. >>> >> >>> The BPO roundup UI drives me nuts. every. single. time. I have to use >>> it. It is not optimized for common workflows users actually need to >>> accomplish when using a bug tracker. Two example usability issues (of >>> many): Users can't read the latest update to a bug of length because it is >>> hidden within the *middle* of the scrolling region, they must hunt for it. >>> >> >> This came up in the past, and two solutions have been proposed already: >> 1) keyboard shortcuts have been added in the issue page to quickly jump >> to the first/previous/next/last message and to the response box [0]. They >> support both mnemonic (f/p/n/l/r) and vim-style (h/k/j/l/i) combinations. >> You can find a summary table in the left sidebar of the issue page, under >> help -> keyboard shortcuts. >> 2) a patch to collapse the history by default (so that the last message >> was at the end of the page) was proposed and merged [1], but it was >> reverted after a few days because some devs wanted direct access to the >> history without having to do an extra click every time to expand it. >> >> After reading, the text box to add to the discussion is oddly located >>> near the *top* of the scrolling region so that a user cannot see context of >>> a bug discussion they are responding to as they type. >>> >> >> This has also been discussed and different people had different opinion. >> Some suggested to reverse the order of the messages so that the last >> message is at the top near the reply box (like Twitter does), but other >> said it's unnatural to read. Some suggested to put the reply box at the >> bottom; however if the other fields are left at the top you would have to >> go back up to set them, and if they are moved down you won't easily see >> them at the top when you open an existing issue. Another solution is >> duplicating the fields and response box at the top and bottom. >> > > The two other modern bugtracker UIs I use on a regular basis do this by > having all such non-conversation metainfo in a persistent top and side bar > boxes such that it is always present. Keeping the conversation and > metainfo changes listed in order (latest message at the bottom, metainfo > deltas interspersed in between messages all ordered/grouped by timestamp, > and reply text entry box below that. These two are typically big-screen > engineering UIs (github being one of them), if mobile is desired i expect > these would effectively wind up as a multi-pane UI. There's a third ticket > that I use for non-engineering stuff which does things in reverse order > with the text entry up top and messages in reverse chronological order > below that. This reversal always annoys me; probably because I spend most > of my time in the others so it forces me to do headstands. I don't know if > there is a *right* answer between the two approaches, I just know what I > prefer. But the common theme is keeping new the update UI elements right > next to the most recent entries which is what BPO lacks today. > > What I mean with *flexible* when I wrote the PEP, is that, unlike GitHub >> (where they decide), we can customize bpo however we want (as long as we >> agree on what we want -- we could even have per-user settings if we really >> want to :) >> > > Definitely true in that sense. Custom things, no matter where, are custom > code. diverge from an upstream project in our own fork and maintenance and > updates become harder. Author a pile of bots to automate tasks within a > standard API or UI (as is working great for our Github PR workflow) and the > equivalent maintenance burden falls on keeping the bots working. > To clarify, Roundup allows you by design to define instances where you can customize the schema (i.e. classes, fields, permissions), templates, and extensions. They provide a default instance that you can start with, but it is expected that you customize it. In every new release they tell you if anything changed and how to update your instances (so far upgrading it's always been quite straightforward). We have a fork of the Roundup core only because it's faster to implement new features, test them, and then contribute them upstream, but otherwise is not required to run customized instances. That said, your point that custom things, no matter where, need to be maintained still stands. > I think I last heard discussion on the position of the response box in >> 2011 (when shortcuts and history collapsing were discussed). Maybe people >> didn't care enough about it so they didn't bother bringing it up or they >> didn't know it could be changed. If people do speak up, we can change >> bpo/Roundup. >> >> >> I file things like this under "User experience design is needed for the >>> common workflows of all classes of users". >>> >>> Roundup needs a modern responsive web interface, not GET/POST request >>> based interface seen on BPO. As a result of that, roundup *feels* like >>> is belongs in the Pre-2004 era interface wise by being web form and full >>> page reload server for everything. A responsive modern "async javascript >>> requests happen in the background of the UI" system that we all expect of >>> any web UI is needed. Not just tweaking the existing thing to have a mobile >>> friendly version of the web form. This includes persistent connections so >>> that updates to an issue show up live as they happen instead of getting an >>> error message "someone/something else has updated this issue since you >>> started typing, the action you wanted to take such as submitting that >>> comment or editing that field is now invalid and cannot be completed >>> without a lot of manual work figuring out what happened, cut and pasting, >>> and fixing things up on the you the users part". >>> >> >> This is a good point and I think it can be done now that Roundup has a >> REST API. >> > > Agreed, that seemed likely to be a pre-requisite. I don't know how many > other non-bpo roundup users are out there, but even as CPython moves away > from roundup, the other users would presumably appreciate such a UI. It'd > also be a good educational project for anyone wanting to learn how to > construct such things on top of an existing system. > We have a Google Summer of Code student working on it over the summer. If we switch to GitHub issues, upstream Roundup and all other projects using it will still benefit from his work. > -gps > >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Fri May 24 21:04:43 2019 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 24 May 2019 20:04:43 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> <20190523183331.2163dd60@fsol> Message-ID: [Tim] > I'll note that the approach I very briefly sketched before > (restructure the list of arenas to partition it into multiple lists > partitioned by number of free pools) "should make" obmalloc > competitive with malloc here ... But it's also intrusive, breaking up a simple linked list into a mutli-headed beast. That affects all code using it, not just the parts mutating it. So instead I suggest leaving `usable_arenas` entirely alone, but add a vector of "search fingers", say, static struct arenaobject* nfp2lasta[MAX_NUMBER_OF_FREE_POOLS_POSSIBLE + 1]; mapping a count of free pools to (a pointer to) the last arena object in usable_arenas with that number of free pools. Then the search loop in py_malloc_free() can be replaced with a single array lookup.: it leaps directly to where the search loop ends now. For example, if we free a pool in an arena that had 23 free pools, then the arena object's count of free pools has to be increased to 24, and the arena object unlinked from its current position in usable_arenas and inserted immediately after nfp2lasta[23]. Note that usable_arenas is a doubly linked list, so there's no _need_ at all to iterate over it. Each node in the list knows what's immediately before and after it. And since a free pool count can only increase by 1, it's always correct to move the arena immediately after the last arena with the same original free pool count. Key invariants: 1. nfp2lasta[n] is NULL if and only if no arena in usable_arenas has nfreepools == n. 2. nfp2lasta[pa->nfreepools] == pa if and only if pa is the only arena in usable_arenas with that many free pools. So, in the example above, nfp2lasta[23] has to be set to NULL if and only if the arena in question was the only one with 23 free pools (which can be determined cheaply via invariant #2). And nfp2lasta[24] has to be set to point to the arena in question if and only if nfp2lasta[24] is NULL. Tricky bit: if the arena in question is the only one with a given original free pool count, no pointers in arena objects need to be changed at all. Following the sketch above literally would leave you trying to insert it after itself, which wouldn't end well ;-) Anyway, this "feels like a right solution" to me, eliminating all searches with simple (albeit delicate) constant-time code, and requiring code changes only where an arena's number of free pools can change. From tim.peters at gmail.com Fri May 24 21:14:45 2019 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 24 May 2019 20:14:45 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> <20190523183331.2163dd60@fsol> Message-ID: [Tim] > Key invariants: > ... > 2. nfp2lasta[pa->nfreepools] == pa if and only if pa is the only arena > in usable_arenas with that many free pools. Ack! Scratch that. I need a nap :-( In fact if that equality holds, it means that nfp2lasta entry has to change if pa is moved and pa->prevarena has the same count of free pools. So forget about the explanation and just think about the goal - the details will work themselves out ;-) From random832 at fastmail.com Sat May 25 00:09:13 2019 From: random832 at fastmail.com (Random832) Date: Sat, 25 May 2019 00:09:13 -0400 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: Message-ID: <7089a29d-1913-4c1c-92e7-0a7b7a26d8eb@www.fastmail.com> On Thu, May 23, 2019, at 15:27, Steve Holden wrote: > Besides which, it would be lovely to have a major release that didn't > involve any pain at all for the majority of users! > > Our erstwhile BDFL always eschewed two-digit version identifiers- due > to the possibility for confusion about collating sequence, I beleive.. > We should honour his preferences by going from 3.9 to 4.0. Is 4.0 still gonna install as "python3", pip3, etc, executable as "py -3" on windows? I can see that being a pain point even if nothing else is. From v+python at g.nevcal.com Sat May 25 01:43:23 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 24 May 2019 22:43:23 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <7089a29d-1913-4c1c-92e7-0a7b7a26d8eb@www.fastmail.com> References: <7089a29d-1913-4c1c-92e7-0a7b7a26d8eb@www.fastmail.com> Message-ID: <864b637a-c39c-ecbc-9d6c-189f67edc32e@g.nevcal.com> On 5/24/2019 9:09 PM, Random832 wrote: > On Thu, May 23, 2019, at 15:27, Steve Holden wrote: >> Besides which, it would be lovely to have a major release that didn't >> involve any pain at all for the majority of users! >> >> Our erstwhile BDFL always eschewed two-digit version identifiers- due >> to the possibility for confusion about collating sequence, I beleive.. >> We should honour his preferences by going from 3.9 to 4.0. > Is 4.0 still gonna install as "python3", pip3, etc, executable as "py -3" on windows? I can see that being a pain point even if nothing else is. Maybe by then it can go back to just being python, as 2.x will be past end-of-life. -------------- next part -------------- An HTML attachment was scrubbed... URL: From armin.rigo at gmail.com Sat May 25 03:29:10 2019 From: armin.rigo at gmail.com (Armin Rigo) Date: Sat, 25 May 2019 09:29:10 +0200 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: Hi, On Thu, 23 May 2019 at 17:28, Steve Dower wrote: > On 23May2019 0636, Nick Coghlan wrote: > > However, I think the PR does show that the proposed technique can be > > implemented without too much additional code complexity, and will > > hopefully be adaptable for all implementations that emulate the frame > > API at all. > > Much excitement! > > One of the things I like best about this change is that it actually > makes it *easier* for alternative implementations to use a simpler frame > object without having to emulate CPython semantics (I'd love to get to a > place where bug-for-bug compatibility wasn't required, but this is where > we are right now so *shrug*). Thanks Nick for getting this through! A bient?t, Armin From guido at python.org Sat May 25 10:36:45 2019 From: guido at python.org (Guido van Rossum) Date: Sat, 25 May 2019 07:36:45 -0700 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: This looks great. I only have two nits with the text. First, why is the snapshot called a "dynamic snapshot"? What exactly is dynamic about it? Second, you use the word "mapping" a lot. Would you mind changing that to "mapping object" in most places? Especially in the phrase "each call to ``locals()`` returns the *same* mapping". To me, without the word "object" added, this *could* be interpreted as "a dict with the same key/value pairs" (since "mapping" is also an abstract mathematical concept describing anything that maps keys to values). Other than that, go for it! (Assuming Nathaniel agrees, of course.) --Guido On Tue, May 21, 2019 at 7:54 AM Nick Coghlan wrote: > Hi folks, > > After a couple of years hiatus, I finally got back to working on the > reference implementation for PEP 558, my proposal to resolve some > weird interactions between closures and trace functions in CPython by > formally defining the expected semantics of the locals() builtin at > function scope, and then ensuring that CPython adheres to those > clarified semantics. > > The full text of the PEP is included below, and the rendered version > is available at https://www.python.org/dev/peps/pep-0558/ > > The gist of the PEP is that: > > 1. The behaviour when no trace functions are installed and no frame > introspection APIs are invoked is fine, so nothing should change in > that regard (except that we elevate that behaviour from "the way > CPython happens to work" to "the way the language and library > reference says that Python implementations are supposed to work", and > make sure CPython continues to behave that way even when a trace hook > *is* installed) > 2. If you get hold of a frame object in CPython (or another > implementation that emulates the CPython frame API), whether via a > trace hook or via a frame introspection API, then writing to the > returned mapping will update the actual function locals and/or closure > reference immediately, rather than relying on the > FastToLocals/LocalsToFast APIs > 3. The LocalsToFast C API changes to always produce RuntimeError > (since there's no way for us to make it actually work correctly and > consistently in the presence of closures, and the trace hook > implementation won't need it any more given the write-through proxy > behaviour on frame objects' "f_locals" attribute) > > The reference implementation still isn't quite done yet, but it's far > enough along that I'm happy with the semantics and C API updates > proposed in the current version of the PEP. > > Cheers, > Nick. > > P.S. I'm away this weekend, so I expect the reference implementation > to be done late next week, and I'd be submitting the PEP to Nathaniel > for formal pronouncement at that point. However, I'm posting this > thread now so that there's more time for discussion prior to the 3.8b1 > deadline. > > ============== > PEP: 558 > Title: Defined semantics for locals() > Author: Nick Coghlan > BDFL-Delegate: Nathaniel J. Smith > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 2017-09-08 > Python-Version: 3.8 > Post-History: 2017-09-08, 2019-05-22 > > > Abstract > ======== > > The semantics of the ``locals()`` builtin have historically been > underspecified > and hence implementation dependent. > > This PEP proposes formally standardising on the behaviour of the CPython > 3.6 > reference implementation for most execution scopes, with some adjustments > to the > behaviour at function scope to make it more predictable and independent of > the > presence or absence of tracing functions. > > > Rationale > ========= > > While the precise semantics of the ``locals()`` builtin are nominally > undefined, > in practice, many Python programs depend on it behaving exactly as it > behaves in > CPython (at least when no tracing functions are installed). > > Other implementations such as PyPy are currently replicating that > behaviour, > up to and including replication of local variable mutation bugs that > can arise when a trace hook is installed [1]_. > > While this PEP considers CPython's current behaviour when no trace hooks > are > installed to be acceptable (and largely desirable), it considers the > current > behaviour when trace hooks are installed to be problematic, as it causes > bugs > like [1]_ *without* even reliably enabling the desired functionality of > allowing > debuggers like ``pdb`` to mutate local variables [3]_. > > > Proposal > ======== > > The expected semantics of the ``locals()`` builtin change based on the > current > execution scope. For this purpose, the defined scopes of execution are: > > * module scope: top-level module code, as well as any other code executed > using > ``exec()`` or ``eval()`` with a single namespace > * class scope: code in the body of a ``class`` statement, as well as any > other > code executed using ``exec()`` or ``eval()`` with separate local and > global > namespaces > * function scope: code in the body of a ``def`` or ``async def`` statement > > We also allow interpreters to define two "modes" of execution, with only > the > first mode being considered part of the language specification itself: > > * regular operation: the way the interpreter behaves by default > * tracing mode: the way the interpreter behaves when a trace hook has been > registered in one or more threads via an implementation dependent > mechanism > like ``sys.settrace`` ([4]_) in CPython's ``sys`` module or > ``PyEval_SetTrace`` ([5]_) in CPython's C API > > For regular operation, this PEP proposes elevating the current behaviour of > the CPython reference implementation to become part of the language > specification. > > For tracing mode, this PEP proposes changes to CPython's behaviour at > function > scope that bring the ``locals()`` builtin semantics closer to those used in > regular operation, while also making the related frame API semantics > clearer > and easier for interactive debuggers to rely on. > > The proposed tracing mode changes also affect the semantics of frame object > references obtained through other means, such as via a traceback, or via > the > ``sys._getframe()`` API. > > > New ``locals()`` documentation > ------------------------------ > > The heart of this proposal is to revise the documentation for the > ``locals()`` > builtin to read as follows: > > Return a dictionary representing the current local symbol table, with > variable names as the keys, and their currently bound references as the > values. This will always be the same dictionary for a given runtime > execution frame. > > At module scope, as well as when using ``exec()`` or ``eval()`` with a > single namespace, this function returns the same namespace as > ``globals()``. > > At class scope, it returns the namespace that will be passed to the > metaclass constructor. > > When using ``exec()`` or ``eval()`` with separate local and global > namespaces, it returns the local namespace passed in to the function > call. > > At function scope (including for generators and coroutines), it > returns a > dynamic snapshot of the function's local variables and any nonlocal > cell > references. In this case, changes made via the snapshot are *not* > written > back to the corresponding local variables or nonlocal cell references, > and > any such changes to the snapshot will be overwritten if the snapshot is > subsequently refreshed (e.g. by another call to ``locals()``). > > CPython implementation detail: the dynamic snapshot for the current > frame > will be implicitly refreshed before each call to the trace function > when a > trace function is active. > > For reference, the current documentation of this builtin reads as follows: > > Update and return a dictionary representing the current local symbol > table. > Free variables are returned by locals() when it is called in function > blocks, but not in class blocks. > > Note: The contents of this dictionary should not be modified; changes > may > not affect the values of local and free variables used by the > interpreter. > > (In other words: the status quo is that the semantics and behaviour of > ``locals()`` are currently formally implementation defined, whereas the > proposed > state after this PEP is that the only implementation defined behaviour > will be > that encountered at function scope when a tracing function is defined, > with the > behaviour in all other cases being defined by the language and library > references) > > > Module scope > ------------ > > At module scope, as well as when using ``exec()`` or ``eval()`` with a > single namespace, ``locals()`` must return the same object as > ``globals()``, > which must be the actual execution namespace (available as > ``inspect.currentframe().f_locals`` in implementations that provide access > to frame objects). > > Variable assignments during subsequent code execution in the same scope > must > dynamically change the contents of the returned mapping, and changes to the > returned mapping must change the values bound to local variable names in > the > execution environment. > > The semantics at module scope are required to be the same in both tracing > mode (if provided by the implementation) and in regular operation. > > To capture this expectation as part of the language specification, the > following > paragraph will be added to the documentation for ``locals()``: > > At module scope, as well as when using ``exec()`` or ``eval()`` with a > single namespace, this function returns the same namespace as > ``globals()``. > > This part of the proposal does not require any changes to the reference > implementation - it is standardisation of the current behaviour. > > > Class scope > ----------- > > At class scope, as well as when using ``exec()`` or ``eval()`` with > separate > global and local namespaces, ``locals()`` must return the specified local > namespace (which may be supplied by the metaclass ``__prepare__`` method > in the case of classes). As for module scope, this must be a direct > reference > to the actual execution namespace (available as > ``inspect.currentframe().f_locals`` in implementations that provide access > to frame objects). > > Variable assignments during subsequent code execution in the same scope > must > change the contents of the returned mapping, and changes to the returned > mapping > must change the values bound to local variable names in the > execution environment. > > The mapping returned by ``locals()`` will *not* be used as the actual class > namespace underlying the defined class (the class creation process will > copy > the contents to a fresh dictionary that is only accessible by going > through the > class machinery). > > For nested classes defined inside a function, any nonlocal cells > referenced from > the class scope are *not* included in the ``locals()`` mapping. > > The semantics at class scope are required to be the same in both tracing > mode (if provided by the implementation) and in regular operation. > > To capture this expectation as part of the language specification, the > following > two paragraphs will be added to the documentation for ``locals()``: > > When using ``exec()`` or ``eval()`` with separate local and global > namespaces, [this function] returns the given local namespace. > > At class scope, it returns the namespace that will be passed to the > metaclass > constructor. > > This part of the proposal does not require any changes to the reference > implementation - it is standardisation of the current behaviour. > > > Function scope > -------------- > > At function scope, interpreter implementations are granted significant > freedom > to optimise local variable access, and hence are NOT required to permit > arbitrary modification of local and nonlocal variable bindings through the > mapping returned from ``locals()``. > > Historically, this leniency has been described in the language > specification > with the words "The contents of this dictionary should not be modified; > changes > may not affect the values of local and free variables used by the > interpreter." > > This PEP proposes to change that text to instead say: > > At function scope (including for generators and coroutines), [this > function] > returns a dynamic snapshot of the function's local variables and any > nonlocal cell references. In this case, changes made via the snapshot > are > *not* written back to the corresponding local variables or nonlocal > cell > references, and any such changes to the snapshot will be overwritten > if the > snapshot is subsequently refreshed (e.g. by another call to > ``locals()``). > > CPython implementation detail: the dynamic snapshot for the currently > executing frame will be implicitly refreshed before each call to the > trace > function when a trace function is active. > > This part of the proposal *does* require changes to the CPython reference > implementation, as while it accurately describes the behaviour in regular > operation, the "write back" strategy currently used to support namespace > changes > from trace functions doesn't comply with it (and also causes the quirky > behavioural problems mentioned in the Rationale). > > > CPython Implementation Changes > ============================== > > The current cause of CPython's tracing mode quirks (both the side effects > from > simply installing a tracing function and the fact that writing values back > to > function locals only works for the specific function being traced) is the > way > that locals mutation support for trace hooks is currently implemented: the > ``PyFrame_LocalsToFast`` function. > > When a trace function is installed, CPython currently does the following > for > function frames (those where the code object uses "fast locals" semantics): > > 1. Calls ``PyFrame_FastToLocals`` to update the dynamic snapshot > 2. Calls the trace hook (with tracing of the hook itself disabled) > 3. Calls ``PyFrame_LocalsToFast`` to capture any changes made to the > dynamic > snapshot > > This approach is problematic for a few different reasons: > > * Even if the trace function doesn't mutate the snapshot, the final step > resets > any cell references back to the state they were in before the trace > function > was called (this is the root cause of the bug report in [1]_) > * If the trace function *does* mutate the snapshot, but then does something > that causes the snapshot to be refreshed, those changes are lost (this is > one aspect of the bug report in [3]_) > * If the trace function attempts to mutate the local variables of a frame > other > than the one being traced (e.g. ``frame.f_back.f_locals``), those changes > will almost certainly be lost (this is another aspect of the bug report > in > [3]_) > * If a ``locals()`` reference is passed to another function, and *that* > function mutates the snapshot namespace, then those changes *may* be > written > back to the execution frame *if* a trace hook is installed > > The proposed resolution to this problem is to take advantage of the fact > that > whereas functions typically access their *own* namespace using the language > defined ``locals()`` builtin, trace functions necessarily use the > implementation > dependent ``frame.f_locals`` interface, as a frame reference is what gets > passed to hook implementations. > > Instead of being a direct reference to the dynamic snapshot returned by > ``locals()``, ``frame.f_locals`` will be updated to instead return a > dedicated > proxy type (implemented as a private subclass of the existing > ``types.MappingProxyType``) that has two internal attributes not exposed as > part of either the Python or public C API: > > * *mapping*: the dynamic snapshot that is returned by the ``locals()`` > builtin > * *frame*: the underlying frame that the snapshot is for > > ``__setitem__`` and ``__delitem__`` operations on the proxy will affect > not only > the dynamic snapshot, but *also* the corresponding fast local or cell > reference > on the underlying frame. > > The ``locals()`` builtin will be made aware of this proxy type, and > continue to > return a reference to the dynamic snapshot rather than to the write-through > proxy. > > At the C API layer, ``PyEval_GetLocals()`` will implement the same > semantics > as the Python level ``locals()`` builtin, and a new > ``PyFrame_GetPyLocals(frame)`` accessor API will be provided to allow the > function level proxy bypass logic to be encapsulated entirely inside the > frame > implementation. > > The C level equivalent of accessing ``pyframe.f_locals`` in Python will be > a > new ``PyFrame_GetLocalsAttr(frame)`` API. Like the Python level > descriptor, the > new API will implicitly refresh the dynamic snapshot at function scope > before > returning a reference to the write-through proxy. > > The ``PyFrame_LocalsToFast()`` function will be changed to always emit > ``RuntimeError``, explaining that it is no longer a supported operation, > and > affected code should be updated to rely on the write-through tracing mode > proxy instead. > > > Design Discussion > ================= > > Ensuring ``locals()`` returns a shared snapshot at function scope > ----------------------------------------------------------------- > > The ``locals()`` builtin is a required part of the language, and in the > reference implementation it has historically returned a mutable mapping > with > the following characteristics: > > * each call to ``locals()`` returns the *same* mapping > * for namespaces where ``locals()`` returns a reference to something other > than > the actual local execution namespace, each call to ``locals()`` updates > the > mapping with the current state of the local variables and any referenced > nonlocal cells > * changes to the returned mapping *usually* aren't written back to the > local variable bindings or the nonlocal cell references, but write backs > can be triggered by doing one of the following: > > * installing a Python level trace hook (write backs then happen whenever > the trace hook is called) > * running a function level wildcard import (requires bytecode > injection in Py3) > * running an ``exec`` statement in the function's scope (Py2 only, since > ``exec`` became an ordinary builtin in Python 3) > > The proposal in this PEP aims to retain the first two properties (to > maintain > backwards compatibility with as much code as possible) while ensuring that > simply installing a trace hook can't enable rebinding of function locals > via > the ``locals()`` builtin (whereas enabling rebinding via > ``frame.f_locals`` inside the tracehook implementation is fully intended). > > > Keeping ``locals()`` as a dynamic snapshot at function scope > ------------------------------------------------------------ > > It would theoretically be possible to change the semantics of the > ``locals()`` > builtin to return the write-through proxy at function scope, rather than > continuing to return a dynamic snapshot. > > This PEP doesn't (and won't) propose this as it's a backwards incompatible > change in practice, even though code that relies on the current behaviour > is > technically operating in an undefined area of the language specification. > > Consider the following code snippet:: > > def example(): > x = 1 > locals()["x"] = 2 > print(x) > > Even with a trace hook installed, that function will consistently print > ``1`` > on the current reference interpreter implementation:: > > >>> example() > 1 > >>> import sys > >>> def basic_hook(*args): > ... return basic_hook > ... > >>> sys.settrace(basic_hook) > >>> example() > 1 > > Similarly, ``locals()`` can be passed to the ``exec()`` and ``eval()`` > builtins > at function scope without risking unexpected rebinding of local variables. > > Provoking the reference interpreter into incorrectly mutating the local > variable > state requires a more complex setup where a nested function closes over a > variable being rebound in the outer function, and due to the use of either > threads, generators, or coroutines, it's possible for a trace function to > start > running for the nested function before the rebinding operation in the outer > function, but finish running after the rebinding operation has taken place > (in > which case the rebinding will be reverted, which is the bug reported in > [1]_). > > In addition to preserving the de facto semantics which have been in place > since > PEP 227 introduced nested scopes in Python 2.1, the other benefit of > restricting > the write-through proxy support to the implementation-defined frame object > API > is that it means that only interpreter implementations which emulate the > full > frame API need to offer the write-through capability at all, and that > JIT-compiled implementations only need to enable it when a frame > introspection > API is invoked, or a trace hook is installed, not whenever ``locals()`` is > accessed at function scope. > > > What happens with the default args for ``eval()`` and ``exec()``? > ----------------------------------------------------------------- > > These are formally defined as inheriting ``globals()`` and ``locals()`` > from > the calling scope by default. > > There isn't any need for the PEP to change these defaults, so it doesn't. > > > Changing the frame API semantics in regular operation > ----------------------------------------------------- > > Earlier versions of this PEP proposed having the semantics of the frame > ``f_locals`` attribute depend on whether or not a tracing hook was > currently > installed - only providing the write-through proxy behaviour when a > tracing hook > was active, and otherwise behaving the same as the ``locals()`` builtin. > > That was adopted as the original design proposal for a couple of key > reasons, > one pragmatic and one more philosophical: > > * Object allocations and method wrappers aren't free, and tracing functions > aren't the only operations that access frame locals from outside the > function. > Restricting the changes to tracing mode meant that the additional memory > and > execution time overhead of these changes would as close to zero in > regular > operation as we can possibly make them. > * "Don't change what isn't broken": the current tracing mode problems are > caused > by a requirement that's specific to tracing mode (support for external > rebinding of function local variable references), so it made sense to > also > restrict any related fixes to tracing mode > > However, actually attempting to implement and document that dynamic > approach > highlighted the fact that it makes for a really subtle runtime state > dependent > behaviour distinction in how ``frame.f_locals`` works, and creates several > new edge cases around how ``f_locals`` behaves as trace functions are added > and removed. > > Accordingly, the design was switched to the current one, where > ``frame.f_locals`` is always a write-through proxy, and ``locals()`` is > always > a dynamic snapshot, which is both simpler to implement and easier to > explain. > > Regardless of how the CPython reference implementation chooses to handle > this, > optimising compilers and interpreters also remain free to impose additional > restrictions on debuggers, by making local variable mutation through frame > objects an opt-in behaviour that may disable some optimisations (just as > the > emulation of CPython's frame API is already an opt-in flag in some Python > implementations). > > > Historical semantics at function scope > -------------------------------------- > > The current semantics of mutating ``locals()`` and ``frame.f_locals`` in > CPython > are rather quirky due to historical implementation details: > > * actual execution uses the fast locals array for local variable bindings > and > cell references for nonlocal variables > * there's a ``PyFrame_FastToLocals`` operation that populates the frame's > ``f_locals`` attribute based on the current state of the fast locals > array > and any referenced cells. This exists for three reasons: > > * allowing trace functions to read the state of local variables > * allowing traceback processors to read the state of local variables > * allowing ``locals()`` to read the state of local variables > * a direct reference to ``frame.f_locals`` is returned from ``locals()``, > so if > you hand out multiple concurrent references, then all those references > will be > to the exact same dictionary > * the two common calls to the reverse operation, ``PyFrame_LocalsToFast``, > were > removed in the migration to Python 3: ``exec`` is no longer a statement > (and > hence can no longer affect function local namespaces), and the compiler > now > disallows the use of ``from module import *`` operations at function > scope > * however, two obscure calling paths remain: ``PyFrame_LocalsToFast`` is > called > as part of returning from a trace function (which allows debuggers to > make > changes to the local variable state), and you can also still inject the > ``IMPORT_STAR`` opcode when creating a function directly from a code > object > rather than via the compiler > > This proposal deliberately *doesn't* formalise these semantics as is, > since they > only make sense in terms of the historical evolution of the language and > the > reference implementation, rather than being deliberately designed. > > > Implementation > ============== > > The reference implementation update is in development as a draft pull > request on GitHub ([6]_). > > > Acknowledgements > ================ > > Thanks to Nathaniel J. Smith for proposing the write-through proxy idea in > [1]_ and pointing out some critical design flaws in earlier iterations of > the > PEP that attempted to avoid introducing such a proxy. > > > References > ========== > > .. [1] Broken local variable assignment given threads + trace hook + > closure > (https://bugs.python.org/issue30744) > > .. [2] Clarify the required behaviour of ``locals()`` > (https://bugs.python.org/issue17960) > > .. [3] Updating function local variables from pdb is unreliable > (https://bugs.python.org/issue9633) > > .. [4] CPython's Python API for installing trace hooks > (https://docs.python.org/dev/library/sys.html#sys.settrace) > > .. [5] CPython's C API for installing trace hooks > (https://docs.python.org/3/c-api/init.html#c.PyEval_SetTrace) > > .. [6] PEP 558 reference implementation > (https://github.com/python/cpython/pull/3640/files) > > > Copyright > ========= > > This document has been placed in the public domain. > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat May 25 14:12:14 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 25 May 2019 14:12:14 -0400 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: On 5/25/2019 10:36 AM, Guido van Rossum wrote: > This looks great. I agree. I understand and have tried to explain normal operation multiple times. The proposed new doc looks better than anything I ever wrote. (I never even thought about locals() with tracing on.) The improved clarity well justifies the increased space. > I only have two nits with the text. > > First, why is the snapshot called a "dynamic snapshot"? What exactly is > dynamic about it? Good catch. 'snapshot' is from "At function scope (including for generators and coroutines), [locals()] returns a dynamic snapshot of the function's local variables and any nonlocal cell references." 'Dynamic' could be misunderstood to mean the function locals() snapshot always tracks the underlying function namespace. The point of using the 'snapshot' metaphor is that it does not. The snapshot is 'dynamic' in that it can be changed, but the same is true of real photos. But just as with real photos, changing the snapshot does not change the reality it represents (as explained in the next proposed sentence). The 'snapshot' metaphor does not need 'dynamic' and I think it works better without it. > Second, you use the word "mapping" a lot. Would you mind changing that > to "mapping object" in most places? Especially in the phrase "each call > to ``locals()`` returns the *same* mapping". To me, without the word > "object" added, this *could* be interpreted as "a dict with the same > key/value pairs" (since "mapping" is also an abstract mathematical > concept describing anything that maps keys to values). Agreed also. -- Terry Jan Reedy From brett at python.org Sat May 25 13:05:11 2019 From: brett at python.org (Brett Cannon) Date: Sat, 25 May 2019 10:05:11 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <864b637a-c39c-ecbc-9d6c-189f67edc32e@g.nevcal.com> References: <7089a29d-1913-4c1c-92e7-0a7b7a26d8eb@www.fastmail.com> <864b637a-c39c-ecbc-9d6c-189f67edc32e@g.nevcal.com> Message-ID: Please start a new thread if you want to continue discussing what will come after Python 3.9. On Fri., May 24, 2019, 22:45 Glenn Linderman, wrote: > On 5/24/2019 9:09 PM, Random832 wrote: > > On Thu, May 23, 2019, at 15:27, Steve Holden wrote: > > Besides which, it would be lovely to have a major release that didn't > involve any pain at all for the majority of users! > > Our erstwhile BDFL always eschewed two-digit version identifiers- due > to the possibility for confusion about collating sequence, I beleive.. > We should honour his preferences by going from 3.9 to 4.0. > > > Is 4.0 still gonna install as "python3", pip3, etc, executable as "py -3" on windows? I can see that being a pain point even if nothing else is. > > > Maybe by then it can go back to just being python, as 2.x will be past > end-of-life. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat May 25 17:09:20 2019 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 25 May 2019 14:09:20 -0700 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: On Sat, May 25, 2019, 07:38 Guido van Rossum wrote: > This looks great. > > I only have two nits with the text. > > First, why is the snapshot called a "dynamic snapshot"? What exactly is > dynamic about it? > It's dynamic in that it can spontaneously change when certain other events happen. For example, imagine this code runs at function scope: # take a snapshot a = locals() # it's a snapshot, so it doesn't include the new variable assert "a" not in a # take another snapshot b = locals() # now our first "snapshot" has changed assert "a" in a Overall I'm happy with the PEP, but I'm still a bit uneasy about whether we've gotten the details of this "dynamicity" exactly right, esp. since the PEP promotes them from implementation detail to language features. There are a lot of complicated tradeoffs so I'm working on a longer response that tries to lay out all the options and hopefully convince myself (and everyone else). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat May 25 17:26:35 2019 From: guido at python.org (Guido van Rossum) Date: Sat, 25 May 2019 14:26:35 -0700 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: That's a good fine point that the PEP could call out, but just adding "dynamic" in front of "snapshot" everywhere doesn't tell me any of that. Given that the code calling locals() must of necessity live in the same function body (except for the special case of trace functions), I don't think that what you describe here is too worrisome a scenario. On Sat, May 25, 2019 at 2:09 PM Nathaniel Smith wrote: > On Sat, May 25, 2019, 07:38 Guido van Rossum wrote: > >> This looks great. >> >> I only have two nits with the text. >> >> First, why is the snapshot called a "dynamic snapshot"? What exactly is >> dynamic about it? >> > > It's dynamic in that it can spontaneously change when certain other events > happen. For example, imagine this code runs at function scope: > > # take a snapshot > a = locals() > > # it's a snapshot, so it doesn't include the new variable > assert "a" not in a > > # take another snapshot > b = locals() > > # now our first "snapshot" has changed > assert "a" in a > > Overall I'm happy with the PEP, but I'm still a bit uneasy about whether > we've gotten the details of this "dynamicity" exactly right, esp. since the > PEP promotes them from implementation detail to language features. There > are a lot of complicated tradeoffs so I'm working on a longer response that > tries to lay out all the options and hopefully convince myself (and > everyone else). > > -n > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From Richard at Damon-Family.org Sat May 25 18:37:22 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Sat, 25 May 2019 18:37:22 -0400 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: On 5/25/19 5:09 PM, Nathaniel Smith wrote: > On Sat, May 25, 2019, 07:38 Guido van Rossum > wrote: > > This looks great. > > I only have two nits with the text. > > First, why is the snapshot called a "dynamic snapshot"? What > exactly is dynamic about it? > > > It's dynamic in that it can spontaneously change when certain other > events happen. For example, imagine this code runs at function scope: > > # take a snapshot > a = locals() > > # it's a snapshot, so it doesn't include the new variable > assert "a" not in a > > # take another snapshot > b = locals() > > # now our first "snapshot" has changed > assert "a" in a > > Overall I'm happy with the PEP, but I'm still a bit uneasy about > whether we've gotten the details of this "dynamicity" exactly right, > esp. since the PEP promotes them from implementation detail to > language features. There are a lot of complicated tradeoffs so I'm > working on a longer response that tries to lay out all the options and > hopefully convince myself (and everyone else). > > -n To me that is a static snapshot of a dynamic environment, not a dynamic snapshot. The snapshot you get at THAT moment in time won't change, as time progresses, so that snapshot itself isn't dynamic. Calling something a 'dynamic snapshot' could be take to imply that the snapshot itself is dynamic, and thus changes at that environment changes (and you could pass that snapshot to some other place, and they could get a view of things just like you would see it there, -- Richard Damon From rosuav at gmail.com Sat May 25 18:44:33 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 26 May 2019 08:44:33 +1000 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: On Sun, May 26, 2019 at 8:38 AM Richard Damon wrote: > > On 5/25/19 5:09 PM, Nathaniel Smith wrote: > > a = locals() > > b = locals() > > # now our first "snapshot" has changed > > assert "a" in a > > > To me that is a static snapshot of a dynamic environment, not a dynamic > snapshot. The snapshot you get at THAT moment in time won't change, as > time progresses, so that snapshot itself isn't dynamic. Except that it does. After calling locals() a second time, the result of the *first* call will be updated to reflect changes. It's not a frozen snapshot; you can't diff the two dictionaries to see what's changed (unless you use "a = dict(locals())" or something, which IS a snapshot). >From my reading of the description, you could also "assert a is b" - is that correct? ChrisA From dfmoisset at gmail.com Sat May 25 22:31:38 2019 From: dfmoisset at gmail.com (Daniel Moisset) Date: Sun, 26 May 2019 03:31:38 +0100 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: Hi, thanks for the work on this proposal, I think this is at least a tip of the iceberg and a good start for the bigger question of how the stdlib should evolve.. I think that the PEP should include an idea of what should happen if existing stdlib pieces depend on this deprecated modules. For example, email.mime.audio is part of a non-deprecated module but it depends on sndhdr which is being planned for deprecation. Should that part of the functionality be deprecated too? rewritten to not depend on the deprecated module (but keep the pieces of relevant code? depend on optional 3rd party modules and degrade if those are not available? The sndhdr case is just an example but I can imagine there are others (optparse is no longer schedule for removal in your PEP, but would have been another case, being used by the profile module). Best, Daniel On Tue, 21 May 2019 at 15:15, Christian Heimes wrote: > Hi, > > I have updated the PEP with feedback from discussions. The highlights are: > > * Deprecate parser module > * Keep fileinput module > * Elaborate why crypt and spwd are dangerous and bad > * Improve sections for cgitb, colorsys, nntplib, and smtpd modules > * The colorsys, crypt, imghdr, sndhdr, and spwd sections now list suitable > substitutions. > * Mention that socketserver is going to stay for http.server and > xmlrpc.server > * The future maintenance section now states that the deprecated modules > may be adopted by Python community members. > > https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff-ae358c21fa7968ee3b6c64479e051574 > > > I'll be traveling the next couple of days and will only have limited > opportunities to respond on feedback. > > Christian > > ------------------------------------------------------------------- > PEP: 594 > Title: Removing dead batteries from the standard library > Author: Christian Heimes > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 20-May-2019 > Post-History: 21-May-2019 > > > Abstract > ======== > > This PEP proposed a list of standard library modules to be removed from the > standard library. The modules are mostly historic data formats and APIs > that > have been superseded a long time ago, e.g. Mac OS 9 and Commodore. > > > Rationale > ========= > > Back in the early days of Python, the interpreter came with a large set of > useful modules. This was often refrained to as "batteries included" > philosophy and was one of the corner stones to Python's success story. > Users didn't have to figure out how to download and install separate > packages in order to write a simple web server or parse email. > > Times have changed. The introduction of the cheese shop (PyPI), setuptools, > and later pip, it became simple and straight forward to download and > install > packages. Nowadays Python has a rich and vibrant ecosystem of third party > packages. It's pretty much standard to either install packages from PyPI or > use one of the many Python or Linux distributions. > > On the other hand, Python's standard library is piling up cruft, > unnecessary > duplication of functionality, and dispensable features. This is undesirable > for several reasons. > > * Any additional module increases the maintenance cost for the Python core > development team. The team has limited resources, reduced maintenance > cost > frees development time for other improvements. > * Modules in the standard library are generally favored and seen as the > de-facto solution for a problem. A majority of users only pick 3rd party > modules to replace a stdlib module, when they have a compelling reason, > e.g. > lxml instead of `xml`. The removal of an unmaintained stdlib module > increases the chances of a community contributed module to become widely > used. > * A lean and mean standard library benefits platforms with limited > resources > like devices with just a few hundred kilobyte of storage (e.g. BBC > Micro:bit). Python on mobile platforms like BeeWare or WebAssembly > (e.g. pyodide) also benefit from reduced download size. > > The modules in the PEP have been selected for deprecation because their > removal is either least controversial or most beneficial. For example > least controversial are 30 years old multimedia formats like ``sunau`` > audio format, which was used on SPARC and NeXT workstations in the late > 1980ties. The ``crypt`` module has fundamental flaws that are better solved > outside the standard library. > > This PEP also designates some modules as not scheduled for removal. Some > modules have been deprecated for several releases or seem unnecessary at > first glance. However it is beneficial to keep the modules in the standard > library, mostly for environments where installing a package from PyPI is > not > an option. This can be cooperate environments or class rooms where external > code is not permitted without legal approval. > > * The usage of FTP is declining, but some files are still provided over > the FTP protocol or hosters offer FTP to upload content. Therefore > ``ftplib`` is going to stay. > * The ``optparse`` and ``getopt`` module are widely used. They are mature > modules with very low maintenance overhead. > * According to David Beazley [5]_ the ``wave`` module is easy to teach to > kids and can make crazy sounds. Making a computer generate crazy sounds > is > powerful and highly motivating exercise for a 9yo aspiring developer. > It's > a fun battery to keep. > > > Deprecation schedule > ==================== > > 3.8 > --- > > This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be > released > a few months before Python 2.7 will reach its end of lifetime. We expect > that > Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and > 2020. To reduce churn and to allow a smooth transition from Python 2, > Python 3.8 will neither raise `DeprecationWarning` nor remove any > modules that have been scheduled for removal. Instead deprecated modules > will > just be *documented* as deprecated. Optionally modules may emit a > `PendingDeprecationWarning`. > > All deprecated modules will also undergo a feature freeze. No additional > features should be added. Bug should still be fixed. > > 3.9 > --- > > Starting with Python 3.9, deprecated modules will start issuing > `DeprecationWarning`. The `parser`_ module is removed and potentially > replaced with a new module. > > All other deprecated modules are fully supported and will receive security > updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will > be released about 18 months after 3.8.0 (April 2021?) and most likely > be supported for 5 years after the release. The estimated EOL of Python 3.9 > is in 2026. > > 3.10 > ---- > > In 3.10 all deprecated modules will be removed from the CPython repository > together with tests, documentation, and autoconf rules. > > > PEP acceptance process > ====================== > > 3.8.0b1 is scheduled to be release shortly after the PEP is officially > submitted. Since it's improbable that the PEP will pass all stages of the > PEP process in time, I propose a two step acceptance process that is > analogous Python's two release deprecation process. > > The first *provisionally accepted* phase targets Python 3.8.0b1. In the > first > phase no code is changes or removed. Modules are only documented as > deprecated. The only exception is the `parser`_ module. It has been > documented as deprecated since Python 2.5 and is scheduled for removal for > 3.9 to make place for a more advanced parser. > > The final decision, which modules will be removed and how the removed code > is preserved, can be delayed for another year. > > > Deprecated modules > ================== > > The modules are grouped as data encoding, multimedia, network, OS > interface, > and misc modules. The majority of modules are for old data formats or > old APIs. Some others are rarely useful and have better replacements on > PyPI, e.g. Pillow for image processing or NumPy-based projects to deal with > audio processing. > > .. csv-table:: Table 1: Proposed modules deprecations > :header: "Module", "Deprecated in", "To be removed", "Replacement" > :widths: 1, 1, 1, 2 > > aifc,3.8,3.10,\- > asynchat,3.8,3.10,asyncio > asyncore,3.8,3.10,asyncio > audioop,3.8,3.10,\- > binhex,3.8,3.10,\- > cgi,3.8,3.10,\- > cgitb,3.8,3.10,\- > chunk,3.8,3.10,\- > colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" > crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" > fileinput,\-,**keep**,argparse > formatter,3.4,3.10,\- > fpectl,**3.7**,**3.7**,\- > getopt,**3.2**,**keep**,"argparse, optparse" > imghdr,3.8,3.10,"filetype, puremagic, python-magic" > imp,**3.4**,3.10,importlib > lib2to3,\-,**keep**, > macpath,**3.7**,**3.8**,\- > msilib,3.8,3.10,\- > nntplib,3.8,3.10,\- > nis,3.8,3.10,\- > optparse,\-,**keep**,argparse > ossaudiodev,3.8,3.10,\- > parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" > pipes,3.8,3.10,subprocess > smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd > sndhdr,3.8,3.10,"filetype, puremagic, python-magic" > spwd,3.8,3.10,"python-pam, simplepam" > sunau,3.8,3.10,\- > uu,3.8,3.10,\- > wave,\-,**keep**, > xdrlib,3.8,3.10,\- > > > Data encoding modules > --------------------- > > binhex > ~~~~~~ > > The `binhex `_ module > encodes > and decodes Apple Macintosh binhex4 data. It was originally developed for > TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to > encode binary email attachments. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > uu > ~~ > > The `uu `_ module provides > uuencode format, an old binary encoding format for email from 1980. The uu > format has been replaced by MIME. The uu codec is provided by the binascii > module. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > xdrlib > ~~~~~~ > > The `xdrlib `_ module > supports > the Sun External Data Representation Standard. XDR is an old binary > serialization format from 1987. These days it's rarely used outside > specialized domains like NFS. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Multimedia modules > ------------------ > > aifc > ~~~~ > > The `aifc `_ module provides > support for reading and writing AIFF and AIFF-C files. The Audio > Interchange > File Format is an old audio format from 1988 based on Amiga IFF. It was > most > commonly used on the Apple Macintosh. These days only few specialized > application use AIFF. > > Module type > pure Python (depends on `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > audioop > ~~~~~~~ > > The `audioop `_ module > contains helper functions to manipulate raw audio data and adaptive > differential pulse-code modulated audio data. The module is implemented in > C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ > module depend on `audioop`_ for some operations. The byteswap operation in > the `wave`_ module can be substituted with little work. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > colorsys > ~~~~~~~~ > > The `colorsys `_ module > defines color conversion functions between RGB, YIQ, HSL, and HSV > coordinate > systems. > > The PyPI packages *colormath*, *colour*, and *colorspacious* provide more > and > advanced features. The Pillow library is better suited to transform images > between color systems. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `colormath `_, > `colour `_ > `colorspacious `_, > `Pillow `_ > > chunk > ~~~~~ > > The `chunk `_ module > provides > support for reading and writing Electronic Arts' Interchange File Format. > IFF is an old audio file format originally introduced for Commodore and > Amiga. The format is no longer relevant. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > imghdr > ~~~~~~ > > The `imghdr `_ module is a > simple tool to guess the image file format from the first 32 bytes > of a file or buffer. It supports only a limited amount of formats and > neither returns resolution nor color depth. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `puremagic `_, > `filetype `_, > `python-magic `_ > > ossaudiodev > ~~~~~~~~~~~ > > The `ossaudiodev `_ > module provides support for Open Sound System, an interface to sound > playback and capture devices. OSS was initially free software, but later > support for newer sound devices and improvements were proprietary. Linux > community abandoned OSS in favor of ALSA [1]_. Some operation systems like > OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > sndhdr > ~~~~~~ > > The `sndhdr `_ module is > similar to the `imghdr`_ module but for audio formats. It guesses file > format, channels, frame rate, and sample widths from the first 512 bytes of > a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and > other ancient formats. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `puremagic `_, > `filetype `_, > `python-magic `_ > > sunau > ~~~~~ > > The `sunau `_ module > provides > support for Sun AU sound format. It's yet another old, obsolete file > format. > > Module type > pure Python (depends on `audioop`_ C extension for some operations) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Networking modules > ------------------ > > asynchat > ~~~~~~~~ > > The `asynchat `_ module > is build on top of `asyncore`_ and has been deprecated since Python 3.6. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > asyncore > ~~~~~~~~ > > The `asyncore `_ module > was > the first module for asynchronous socket service clients and servers. It > has been replaced by asyncio and is deprecated since Python 3.6. > > The ``asyncore`` module is also used in stdlib tests. The tests for > ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly > based on ``asyncore``. These tests must be updated to use asyncio or > threading. > > Module type > pure Python > Deprecated in > 3.6 > Removed in > 3.10 > Substitute > asyncio > > > cgi > ~~~ > > The `cgi `_ module is a > support > module for Common Gateway Interface (CGI) scripts. CGI is deemed as > inefficient because every incoming request is handled in a new process. PEP > 206 considers the module as *designed poorly and are now near-impossible > to fix*. > > Several people proposed to either keep the cgi module for features like > `cgi.parse_qs()` or move `cgi.escape()` to a different module. The > functions `cgi.parse_qs` and `cgi.parse_qsl` have been > deprecated for a while and are actually aliases for > `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The > function `cgi.quote` has been deprecated in favor of `html.quote` > with secure default values. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > cgitb > ~~~~~ > > The `cgitb `_ module is a > helper for the cgi module for configurable tracebacks. > > The ``cgitb`` module is not used by any major Python web framework (Django, > Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an > optional debugging middleware. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > smtpd > ~~~~~ > > The `smtpd `_ module > provides > a simple implementation of a SMTP mail server. The module documentation > marks the module as deprecated and recommends ``aiosmtpd`` instead. The > deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. > > Module type > pure Python > Deprecated in > **3.7** > To be removed in > 3.10 > Substitute > aiosmtpd > > nntplib > ~~~~~~~ > > The `nntplib `_ module > implements the client side of the Network News Transfer Protocol (nntp). > News > groups used to be a dominant platform for online discussions. Over the last > two decades, news has been slowly but steadily replaced with mailing lists > and web-based discussion platforms. Twisted is also > `planning `_ to deprecate NNTP > support and `pynnt `_ hasn't seen > any > activity since 2014. This is a good indicator that the public interest in > NNTP support is declining. > > The ``nntplib`` tests have been the cause of additional work in the recent > past. Python only contains client side of NNTP. The tests connect to > external news server. The servers are sometimes unavailble, too slow, or do > not work correctly over IPv6. The situation causes flaky test runs on > buildbots. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > > Operating system interface > -------------------------- > > crypt > ~~~~~ > > The `crypt `_ module > implements > password hashing based on ``crypt(3)`` function from ``libcrypt`` or > ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of poor > quality and insecure. Users are discouraged to use them. > > * The module is not available on Windows. Cross-platform application need > an alternative implementation any way. > * Only DES encryption is guarenteed to be available. DES has an extremely > limited key space of 2**56. > * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. > SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only > algorithm that is still secure. However it's in glibc and therefore not > commonly available on Linux. > * Depending on the platform, the ``crypt`` module is not thread safe. Only > implementations with ``crypt_r(3)`` are thread safe. > * The module was never useful to interact with system user and password > databases. On BSD, macOS, and Linux, all user authentication and > password modification operations must go through PAM (pluggable > authentication module), see `spwd`_ deprecation. > > Module type > C extension + Python module > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `bcrypt `_, > `passlib `_, > `argon2cffi `_, > hashlib module (PBKDF2, scrypt) > > macpath > ~~~~~~~ > > The `macpath `_ module > provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no longer > supported > > Module type > pure Python > Deprecated in > 3.7 > Removed in > 3.8 > Substitute > **none** > > nis > ~~~ > > The `nis `_ module provides > NIS/YP support. Network Information Service / Yellow Pages is an old and > deprecated directory service protocol developed by Sun Microsystems. It's > designed successor NIS+ from 1992 never took off. For a long time, libc's > Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more > powerful > and more secure replacement of NIS. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > spwd > ~~~~ > > The `spwd `_ module provides > direct access to Unix shadow password database using non-standard APIs. > > In general it's a bad idea to use the spwd. The spwd circumvents system > security policies, it does not use the PAM stack, and is only compatible > with local user accounts, because it ignores NSS. The use of the ``spwd`` > module for access control must be consider a *security bug*, as it bypasses > PAM's access control. > > Further more the ``spwd`` module uses the > `shadow(3) `_ APIs. > Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. > This > is dangerous and even forbidden for confined services on systems with a > security engine like SELinux or AppArmor. > > Module type > C extension > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > `python-pam `_, > `simpleplam `_ > > Misc modules > ------------ > > formatter > ~~~~~~~~~ > > The `formatter `_ module > is an old text formatting module which has been deprecated since Python > 3.4. > > Module type > pure Python > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > *n/a* > > imp > ~~~ > > The `imp `_ module is the > predecessor of the > `importlib `_ module. > Most > functions have been deprecated since Python 3.3 and the module since > Python 3.4. > > Module type > C extension > Deprecated in > 3.4 > To be removed in > 3.10 > Substitute > importlib > > msilib > ~~~~~~ > > The `msilib `_ package is a > Windows-only package. It supports the creation of Microsoft Installers > (MSI). > The package also exposes additional APIs to create cabinet files (CAB). The > module is used to facilitate distutils to create MSI installers with > ``bdist_msi`` command. In the past it was used to create CPython's official > Windows installer, too. > > Microsoft is slowly moving away from MSI in favor of Windows 10 Apps (AppX) > as new deployment model [3]_. > > Module type > C extension + Python code > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > **none** > > parser > ~~~~~~ > > The `parser `_ module > provides > an interface to Python?s internal parser and byte-code compiler. The stdlib > has superior ways to interact with the parse tree. From Python 2.5 onward, > it's much more convenient to cut in at the Abstract Syntax Tree (AST) > generation and compilation stage. > > The ``parser`` module causes additional work. It's C code that must be > kept in sync with any change to Python's grammar and internal parser. > Pablo wants to remove the parser module and promote lib2to3's pgen2 instead > [6]_. > > Most importantly the presence of the ``parser`` module makes it harder to > switch to something more powerful than a LL(1) parser [7]_. Since the > ``parser`` module is documented as deprecated since Python 2.5 and a new > parsing technology is planned for 3.9, the ``parser`` module is scheduled > for > removal in 3.9. > > Module type > C extension > Deprecated in > 3.8, documented as deprecated since **2.5** > To be removed in > **3.9** > Substitute > ast, lib2to3.pgen2 > > pipes > ~~~~~ > > The `pipes `_ module > provides > helpers to pipe the input of one command into the output of another > command. > The module is built on top of ``os.popen``. Users are encouraged to use > the subprocess module instead. > > Module type > pure Python > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > subprocess module > > Removed modules > =============== > > fpectl > ------ > > The `fpectl `_ module was > never built by default, its usage was discouraged and considered dangerous. > It also required a configure flag that caused an ABI incompatibility. The > module was removed in 3.7 by Nathaniel J. Smith in > `bpo-29137 `_. > > Module type > C extension + CAPI > Deprecated in > 3.7 > Removed in > 3.7 > Substitute > **none** > > > Modules to keep > =============== > > Some modules were originally proposed for deprecation. > > fileinput > --------- > > The `fileinput `_ module > implements a helpers to iterate over a list of files from ``sys.argv``. The > module predates the optparser and argparser module. The same functionality > can be implemented with the argparser module. > > Several core developers expressed their interest to keep the module in the > standard library, as it is handy for quick scripts. > > Module type > pure Python > > lib2to3 > ------- > > The `lib2to3 `_ package > provides > the ``2to3`` command to transpile Python 2 code to Python 3 code. > > The package is useful for other tasks besides porting code from Python 2 to > 3. For example `black`_ uses it for code reformatting. > > Module type > pure Python > > getopt > ------ > > The `getopt `_ module > mimics > C's getopt() option parser. > > Although users are encouraged to use argparse instead, the getopt module is > still widely used. The module is small, simple, and handy for C developers > to write simple Python scripts. > > Module type > pure Python > Substitute > argparse > > optparse > -------- > > The `optparse `_ module > is > the predecessor of the argparse module. > > Although it has been deprecated for many years, it's still too widely used > to remove it. > > Module type > pure Python > Deprecated in > 3.2 > Substitute > argparse > > wave > ---- > > The `wave `_ module provides > support for the WAV sound format. > > The module is not deprecated, because The WAV format is still relevant > these > days. The ``wave`` module is also used in education, e.g. to show kids how > to make noise with a computer. > > The module uses one simple function from the `audioop`_ module to perform > byte swapping between little and big endian formats. Before 24 bit WAV > support was added, byte swap used to be implemented with the ``array`` > module. To remove ``wave``'s dependency on the ``audioop``, the byte swap > function could be either be moved to another module (e.g. ``operator``) or > the ``array`` module could gain support for 24 bit (3 byte) arrays. > > Module type > pure Python (depends on *byteswap* from `audioop`_ C extension) > Deprecated in > 3.8 > To be removed in > 3.10 > Substitute > *n/a* > > > Future maintenance of removed modules > ===================================== > > The main goal of the PEP is to reduce the burden and workload on the Python > core developer team. Therefore removed modules will not be maintained by > the core team as separate PyPI packages. However the removed code, tests > and > documentation may be moved into a new git repository, so community members > have a place from which they can pick up and fork code. > > A first draft of a `legacylib `_ > repository is available on my private Github account. The modules could be > made available on PyPI. The Python core team will not publish or maintain > the packages. It is my hope that members of the Python community will > adopt, maintain, and perhaps improve the deprecated modules. > > It's my hope that some of the deprecated modules will be picked up and > adopted by users that actually care about them. For example ``colorsys`` > and > ``imghdr`` are useful modules, but have limited feature set. A fork of > ``imghdr`` can add new features and support for more image formats, without > being constrained by Python's release cycle. > > Most of the modules are in pure Python and can be easily packaged. Some > depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since `audioop`_ > does not depend on any external libraries, it can be shipped in as binary > wheels with some effort. Other C modules can be replaced with ctypes or > cffi. > For example I created `legacycrypt >`_ > with ``_crypt`` extension reimplemented with a few lines of ctypes code. > > > Discussions > =========== > > * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. > * Berker Peksag proposed to deprecate and removed *msilib*. > * Brett Cannon recommended to delay active deprecation warnings and removal > of modules like *imp* until Python 3.10. Version 3.8 will be released > shortly before Python 2 reaches end of lifetime. A delay reduced churn > for > users that migrate from Python 2 to 3.8. > * Brett also came up with the idea to keep lib2to3. The package is useful > for other purposes, e.g. `black `_ uses > it to reformat Python code. > * At one point, distutils was mentioned in the same sentence as this PEP. > To avoid lengthy discussion and delay of the PEP, I decided against > dealing > with distutils. Deprecation of the distutils package will be handled by > another PEP. > * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) > convinced me to keep the `wave`_ module. [4]_ > * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ > * Andrew Svetlov mentioned the ``socketserver`` module is questionable. > However it's used to implement ``http.server`` and ``xmlrpc.server``. The > stdlib doesn't have a replacement for the servers, yet. > > > Update history > ============== > > Update 1 > -------- > > * Deprecate `parser`_ module > * Keep `fileinput`_ module > * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad > * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ > modules > * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections now > list suitable substitutions. > * Mention that ``socketserver`` is going to stay for ``http.server`` and > ``xmlrpc.server`` > * The future maintenance section now states that the deprecated modules > may be adopted by Python community members. > > References > ========== > > .. [1] > https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free > .. [2] https://man.openbsd.org/ossaudio > .. [3] > https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ > .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 > .. [5] https://twitter.com/dabeaz/status/1130278844479545351 > .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html > .. [7] > https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/dfmoisset%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun May 26 05:24:15 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 26 May 2019 11:24:15 +0200 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. References: <64d3f69a-b900-d17d-679e-aa748d0a23ab@python.org> Message-ID: <20190526112415.33e4a02d@fsol> On Fri, 24 May 2019 14:23:21 +0200 Thomas Wouters wrote: > On Thu, May 23, 2019 at 5:15 PM Steve Dower wrote: > > > On 23May2019 0542, Inada Naoki wrote: > > > 1. perf shows 95% of CPU time is eaten by _PyObject_Free, not kernel > > space. > > > 2. This loop is cleary hot: > > > > > https://github.com/python/cpython/blob/51aa35e9e17eef60d04add9619fe2a7eb938358c/Objects/obmalloc.c#L1816-L1819 > > > > > > I can attach the process by gdb and I confirmed many arenas have > > > same nfreepools. > > > > It's relatively easy to test replacing our custom allocators with the > > system ones, yes? Can we try those to see whether they have the same > > characteristic? > > > > Given the relative amount of investment over the last 19 years [1], I > > wouldn't be surprised if most system ones are at least as good for our > > needs now. Certainly Windows HeapAlloc has had serious improvements in > > that time to help with fragmentation and small allocations. > > > > FYI, and I've mentioned this at PyCon to a few people (might've been you, > Steve, I don't remember) -- but at Google we've experimented with disabling > obmalloc when using TCMalloc (a faster and thread-aware malloc, which makes > a huge difference within Google when dealing with multi-threaded C++ > libraries), both using the usual Python benchmarks and real-world code with > real-world workloads (a core part of YouTube, for example), all on Linux. > There's still a significant benefit to using obmalloc when using glibc's > malloc, and also a noticeable benefit when using TCMalloc. There are > certainly cases where it doesn't matter much, and there may even be cases > where the overhead of obmalloc isn't worth it, but I do believe it's still > a firm net benefit. Interesting that a 20-year simple allocator (obmalloc) is able to do better than the sophisticated TCMalloc. (well, of course, obmalloc doesn't have to worry about concurrent scenarios, which explains some of the simplicity) Regards Antoine. From larry at hastings.org Sun May 26 05:29:14 2019 From: larry at hastings.org (Larry Hastings) Date: Sun, 26 May 2019 02:29:14 -0700 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: On 5/22/19 12:15 PM, Tim Peters wrote: > There's a Stackoverflow report[1] I suspect is worth looking into, but > it requires far more RAM (over 80GB) than I have). [...] > But, for a start, it would be good to know if anyone else can actually > reproduce the problem. > > [1] https://stackoverflow.com/questions/56228799/python-hangs-indefinitely-trying-to-delete-deeply-recursive-object I have a computer with two Xeon CPUs and 256GB of RAM.? So, even though it's NUMA, I still have 128GB of memory per CPU.? It's running a "spin" of Ubuntu 18.10. I compiled a fresh Python 3.7.3 --with-optimizations.? I copied the sample program straight off the StackOverflow page.? The program ran for about five and a half hours then exited normally. During the run it printed: This gets printed! This doesn't get printed Statistics reported by "time": 19811.05s user 123.56s system 99% cpu 5:32:15.04 total Checking in on it now and then, peak observed memory usage (as reported by "top") was just under 80GB. I take it that the interesting part was confirming that "This doesn't get printed" gets printed when you have enough RAM for the program to run to completion.? So I guess there's no bug here? Just an observation about CPython's garbage collector being kinda slow?? Or maybe CPython gc + swap = super bombad slow? //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sun May 26 05:46:16 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 26 May 2019 11:46:16 +0200 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules References: <20190524065544.GS4221@ando.pearwood.info> <25c05229-7e8a-c8b7-8445-ccbb5836aecb@python.org> Message-ID: <20190526114616.3b53745e@fsol> On Fri, 24 May 2019 09:54:05 -0700 Steve Dower wrote: > > All in all, this is basically where we are today, with the exception > that we haven't officially said that we no longer support these modules. > PEP 594 is this official statement, and our usual process for things we > don't support is to remove them in two versions time. It doesn't have to > be so controversial - either the people who are saying "we rely on this" > are also willing to help us maintain them, or they're not. And if > they're not, they clearly don't rely on it (or realize the cost of > relying on volunteer-maintained software). This is an absurd statement. You can rely on something without having the time or competence required to maintain it yourself. Division of labour exists for a reason. Obviously this doesn't mean that the CPython development team must absorb any development activity that may be useful to the greater public. But widely-used software packages such as a Python and its stdlib have a tremendous enabling effect. Regards Antoine. From steve at pearwood.info Sun May 26 06:06:02 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 26 May 2019 20:06:02 +1000 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: <20190526100601.GY4221@ando.pearwood.info> (And this time I will remember to remove the SPAM label...) On Sun, May 26, 2019 at 08:44:33AM +1000, Chris Angelico wrote: > From my reading of the description, you could also "assert a is b" - > is that correct? Yes, that's already the behaviour. py> def demo(): ... a = locals() ... b = locals() ... print(a is b) ... py> demo() True -- Steven From steve at pearwood.info Sun May 26 06:04:11 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 26 May 2019 20:04:11 +1000 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: <20190526100410.GX4221@ando.pearwood.info> Richard, your email seems to have introduced a spurious "SPAM" label to this thread, which may confuse some email clients into treating it as spam. Can you teach your email program that this mailing list is ham, not spam, or failing that, at least edit the subject line to remove the label? Thanks. I've done so for this response, but please take care that you don't re-introduce the label again, thanks. On Sat, May 25, 2019 at 06:37:22PM -0400, Richard Damon wrote: > To me that is a static snapshot of a dynamic environment, not a dynamic > snapshot. The snapshot you get at THAT moment in time won't change, as > time progresses, so that snapshot itself isn't dynamic. Actually, it does change -- but the confusing part is that it doesn't change automatically but only when you call the locals() function again. This already CPython's behaviour, so that is not changing. def demo1(): a = b = c = 1 a = locals() print(a) b = 999 print(a) def demo2(): a = b = c = 1 a = locals() print(a) b = 999 locals() # call function but throw the result away print(a) And running those two functions in Python 3.5: py> demo1() # No change to the dict. {'a': 1, 'b': 1, 'c': 1} {'a': 1, 'b': 1, 'c': 1} py> demo2() # Dict magically updates! {'a': 1, 'b': 1, 'c': 1} {'a': {...}, 'b': 999, 'c': 1} I know this is the backwards-compatible behaviour, but I would like to question whether we want to enshrine it in stone. This seems to me to be the worst possible combinations of features: - writing to the locals() dict doesn't write changes back to the actual local variables; - the dict returned isn't a fixed snapshot, but the PEP calls it a snapshot despite not being one (naming things is hard); - the "snapshop" can change as a side-effect of another operation. If this wasn't already the behaviour, would we want it? -- Steven From steve at pearwood.info Sun May 26 06:14:02 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 26 May 2019 20:14:02 +1000 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: <20190526100410.GX4221@ando.pearwood.info> References: <20190526100410.GX4221@ando.pearwood.info> Message-ID: <20190526101401.GZ4221@ando.pearwood.info> On Sun, May 26, 2019 at 08:04:11PM +1000, Steven D'Aprano wrote: > Richard, your email seems to have introduced a spurious "SPAM" label [...] > edit the subject line to remove the > label? Thanks. I've done so for this response Oops. Done now. -- Steven From rosuav at gmail.com Sun May 26 07:20:53 2019 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 26 May 2019 21:20:53 +1000 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: <20190526100601.GY4221@ando.pearwood.info> References: <20190526100601.GY4221@ando.pearwood.info> Message-ID: On Sun, May 26, 2019 at 8:07 PM Steven D'Aprano wrote: > On Sun, May 26, 2019 at 08:44:33AM +1000, Chris Angelico wrote: > > > From my reading of the description, you could also "assert a is b" - > > is that correct? > > Yes, that's already the behaviour. > > py> def demo(): > ... a = locals() > ... b = locals() > ... print(a is b) > ... > py> demo() > True > Sure, but this PEP is all about defining things that weren't previously defined, so I wanted to clarify intent rather than current behaviour. ChrisA From p.f.moore at gmail.com Sun May 26 07:53:42 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 26 May 2019 12:53:42 +0100 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: <20190526100601.GY4221@ando.pearwood.info> Message-ID: On Sun, 26 May 2019 at 12:23, Chris Angelico wrote: > > On Sun, May 26, 2019 at 8:07 PM Steven D'Aprano wrote: > > On Sun, May 26, 2019 at 08:44:33AM +1000, Chris Angelico wrote: > > > > > From my reading of the description, you could also "assert a is b" - > > > is that correct? > > > > Yes, that's already the behaviour. > > > > py> def demo(): > > ... a = locals() > > ... b = locals() > > ... print(a is b) > > ... > > py> demo() > > True > > > > Sure, but this PEP is all about defining things that weren't > previously defined, so I wanted to clarify intent rather than current > behaviour. +1 on the PEP being explicit over this. Even though it's current behaviour, it's surprising and making a clear and definitive statement that the PEP intends to make the behaviour part of the language definition and not just a CPython detail, is IMO worthwhile. Paul From steve at pearwood.info Sun May 26 07:50:09 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 26 May 2019 21:50:09 +1000 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: <20190526100601.GY4221@ando.pearwood.info> Message-ID: <20190526115009.GA4221@ando.pearwood.info> On Sun, May 26, 2019 at 09:20:53PM +1000, Chris Angelico wrote: > Sure, but this PEP is all about defining things that weren't > previously defined, so I wanted to clarify intent rather than current > behaviour. As I understand it, the intent is to: - fix some annoyances/bugs involved when you have a trace hook; - make the current CPython behaviour a language behaviour. -- Steven From vano at mail.mipt.ru Sun May 26 09:03:11 2019 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Sun, 26 May 2019 16:03:11 +0300 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules In-Reply-To: <20190524065544.GS4221@ando.pearwood.info> References: <20190524065544.GS4221@ando.pearwood.info> Message-ID: On 24.05.2019 9:55, Steven D'Aprano wrote: > I don't know if this is a good idea or a terrible idea or somewhere in > between, so I'm throwing it out to see if anyone likes it. > > Let's add a third option to PEP 594 between "keep" and "remove": > explicitly flagging a module as unmaintained. Unmaintained modules: > > - will raise a warning when imported that they are unmaintained; > > - will have their tests disabled; > > - possibly we move them into a seperate namespace: > ``from unmaintained import aardvark`` > > - bug reports without patches will be closed Will Not Fix; > > - bug reports with patches *may* be accepted if some core dev is > willing to review and check it in, but there is no obligation > to do so; > > - should it turn out that someone is willing to maintain the module, > it can be returned to regular status. > > > Plus side: > > - reduce the maintenance burden (if any) from the module; > > - while still distributing the module and allowing users to use > it: "no promises, but here you go"; > > - other implementations are under no obligation to distribute > unmaintained modules. > > Minus side: > > - this becomes a self-fulfilling prophesy: with tests turned off, > bit-rot will eventually set in and break modules that currently > aren't broken. > > > > Thoughts? This would greately damage Python's image as a mature, reliable product, something that you can bank on for your critical tasks. By including something into the offer, we implicitly endorse it and claim that we want out users to use that rather than competing offers. You may look at the archives where I relied on TKinter for an important project and it let me down big time. I would very much prefer Python to honestly admit that there's not enough interest in this module and drop it so I would look elsewhere right off the bat and don't. You may argue that marking it as unsupported would do the same but it wouldn't. This mean I'll have to be constantly on the lookout for hidden notes and such, I will forever lose confidence that I can just take whatever is in Python and run with it, being confident that the Python team has my back. -- Regards, Ivan From thomas at python.org Sun May 26 10:02:21 2019 From: thomas at python.org (Thomas Wouters) Date: Sun, 26 May 2019 16:02:21 +0200 Subject: [Python-Dev] Accepting PEP 587 (Python Initialization Configuration) Message-ID: As PEP-delegate, I'm accepting version 5 of PEP 587 (Python Initialization Configuration). I don't think Victor has posted the latest version, but it contains a couple of minor renames, and a new private API to allow us to experiment with two-phase initialisation of Python. There's still time to debate exact names and work on the wording of the PEP, but I'm satisfied the design is solid, a clear improvement over what we currently have, and a step in the right direction for some of the more extensive changes that have been proposed (e.g. using Python *code* for the Python configuration. If we do get to that point, the PEP contains a few things that are superfluous, but I'd consider them minor transitionary warts at that point -- if we ever get there). Thanks, Victor, for pushing this massive amount of work through; it won't affect most people using Python (at least until the rest of PEP 432 lands) but it's a giant boon to those of us who embed Python :) -- Thomas Wouters Hi! I'm an email virus! Think twice before sending your email to help me spread! -------------- next part -------------- An HTML attachment was scrubbed... URL: From mark at hotpy.org Sun May 26 11:34:02 2019 From: mark at hotpy.org (Mark Shannon) Date: Sun, 26 May 2019 16:34:02 +0100 Subject: [Python-Dev] Getting #6641 merged Message-ID: <9f25edd4-dfb3-0934-97d5-05a40e99b50d@hotpy.org> Hi, I'd like to get https://github.com/python/cpython/pull/6641 merged. I keep having to rebase it and regenerate all the importlib header files, which is becoming a bit annoying. So, I can I ask that if you are going to modify Python/ceval.c can you hold on just a little while, until #6641 is merged. Thanks, Mark. From tim.peters at gmail.com Sun May 26 11:55:27 2019 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 26 May 2019 10:55:27 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: [Larry Hastings ] > I have a computer with two Xeon CPUs and 256GB of RAM. So, even > though it's NUMA, I still have 128GB of memory per CPU. It's running a > "spin" of Ubuntu 18.10. > > I compiled a fresh Python 3.7.3 --with-optimizations. I copied the sample > program straight off the StackOverflow page. The program ran for about > five and a half hours then exited normally. Thanks, Larry! > During the run it printed: > > This gets printed! > This doesn't get printed > > Statistics reported by "time": > > 19811.05s user 123.56s system 99% cpu 5:32:15.04 total > > Checking in on it now and then, peak observed memory usage (as reported > by "top") was just under 80GB. All of that is consistent with what others reported, although you've given more quantified details, and that's helpful. > I take it that the interesting part was confirming that "This doesn't get printed" > gets printed when you have enough RAM for the program to run to completion. That was the primary point at first, yes, > So I guess there's no bug here? Just an observation about CPython's > garbage collector being kinda slow? Or maybe CPython gc + swap = > super bombad slow? No need to confirm this: if you ran it again with a bit more output, you'd find that the vast bulk of the time was spent between the two output lines. That is, it takes hours from the time the train() function starts to return and completes returning. That's all consumed as a side effect of `tree` losing its last reference. We know why now, and it's hard to say whether it's "a bug". It's functioning as designed, but the design sucks ;-) So I'd call it a design flaw. The heuristics that were introduced to try to make it likely we could return more empty arenas to C were designed when machines had far less RAM, and can burn time quadratic in the number of arenas. That really didn't matter in a world with a few hundred arenas, but can be a timing disaster in a world with hundreds of thousands (or more) of arenas. The Stackoverflow test case was whittled way down from the OP's real program, which ran "for days" without completing. I've sketched two (albeit closely related, both to each other and to what we currently do) ways the worst-case overall time could be cut from quadratic to linear in the number of arenas, which is asymptotically optimal. More ambitious would be to dream up an entirely different heuristic. Keeping the list of usable arenas fully sorted in order of number of free pools seems a very strong requirement for a poke-and-hope heuristic. But, best I can tell, the person who came up with this heuristic to begin with didn't check in any motivating test programs. So we'd be starting over from scratch. For this specific Stackoverflow case, I've confirmed that it doesn't make a real difference if we didn't bother to sort the list of arenas _at all_ (then it still returns about the same number of arenas to C, both at the point the tree finishes building, and at the time it completes tearing down the tree). So I have a program showing the current scheme can be a disaster, but none where it shows it actually helps anything ;-) From brett at python.org Sun May 26 13:07:21 2019 From: brett at python.org (Brett Cannon) Date: Sun, 26 May 2019 10:07:21 -0700 Subject: [Python-Dev] PEP 594: update 1 In-Reply-To: References: <0acd0f02-ecff-8ea0-757b-30884b4ea8b0@python.org> Message-ID: On Sat, May 25, 2019 at 8:00 PM Daniel Moisset wrote: > Hi, thanks for the work on this proposal, I think this is at least a tip > of the iceberg and a good start for the bigger question of how the stdlib > should evolve.. > > I think that the PEP should include an idea of what should happen if > existing stdlib pieces depend on this deprecated modules. For example, > email.mime.audio is part of a non-deprecated module but it depends on > sndhdr which is being planned for deprecation. Should that part of the > functionality be deprecated too? rewritten to not depend on the deprecated > module (but keep the pieces of relevant code? depend on optional 3rd party > modules and degrade if those are not available? > It's a case-by-case situation. In some instances we may copy the relevant code over and keep it private. In other situations some functionality may get deprecated. It all depends and thus why you can't really write a general solution that applies uniformly. -Brett > > The sndhdr case is just an example but I can imagine there are others > (optparse is no longer schedule for removal in your PEP, but would have > been another case, being used by the profile module). > > Best, > Daniel > > On Tue, 21 May 2019 at 15:15, Christian Heimes > wrote: > >> Hi, >> >> I have updated the PEP with feedback from discussions. The highlights are: >> >> * Deprecate parser module >> * Keep fileinput module >> * Elaborate why crypt and spwd are dangerous and bad >> * Improve sections for cgitb, colorsys, nntplib, and smtpd modules >> * The colorsys, crypt, imghdr, sndhdr, and spwd sections now list >> suitable substitutions. >> * Mention that socketserver is going to stay for http.server and >> xmlrpc.server >> * The future maintenance section now states that the deprecated modules >> may be adopted by Python community members. >> >> https://github.com/python/peps/compare/7799178a...2d536899?diff=unified#diff-ae358c21fa7968ee3b6c64479e051574 >> >> >> I'll be traveling the next couple of days and will only have limited >> opportunities to respond on feedback. >> >> Christian >> >> ------------------------------------------------------------------- >> PEP: 594 >> Title: Removing dead batteries from the standard library >> Author: Christian Heimes >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 20-May-2019 >> Post-History: 21-May-2019 >> >> >> Abstract >> ======== >> >> This PEP proposed a list of standard library modules to be removed from >> the >> standard library. The modules are mostly historic data formats and APIs >> that >> have been superseded a long time ago, e.g. Mac OS 9 and Commodore. >> >> >> Rationale >> ========= >> >> Back in the early days of Python, the interpreter came with a large set of >> useful modules. This was often refrained to as "batteries included" >> philosophy and was one of the corner stones to Python's success story. >> Users didn't have to figure out how to download and install separate >> packages in order to write a simple web server or parse email. >> >> Times have changed. The introduction of the cheese shop (PyPI), >> setuptools, >> and later pip, it became simple and straight forward to download and >> install >> packages. Nowadays Python has a rich and vibrant ecosystem of third party >> packages. It's pretty much standard to either install packages from PyPI >> or >> use one of the many Python or Linux distributions. >> >> On the other hand, Python's standard library is piling up cruft, >> unnecessary >> duplication of functionality, and dispensable features. This is >> undesirable >> for several reasons. >> >> * Any additional module increases the maintenance cost for the Python core >> development team. The team has limited resources, reduced maintenance >> cost >> frees development time for other improvements. >> * Modules in the standard library are generally favored and seen as the >> de-facto solution for a problem. A majority of users only pick 3rd party >> modules to replace a stdlib module, when they have a compelling reason, >> e.g. >> lxml instead of `xml`. The removal of an unmaintained stdlib module >> increases the chances of a community contributed module to become widely >> used. >> * A lean and mean standard library benefits platforms with limited >> resources >> like devices with just a few hundred kilobyte of storage (e.g. BBC >> Micro:bit). Python on mobile platforms like BeeWare or WebAssembly >> (e.g. pyodide) also benefit from reduced download size. >> >> The modules in the PEP have been selected for deprecation because their >> removal is either least controversial or most beneficial. For example >> least controversial are 30 years old multimedia formats like ``sunau`` >> audio format, which was used on SPARC and NeXT workstations in the late >> 1980ties. The ``crypt`` module has fundamental flaws that are better >> solved >> outside the standard library. >> >> This PEP also designates some modules as not scheduled for removal. Some >> modules have been deprecated for several releases or seem unnecessary at >> first glance. However it is beneficial to keep the modules in the standard >> library, mostly for environments where installing a package from PyPI is >> not >> an option. This can be cooperate environments or class rooms where >> external >> code is not permitted without legal approval. >> >> * The usage of FTP is declining, but some files are still provided over >> the FTP protocol or hosters offer FTP to upload content. Therefore >> ``ftplib`` is going to stay. >> * The ``optparse`` and ``getopt`` module are widely used. They are mature >> modules with very low maintenance overhead. >> * According to David Beazley [5]_ the ``wave`` module is easy to teach to >> kids and can make crazy sounds. Making a computer generate crazy sounds >> is >> powerful and highly motivating exercise for a 9yo aspiring developer. >> It's >> a fun battery to keep. >> >> >> Deprecation schedule >> ==================== >> >> 3.8 >> --- >> >> This PEP targets Python 3.8. Version 3.8.0 final is scheduled to be >> released >> a few months before Python 2.7 will reach its end of lifetime. We expect >> that >> Python 3.8 will be targeted by users that migrate to Python 3 in 2019 and >> 2020. To reduce churn and to allow a smooth transition from Python 2, >> Python 3.8 will neither raise `DeprecationWarning` nor remove any >> modules that have been scheduled for removal. Instead deprecated modules >> will >> just be *documented* as deprecated. Optionally modules may emit a >> `PendingDeprecationWarning`. >> >> All deprecated modules will also undergo a feature freeze. No additional >> features should be added. Bug should still be fixed. >> >> 3.9 >> --- >> >> Starting with Python 3.9, deprecated modules will start issuing >> `DeprecationWarning`. The `parser`_ module is removed and potentially >> replaced with a new module. >> >> All other deprecated modules are fully supported and will receive security >> updates until Python 3.9 reaches its end of lifetime. Python 3.9.0 will >> be released about 18 months after 3.8.0 (April 2021?) and most likely >> be supported for 5 years after the release. The estimated EOL of Python >> 3.9 >> is in 2026. >> >> 3.10 >> ---- >> >> In 3.10 all deprecated modules will be removed from the CPython repository >> together with tests, documentation, and autoconf rules. >> >> >> PEP acceptance process >> ====================== >> >> 3.8.0b1 is scheduled to be release shortly after the PEP is officially >> submitted. Since it's improbable that the PEP will pass all stages of the >> PEP process in time, I propose a two step acceptance process that is >> analogous Python's two release deprecation process. >> >> The first *provisionally accepted* phase targets Python 3.8.0b1. In the >> first >> phase no code is changes or removed. Modules are only documented as >> deprecated. The only exception is the `parser`_ module. It has been >> documented as deprecated since Python 2.5 and is scheduled for removal for >> 3.9 to make place for a more advanced parser. >> >> The final decision, which modules will be removed and how the removed code >> is preserved, can be delayed for another year. >> >> >> Deprecated modules >> ================== >> >> The modules are grouped as data encoding, multimedia, network, OS >> interface, >> and misc modules. The majority of modules are for old data formats or >> old APIs. Some others are rarely useful and have better replacements on >> PyPI, e.g. Pillow for image processing or NumPy-based projects to deal >> with >> audio processing. >> >> .. csv-table:: Table 1: Proposed modules deprecations >> :header: "Module", "Deprecated in", "To be removed", "Replacement" >> :widths: 1, 1, 1, 2 >> >> aifc,3.8,3.10,\- >> asynchat,3.8,3.10,asyncio >> asyncore,3.8,3.10,asyncio >> audioop,3.8,3.10,\- >> binhex,3.8,3.10,\- >> cgi,3.8,3.10,\- >> cgitb,3.8,3.10,\- >> chunk,3.8,3.10,\- >> colorsys,3.8,3.10,"colormath, colour, colorspacious, Pillow" >> crypt,3.8,3.10,"bcrypt, argon2cffi, hashlib, passlib" >> fileinput,\-,**keep**,argparse >> formatter,3.4,3.10,\- >> fpectl,**3.7**,**3.7**,\- >> getopt,**3.2**,**keep**,"argparse, optparse" >> imghdr,3.8,3.10,"filetype, puremagic, python-magic" >> imp,**3.4**,3.10,importlib >> lib2to3,\-,**keep**, >> macpath,**3.7**,**3.8**,\- >> msilib,3.8,3.10,\- >> nntplib,3.8,3.10,\- >> nis,3.8,3.10,\- >> optparse,\-,**keep**,argparse >> ossaudiodev,3.8,3.10,\- >> parser,**2.5**,**3.9**,"ast, lib2to3.pgen2" >> pipes,3.8,3.10,subprocess >> smtpd,"**3.4.7**, **3.5.4**",3.10,aiosmtpd >> sndhdr,3.8,3.10,"filetype, puremagic, python-magic" >> spwd,3.8,3.10,"python-pam, simplepam" >> sunau,3.8,3.10,\- >> uu,3.8,3.10,\- >> wave,\-,**keep**, >> xdrlib,3.8,3.10,\- >> >> >> Data encoding modules >> --------------------- >> >> binhex >> ~~~~~~ >> >> The `binhex `_ module >> encodes >> and decodes Apple Macintosh binhex4 data. It was originally developed for >> TSR-80. In the 1980s and early 1990s it was used on classic Mac OS 9 to >> encode binary email attachments. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> uu >> ~~ >> >> The `uu `_ module provides >> uuencode format, an old binary encoding format for email from 1980. The uu >> format has been replaced by MIME. The uu codec is provided by the binascii >> module. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> xdrlib >> ~~~~~~ >> >> The `xdrlib `_ module >> supports >> the Sun External Data Representation Standard. XDR is an old binary >> serialization format from 1987. These days it's rarely used outside >> specialized domains like NFS. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Multimedia modules >> ------------------ >> >> aifc >> ~~~~ >> >> The `aifc `_ module provides >> support for reading and writing AIFF and AIFF-C files. The Audio >> Interchange >> File Format is an old audio format from 1988 based on Amiga IFF. It was >> most >> commonly used on the Apple Macintosh. These days only few specialized >> application use AIFF. >> >> Module type >> pure Python (depends on `audioop`_ C extension) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> audioop >> ~~~~~~~ >> >> The `audioop `_ module >> contains helper functions to manipulate raw audio data and adaptive >> differential pulse-code modulated audio data. The module is implemented in >> C without any additional dependencies. The `aifc`_, `sunau`_, and `wave`_ >> module depend on `audioop`_ for some operations. The byteswap operation in >> the `wave`_ module can be substituted with little work. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> colorsys >> ~~~~~~~~ >> >> The `colorsys `_ module >> defines color conversion functions between RGB, YIQ, HSL, and HSV >> coordinate >> systems. >> >> The PyPI packages *colormath*, *colour*, and *colorspacious* provide more >> and >> advanced features. The Pillow library is better suited to transform images >> between color systems. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `colormath `_, >> `colour `_ >> `colorspacious `_, >> `Pillow `_ >> >> chunk >> ~~~~~ >> >> The `chunk `_ module >> provides >> support for reading and writing Electronic Arts' Interchange File Format. >> IFF is an old audio file format originally introduced for Commodore and >> Amiga. The format is no longer relevant. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> imghdr >> ~~~~~~ >> >> The `imghdr `_ module is a >> simple tool to guess the image file format from the first 32 bytes >> of a file or buffer. It supports only a limited amount of formats and >> neither returns resolution nor color depth. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `puremagic `_, >> `filetype `_, >> `python-magic `_ >> >> ossaudiodev >> ~~~~~~~~~~~ >> >> The `ossaudiodev `_ >> module provides support for Open Sound System, an interface to sound >> playback and capture devices. OSS was initially free software, but later >> support for newer sound devices and improvements were proprietary. Linux >> community abandoned OSS in favor of ALSA [1]_. Some operation systems like >> OpenBSD and NetBSD provide an incomplete [2]_ emulation of OSS. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> sndhdr >> ~~~~~~ >> >> The `sndhdr `_ module is >> similar to the `imghdr`_ module but for audio formats. It guesses file >> format, channels, frame rate, and sample widths from the first 512 bytes >> of >> a file or buffer. The module only supports AU, AIFF, HCOM, VOC, WAV, and >> other ancient formats. >> >> Module type >> pure Python (depends on `audioop`_ C extension for some operations) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `puremagic `_, >> `filetype `_, >> `python-magic `_ >> >> sunau >> ~~~~~ >> >> The `sunau `_ module >> provides >> support for Sun AU sound format. It's yet another old, obsolete file >> format. >> >> Module type >> pure Python (depends on `audioop`_ C extension for some operations) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Networking modules >> ------------------ >> >> asynchat >> ~~~~~~~~ >> >> The `asynchat `_ module >> is build on top of `asyncore`_ and has been deprecated since Python 3.6. >> >> Module type >> pure Python >> Deprecated in >> 3.6 >> Removed in >> 3.10 >> Substitute >> asyncio >> >> asyncore >> ~~~~~~~~ >> >> The `asyncore `_ module >> was >> the first module for asynchronous socket service clients and servers. It >> has been replaced by asyncio and is deprecated since Python 3.6. >> >> The ``asyncore`` module is also used in stdlib tests. The tests for >> ``ftplib``, ``logging``, ``smptd``, ``smtplib``, and ``ssl`` are partly >> based on ``asyncore``. These tests must be updated to use asyncio or >> threading. >> >> Module type >> pure Python >> Deprecated in >> 3.6 >> Removed in >> 3.10 >> Substitute >> asyncio >> >> >> cgi >> ~~~ >> >> The `cgi `_ module is a >> support >> module for Common Gateway Interface (CGI) scripts. CGI is deemed as >> inefficient because every incoming request is handled in a new process. >> PEP >> 206 considers the module as *designed poorly and are now near-impossible >> to fix*. >> >> Several people proposed to either keep the cgi module for features like >> `cgi.parse_qs()` or move `cgi.escape()` to a different module. The >> functions `cgi.parse_qs` and `cgi.parse_qsl` have been >> deprecated for a while and are actually aliases for >> `urllib.parse.parse_qs` and `urllib.parse.parse_qsl`. The >> function `cgi.quote` has been deprecated in favor of `html.quote` >> with secure default values. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> cgitb >> ~~~~~ >> >> The `cgitb `_ module is a >> helper for the cgi module for configurable tracebacks. >> >> The ``cgitb`` module is not used by any major Python web framework >> (Django, >> Pyramid, Plone, Flask, CherryPy, or Bottle). Only Paste uses it in an >> optional debugging middleware. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> smtpd >> ~~~~~ >> >> The `smtpd `_ module >> provides >> a simple implementation of a SMTP mail server. The module documentation >> marks the module as deprecated and recommends ``aiosmtpd`` instead. The >> deprecation message was added in releases 3.4.7, 3.5.4, and 3.6.1. >> >> Module type >> pure Python >> Deprecated in >> **3.7** >> To be removed in >> 3.10 >> Substitute >> aiosmtpd >> >> nntplib >> ~~~~~~~ >> >> The `nntplib `_ module >> implements the client side of the Network News Transfer Protocol (nntp). >> News >> groups used to be a dominant platform for online discussions. Over the >> last >> two decades, news has been slowly but steadily replaced with mailing lists >> and web-based discussion platforms. Twisted is also >> `planning `_ to deprecate >> NNTP >> support and `pynnt `_ hasn't seen >> any >> activity since 2014. This is a good indicator that the public interest in >> NNTP support is declining. >> >> The ``nntplib`` tests have been the cause of additional work in the recent >> past. Python only contains client side of NNTP. The tests connect to >> external news server. The servers are sometimes unavailble, too slow, or >> do >> not work correctly over IPv6. The situation causes flaky test runs on >> buildbots. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> >> Operating system interface >> -------------------------- >> >> crypt >> ~~~~~ >> >> The `crypt `_ module >> implements >> password hashing based on ``crypt(3)`` function from ``libcrypt`` or >> ``libxcrypt`` on Unix-like platform. The algorithms are mostly old, of >> poor >> quality and insecure. Users are discouraged to use them. >> >> * The module is not available on Windows. Cross-platform application need >> an alternative implementation any way. >> * Only DES encryption is guarenteed to be available. DES has an extremely >> limited key space of 2**56. >> * MD5, salted SHA256, salted SHA512, and Blowfish are optional extension. >> SSHA256 and SSHA512 are glibc extensions. Blowfish (bcrypt) is the only >> algorithm that is still secure. However it's in glibc and therefore not >> commonly available on Linux. >> * Depending on the platform, the ``crypt`` module is not thread safe. Only >> implementations with ``crypt_r(3)`` are thread safe. >> * The module was never useful to interact with system user and password >> databases. On BSD, macOS, and Linux, all user authentication and >> password modification operations must go through PAM (pluggable >> authentication module), see `spwd`_ deprecation. >> >> Module type >> C extension + Python module >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `bcrypt `_, >> `passlib `_, >> `argon2cffi `_, >> hashlib module (PBKDF2, scrypt) >> >> macpath >> ~~~~~~~ >> >> The `macpath `_ module >> provides Mac OS 9 implementation of os.path routines. Mac OS 9 is no >> longer >> supported >> >> Module type >> pure Python >> Deprecated in >> 3.7 >> Removed in >> 3.8 >> Substitute >> **none** >> >> nis >> ~~~ >> >> The `nis `_ module provides >> NIS/YP support. Network Information Service / Yellow Pages is an old and >> deprecated directory service protocol developed by Sun Microsystems. It's >> designed successor NIS+ from 1992 never took off. For a long time, libc's >> Name Service Switch, LDAP, and Kerberos/GSSAPI are considered a more >> powerful >> and more secure replacement of NIS. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> spwd >> ~~~~ >> >> The `spwd `_ module provides >> direct access to Unix shadow password database using non-standard APIs. >> >> In general it's a bad idea to use the spwd. The spwd circumvents system >> security policies, it does not use the PAM stack, and is only compatible >> with local user accounts, because it ignores NSS. The use of the ``spwd`` >> module for access control must be consider a *security bug*, as it >> bypasses >> PAM's access control. >> >> Further more the ``spwd`` module uses the >> `shadow(3) `_ APIs. >> Functions like ``getspnam(3)`` access the ``/etc/shadow`` file directly. >> This >> is dangerous and even forbidden for confined services on systems with a >> security engine like SELinux or AppArmor. >> >> Module type >> C extension >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> `python-pam `_, >> `simpleplam `_ >> >> Misc modules >> ------------ >> >> formatter >> ~~~~~~~~~ >> >> The `formatter `_ >> module >> is an old text formatting module which has been deprecated since Python >> 3.4. >> >> Module type >> pure Python >> Deprecated in >> 3.4 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> imp >> ~~~ >> >> The `imp `_ module is the >> predecessor of the >> `importlib `_ module. >> Most >> functions have been deprecated since Python 3.3 and the module since >> Python 3.4. >> >> Module type >> C extension >> Deprecated in >> 3.4 >> To be removed in >> 3.10 >> Substitute >> importlib >> >> msilib >> ~~~~~~ >> >> The `msilib `_ package is >> a >> Windows-only package. It supports the creation of Microsoft Installers >> (MSI). >> The package also exposes additional APIs to create cabinet files (CAB). >> The >> module is used to facilitate distutils to create MSI installers with >> ``bdist_msi`` command. In the past it was used to create CPython's >> official >> Windows installer, too. >> >> Microsoft is slowly moving away from MSI in favor of Windows 10 Apps >> (AppX) >> as new deployment model [3]_. >> >> Module type >> C extension + Python code >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> **none** >> >> parser >> ~~~~~~ >> >> The `parser `_ module >> provides >> an interface to Python?s internal parser and byte-code compiler. The >> stdlib >> has superior ways to interact with the parse tree. From Python 2.5 onward, >> it's much more convenient to cut in at the Abstract Syntax Tree (AST) >> generation and compilation stage. >> >> The ``parser`` module causes additional work. It's C code that must be >> kept in sync with any change to Python's grammar and internal parser. >> Pablo wants to remove the parser module and promote lib2to3's pgen2 >> instead >> [6]_. >> >> Most importantly the presence of the ``parser`` module makes it harder to >> switch to something more powerful than a LL(1) parser [7]_. Since the >> ``parser`` module is documented as deprecated since Python 2.5 and a new >> parsing technology is planned for 3.9, the ``parser`` module is scheduled >> for >> removal in 3.9. >> >> Module type >> C extension >> Deprecated in >> 3.8, documented as deprecated since **2.5** >> To be removed in >> **3.9** >> Substitute >> ast, lib2to3.pgen2 >> >> pipes >> ~~~~~ >> >> The `pipes `_ module >> provides >> helpers to pipe the input of one command into the output of another >> command. >> The module is built on top of ``os.popen``. Users are encouraged to use >> the subprocess module instead. >> >> Module type >> pure Python >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> subprocess module >> >> Removed modules >> =============== >> >> fpectl >> ------ >> >> The `fpectl `_ module >> was >> never built by default, its usage was discouraged and considered >> dangerous. >> It also required a configure flag that caused an ABI incompatibility. The >> module was removed in 3.7 by Nathaniel J. Smith in >> `bpo-29137 `_. >> >> Module type >> C extension + CAPI >> Deprecated in >> 3.7 >> Removed in >> 3.7 >> Substitute >> **none** >> >> >> Modules to keep >> =============== >> >> Some modules were originally proposed for deprecation. >> >> fileinput >> --------- >> >> The `fileinput `_ >> module >> implements a helpers to iterate over a list of files from ``sys.argv``. >> The >> module predates the optparser and argparser module. The same functionality >> can be implemented with the argparser module. >> >> Several core developers expressed their interest to keep the module in the >> standard library, as it is handy for quick scripts. >> >> Module type >> pure Python >> >> lib2to3 >> ------- >> >> The `lib2to3 `_ package >> provides >> the ``2to3`` command to transpile Python 2 code to Python 3 code. >> >> The package is useful for other tasks besides porting code from Python 2 >> to >> 3. For example `black`_ uses it for code reformatting. >> >> Module type >> pure Python >> >> getopt >> ------ >> >> The `getopt `_ module >> mimics >> C's getopt() option parser. >> >> Although users are encouraged to use argparse instead, the getopt module >> is >> still widely used. The module is small, simple, and handy for C developers >> to write simple Python scripts. >> >> Module type >> pure Python >> Substitute >> argparse >> >> optparse >> -------- >> >> The `optparse `_ module >> is >> the predecessor of the argparse module. >> >> Although it has been deprecated for many years, it's still too widely used >> to remove it. >> >> Module type >> pure Python >> Deprecated in >> 3.2 >> Substitute >> argparse >> >> wave >> ---- >> >> The `wave `_ module provides >> support for the WAV sound format. >> >> The module is not deprecated, because The WAV format is still relevant >> these >> days. The ``wave`` module is also used in education, e.g. to show kids how >> to make noise with a computer. >> >> The module uses one simple function from the `audioop`_ module to perform >> byte swapping between little and big endian formats. Before 24 bit WAV >> support was added, byte swap used to be implemented with the ``array`` >> module. To remove ``wave``'s dependency on the ``audioop``, the byte swap >> function could be either be moved to another module (e.g. ``operator``) or >> the ``array`` module could gain support for 24 bit (3 byte) arrays. >> >> Module type >> pure Python (depends on *byteswap* from `audioop`_ C extension) >> Deprecated in >> 3.8 >> To be removed in >> 3.10 >> Substitute >> *n/a* >> >> >> Future maintenance of removed modules >> ===================================== >> >> The main goal of the PEP is to reduce the burden and workload on the >> Python >> core developer team. Therefore removed modules will not be maintained by >> the core team as separate PyPI packages. However the removed code, tests >> and >> documentation may be moved into a new git repository, so community members >> have a place from which they can pick up and fork code. >> >> A first draft of a `legacylib `_ >> repository is available on my private Github account. The modules could be >> made available on PyPI. The Python core team will not publish or maintain >> the packages. It is my hope that members of the Python community will >> adopt, maintain, and perhaps improve the deprecated modules. >> >> It's my hope that some of the deprecated modules will be picked up and >> adopted by users that actually care about them. For example ``colorsys`` >> and >> ``imghdr`` are useful modules, but have limited feature set. A fork of >> ``imghdr`` can add new features and support for more image formats, >> without >> being constrained by Python's release cycle. >> >> Most of the modules are in pure Python and can be easily packaged. Some >> depend on a simple C module, e.g. `audioop`_ and `crypt`_. Since >> `audioop`_ >> does not depend on any external libraries, it can be shipped in as binary >> wheels with some effort. Other C modules can be replaced with ctypes or >> cffi. >> For example I created `legacycrypt > >`_ >> with ``_crypt`` extension reimplemented with a few lines of ctypes code. >> >> >> Discussions >> =========== >> >> * Elana Hashman and Nick Coghlan suggested to keep the *getopt* module. >> * Berker Peksag proposed to deprecate and removed *msilib*. >> * Brett Cannon recommended to delay active deprecation warnings and >> removal >> of modules like *imp* until Python 3.10. Version 3.8 will be released >> shortly before Python 2 reaches end of lifetime. A delay reduced churn >> for >> users that migrate from Python 2 to 3.8. >> * Brett also came up with the idea to keep lib2to3. The package is useful >> for other purposes, e.g. `black `_ >> uses >> it to reformat Python code. >> * At one point, distutils was mentioned in the same sentence as this PEP. >> To avoid lengthy discussion and delay of the PEP, I decided against >> dealing >> with distutils. Deprecation of the distutils package will be handled by >> another PEP. >> * Multiple people (Gregory P. Smith, David Beazley, Nick Coghlan, ...) >> convinced me to keep the `wave`_ module. [4]_ >> * Gregory P. Smith proposed to deprecate `nntplib`_. [4]_ >> * Andrew Svetlov mentioned the ``socketserver`` module is questionable. >> However it's used to implement ``http.server`` and ``xmlrpc.server``. >> The >> stdlib doesn't have a replacement for the servers, yet. >> >> >> Update history >> ============== >> >> Update 1 >> -------- >> >> * Deprecate `parser`_ module >> * Keep `fileinput`_ module >> * Elaborate why `crypt`_ and `spwd`_ are dangerous and bad >> * Improve sections for `cgitb`_, `colorsys`_, `nntplib`_, and `smtpd`_ >> modules >> * The `colorsys`_, `crypt`_, `imghdr`_, `sndhdr`_, and `spwd`_ sections >> now >> list suitable substitutions. >> * Mention that ``socketserver`` is going to stay for ``http.server`` and >> ``xmlrpc.server`` >> * The future maintenance section now states that the deprecated modules >> may be adopted by Python community members. >> >> References >> ========== >> >> .. [1] >> https://en.wikipedia.org/wiki/Open_Sound_System#Free,_proprietary,_free >> .. [2] https://man.openbsd.org/ossaudio >> .. [3] >> https://blogs.msmvps.com/installsite/blog/2015/05/03/the-future-of-windows-installer-msi-in-the-light-of-windows-10-and-the-universal-windows-platform/ >> .. [4] https://twitter.com/ChristianHeimes/status/1130257799475335169 >> .. [5] https://twitter.com/dabeaz/status/1130278844479545351 >> .. [6] https://mail.python.org/pipermail/python-dev/2019-May/157464.html >> .. [7] >> https://discuss.python.org/t/switch-pythons-parsing-tech-to-something-more-powerful-than-ll-1/379 >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/dfmoisset%40gmail.com >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun May 26 14:15:52 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 26 May 2019 14:15:52 -0400 Subject: [Python-Dev] Accepting PEP 587 (Python Initialization Configuration) In-Reply-To: References: Message-ID: On 5/26/2019 10:02 AM, Thomas Wouters wrote: > > As PEP-delegate, I'm accepting version 5 of PEP 587 (Python > Initialization Configuration). ... > Thanks, Victor, for pushing this massive amount of work through; And thanks to you for reviewing and approving. > it > won't affect most people using Python (at least until the rest of PEP > 432 lands) but it's a giant boon to those of us who embed Python :) It affects us when apps decided to embed or not embed Python as their exposed scripting language. The open-source Godot 2D/3D game engine started with Lua, switched to Squirrel (???), and tried to embed Python but ran into 'problems'. Instead they developed a very Python-like GDScript, with a subset of Python syntax and expanded built-ins. (So it is still easy for a Pythonista to use.) It might be worthwhile to ask them what the fatal problems were. -- Terry Jan Reedy From steve at pearwood.info Sun May 26 21:55:02 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 27 May 2019 11:55:02 +1000 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules In-Reply-To: References: <20190524065544.GS4221@ando.pearwood.info> Message-ID: <20190527015501.GC4221@ando.pearwood.info> On Sun, May 26, 2019 at 04:03:11PM +0300, Ivan Pozdeev via Python-Dev wrote: > On 24.05.2019 9:55, Steven D'Aprano wrote: > >I don't know if this is a good idea or a terrible idea or somewhere in > >between, so I'm throwing it out to see if anyone likes it. [...] > This would greately damage Python's image as a mature, reliable product, > something that you can bank on for your critical tasks. > > By including something into the offer, we implicitly endorse it and claim > that we want out users to use that rather than competing offers. Do we? I don't think that is the case for everything -- we certainly don't want people using private or undocumented functions, nor do we want them using deprecated code. [...] > You may argue that marking it as unsupported would do the same but it > wouldn't. This mean I'll have to be constantly on the lookout for hidden > notes and such, Is "from unmaintained import module" enough to tell you that the module is unmaintained? -- Steven From steve at pearwood.info Sun May 26 23:16:25 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 27 May 2019 13:16:25 +1000 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules In-Reply-To: <25c05229-7e8a-c8b7-8445-ccbb5836aecb@python.org> References: <20190524065544.GS4221@ando.pearwood.info> <25c05229-7e8a-c8b7-8445-ccbb5836aecb@python.org> Message-ID: <20190527031625.GD4221@ando.pearwood.info> Thanks Steve for your comments, I appreciate them. As I said I don't know if this is a good idea or not so please read my responses below as part of a friendly debate aimed at reaching consensus, not an argument. (The argument is in Room 12 with Mr. Barnard.) On Fri, May 24, 2019 at 09:54:05AM -0700, Steve Dower wrote: > On 23May2019 2355, Steven D'Aprano wrote: > >I don't know if this is a good idea or a terrible idea or somewhere in > >between, so I'm throwing it out to see if anyone likes it. > > > >Let's add a third option to PEP 594 between "keep" and "remove": > >explicitly flagging a module as unmaintained. Unmaintained modules: > > > >- will raise a warning when imported that they are unmaintained; > > Nobody reads warnings. If nobody reads warnings, we should just remove the warnings module and be done with it. That should probably be a PEP. > >- will have their tests disabled; > > I'm fine with this. Reducing time to run the test suite is the biggest > immediate win from removing unmaintained modules. Can somebody quantify how much time will be saved? Will we be saving 5 seconds in a 20 minute test suite, or 1 minute in a 2 minute test suite? > >- possibly we move them into a seperate namespace: > > ``from unmaintained import aardvark`` > > May as well move them all the way to PyPI and leave the import name the > same. No, that won't be the same. Keeping the import name the same is only valuable if we're not breaking backwards compatibility. If we remove the module from the std lib, we've broken backwards compatibility and people's code will still be broken, same name or not. > >- bug reports without patches will be closed Will Not Fix; > > By whom? By anyone with the ability to close bug reports. On bugs.python.org that's anyone with an account, you don't need to be a core developer, and some volunteers help out by triaging or closing poor-quality bug reports long before the core developers get involved. I assume the same applies on github -- if it doesn't, that's a huge usability regression. > >- bug reports with patches *may* be accepted if some core dev is > > willing to review and check it in, but there is no obligation > > to do so; > > Historically, nobody has been willing to review and merge these modules > for years. Is that true? That's certainly the *impression* people are giving, but the PEP is light on examples or details. The one example I looked into, binhex, didn't seem particularly troublesome. https://mail.python.org/pipermail/python-dev/2019-May/157673.html Although I may have missed some examples. > How long do we have to wait? How long do we wait before closing any other ticket? Less than that, because these would be unmaintained. > >- should it turn out that someone is willing to maintain the module, > > it can be returned to regular status. > > Or we can give them the keys to the PyPI package. Or help pip implement > the "Obsoleted By" header and redirect to a fork. Indeed we could. The question is whether we should. > >Plus side: > > > >- reduce the maintenance burden (if any) from the module; > > Apart from having to read, review, and decide on bug reports, CVEs, and > documentation. They're unmaintained. Nobody is promising to read or review bug reports. Volunteers will surely do the bulk of triaging (as I think they already do now). And besides, if it is correct that nobody is using these modules, who will be making the bug reports? > >- while still distributing the module and allowing users to use > > it: "no promises, but here you go"; > > a.k.a. PyPI? No, nothing like PyPI, for reasons discussed many times before including some of my earlier posts. Every Python user has access to whatever is in the std lib (exception: some Linux distros split out parts of the std lib into packages that need to be apt-installed or equivalent). Not all Python users can use third-party libraries on PyPI. Moving things to PyPI is an implicit decision to cut some users out. > >- other implementations are under no obligation to distribute > > unmaintained modules. > > a.k.a. PyPI? No, nothing like PyPI. > >Minus side: > > > >- this becomes a self-fulfilling prophesy: with tests turned off, > > bit-rot will eventually set in and break modules that currently > > aren't broken. > > True. And then we have the added maintenance burden of explaining > repeatedly that we don't care that it's broken, but we want to put it on > your machine anyway. That's not a "maintenance burden" since there's no maintenance involved. At worst its a small annoyance to the volunteers who close invalid bug reports on the tracker. If there is a flood of bug reports for unmaintained modules, that will be objective evidence that the module is broken and we can remove it for real. [...] > either the people who are saying "we rely on this" > are also willing to help us maintain them, or they're not. And if > they're not, they clearly don't rely on it (or realize the cost of > relying on volunteer-maintained software). That's a very privileged point of view. People contribute to Open Source software for many reasons -- the fame, the money, the power *wink* -- but surely its not *mandatory* that Python is only for those with the skills, time or money to contribute back. We might not have an obligation to maintain software forever, but to the degree that we do have an obligation to our users, we should consider those who cannot contribute. I'm suggesting that there's a middle step for modules, between being fully maintained and distributed, and being fully abandoned and removed, its users left to fend for themselves. And that is to formally say we aren't maintaining this any more, but we'll still distribute it until even that becomes a burden. If it helps, the path becomes: - fully maintained and distributed - deprecation warnings in docs and software - move to "unmaintained" status (distributed but not maintained) - cease distributing. With the last step only happening if and when there's objective evidence that the code is dead, or if closing bug reports becomes a burden, and nobody has stepped up in the intervening years with either money or time to fix the module. -- Steven From montana.burr at gmail.com Sun May 26 23:52:17 2019 From: montana.burr at gmail.com (Montana Burr) Date: Sun, 26 May 2019 21:52:17 -0600 Subject: [Python-Dev] Overloading comparison operator for lists Message-ID: NumPy arrays have this awesome feature, where array == 3 does an element-wise comparison and returns a list. For example: np.array([1,2,3,4,5])==3 returns [False,False,True,False,False] It would be cool if Python had similar functionality for lists. If that is not possible, perhaps we could consider allowing developers to overload operators on built-in types within the context of a project or module. For example, an overload in one module would have no effect on the same operator in a different module (such as any Python standard modules.) Additionally, let's then give the developers the option to explicitly import an overload from other modules. So, people could develop a module with the purpose of providing overloads that make complete sense within a certain context. -------------- next part -------------- An HTML attachment was scrubbed... URL: From cs at cskk.id.au Mon May 27 00:12:12 2019 From: cs at cskk.id.au (Cameron Simpson) Date: Mon, 27 May 2019 14:12:12 +1000 Subject: [Python-Dev] Overloading comparison operator for lists In-Reply-To: References: Message-ID: <20190527041212.GA67673@cskk.homeip.net> This belongs on python-ideas, not python-dev. I've directed replies to this message there. Comments below. On 26May2019 21:52, Montana Burr wrote: >NumPy arrays have this awesome feature, where array == 3 does an >element-wise comparison and returns a list. For example: > >np.array([1,2,3,4,5])==3 > >returns > >[False,False,True,False,False] >It would be cool if Python had similar functionality for lists. map(lamdba item: item==3, [1,2,3,4,5]) I'm not sure this rates extra Python features. Personally I'm -1 on this suggestion because == traditionally returns a Boolean, NumPy notwithstanding. Your example above doesn't return a Boolean. >If that is not possible, perhaps we could consider allowing developers >to overload operators on built-in types within the context of a project or >module. For example, an overload in one module would have no effect on the >same operator in a different module (such as any Python standard modules.) This is usally done by overloading dunder methods on classes. if you class subclasses a builtin eg int or list then the instances get the special behaviour. >Additionally, let's then give the developers the option to explicitly >import an overload from other modules. So, people could develop a module >with the purpose of providing overloads that make complete sense within a >certain context. If you go the subclass route you could do this with a mixin class (a class providing methods but little else, intended to be part of the MRO of a subclass). Cheers, Cameron Simpson From greg.ewing at canterbury.ac.nz Mon May 27 03:18:03 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 27 May 2019 19:18:03 +1200 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: <5CEB8F2B.6050300@canterbury.ac.nz> Chris Angelico wrote: > Except that it does. After calling locals() a second time, the result > of the *first* call will be updated to reflect changes. Yeow. That's *really* unintuitive. There had better be an extremely good reason for this behaviour. -- Greg From tjreedy at udel.edu Mon May 27 09:12:40 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 27 May 2019 09:12:40 -0400 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: <5CEB8F2B.6050300@canterbury.ac.nz> References: <5CEB8F2B.6050300@canterbury.ac.nz> Message-ID: On 5/27/2019 3:18 AM, Greg Ewing wrote: > Chris Angelico wrote: >> Except that it does. After calling locals() a second time, the result >> of the *first* call will be updated to reflect changes. > > Yeow. That's *really* unintuitive. There had better be an extremely > good reason for this behaviour. I believe that the situation is or can be thought of as this: there is exactly 1 function locals dict. Initially, it is empty and inaccessible (unusable) from code. Each locals() call updates the dict to a current snapshot and returns it. -- Terry Jan Reedy From mertz at gnosis.cx Mon May 27 09:27:33 2019 From: mertz at gnosis.cx (David Mertz) Date: Mon, 27 May 2019 09:27:33 -0400 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules In-Reply-To: <20190527031625.GD4221@ando.pearwood.info> References: <20190524065544.GS4221@ando.pearwood.info> <25c05229-7e8a-c8b7-8445-ccbb5836aecb@python.org> <20190527031625.GD4221@ando.pearwood.info> Message-ID: On Sun, May 26, 2019 at 11:17 PM Steven D'Aprano wrote: > > Nobody reads warnings. > > If nobody reads warnings, we should just remove the warnings module and > be done with it. That should probably be a PEP. > We'll have to start issuing a PendingDeprecationWarning when folk import the `warnings` module. :-) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From Richard at Damon-Family.org Mon May 27 09:52:41 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Mon, 27 May 2019 09:52:41 -0400 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: <5CEB8F2B.6050300@canterbury.ac.nz> Message-ID: On 5/27/19 9:12 AM, Terry Reedy wrote: > On 5/27/2019 3:18 AM, Greg Ewing wrote: >> Chris Angelico wrote: >>> Except that it does. After calling locals() a second time, the result >>> of the *first* call will be updated to reflect changes. >> >> Yeow. That's *really* unintuitive. There had better be an extremely >> good reason for this behaviour. > > I believe that the situation is or can be thought of as this: there is > exactly 1 function locals dict.? Initially, it is empty and > inaccessible (unusable) from code.? Each locals() call updates the > dict to a current snapshot and returns it. > I had a similar concern, and one BIG issue with it being define this way is that you get a fundamental re-entrancy problem. If module a uses locals(), and then calls module b that uses locals(), module a has lost its usage. One implication of this is that then you really want ALL modules to define if they use the locals() function or not, then you get the question, does this 1 of apply across threads? does a call to locals in another thread make me lose my locals (or does each thread get its own version), if that is true then if you might possible be in a situation where threads are in play you MUST make the copy anyway, and do it fast enough that the GIL isn't released between the snapshot and the copy (if possible), C made this sort of mistake decades ago for some functions, not thinking about threads or re-entrancy, and had to create solutions to fix it. Let us not forget history and thus repeat it. Is there a fundamental reason that local needs to keep a single dict, as opposed to creating a new one for each call? The way it is currently defined, once it is called, the snapshot will stay forever, consuming resources, while if a new dict was created, the resource would be reclaimed after use. Yes, if called twice you end up with two copies instead of both being updated to the current, but if you WANTED to just update the current copy, you could just rebind it to the new version, otherwise you are just forcing the programmer to be making the copies explicitly. -- Richard Damon From vano at mail.mipt.ru Mon May 27 09:59:28 2019 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Mon, 27 May 2019 16:59:28 +0300 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules In-Reply-To: <20190527015501.GC4221@ando.pearwood.info> References: <20190524065544.GS4221@ando.pearwood.info> <20190527015501.GC4221@ando.pearwood.info> Message-ID: <5867b9b0-653c-70a4-b8c0-4951a3177c33@mail.mipt.ru> On 27.05.2019 4:55, Steven D'Aprano wrote: > On Sun, May 26, 2019 at 04:03:11PM +0300, Ivan Pozdeev via Python-Dev wrote: >> On 24.05.2019 9:55, Steven D'Aprano wrote: >>> I don't know if this is a good idea or a terrible idea or somewhere in >>> between, so I'm throwing it out to see if anyone likes it. > [...] > >> This would greately damage Python's image as a mature, reliable product, >> something that you can bank on for your critical tasks. >> >> By including something into the offer, we implicitly endorse it and claim >> that we want out users to use that rather than competing offers. > Do we? I don't think that is the case for everything -- we certainly > don't want people using private or undocumented functions, nor do we > want them using deprecated code. But we do. We promise that everything works as documented. Even if something is deprecated, one is supposed to be able to rely on it while it's (still) in. > > [...] >> You may argue that marking it as unsupported would do the same but it >> wouldn't. This mean I'll have to be constantly on the lookout for hidden >> notes and such, > Is "from unmaintained import module" enough to tell you that the > module is unmaintained? This comes with its own set of drawbacks: * compatibility break * "unmaintained" is a judgement call, actual usability can change back and forth depending on the amount of interest * it's still a blow to Python's credibility if the team feels the need to lug a truckload of dead code around -- Regards, Ivan From vstinner at redhat.com Mon May 27 10:31:23 2019 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 27 May 2019 16:31:23 +0200 Subject: [Python-Dev] Accepting PEP 587 (Python Initialization Configuration) In-Reply-To: References: Message-ID: Hi, Thanks Thomas for accepting my PEP :-) Thanks everyone who was involved in discussions since it started July 2018! For the first version of my PEP 587, I tried to design an API as small as possible, but it was clear that it didn't fit all use cases. Slowly, I extended and adjusted the API to fix all use cases. The feedback was *very* useful! A special thank to Nick Coghlan, Steve Dower and Gregory Szorc!!! I'm now implementing the PEP ;-) I wrote 1000 lines of documentation (in Doc/c-api/init_config.rst, well it's mostly code copied from the PEP)! I will merge it before 3.8beta1. Victor Le dim. 26 mai 2019 ? 16:05, Thomas Wouters a ?crit : > > > As PEP-delegate, I'm accepting version 5 of PEP 587 (Python Initialization Configuration). I don't think Victor has posted the latest version, but it contains a couple of minor renames, and a new private API to allow us to experiment with two-phase initialisation of Python. > > There's still time to debate exact names and work on the wording of the PEP, but I'm satisfied the design is solid, a clear improvement over what we currently have, and a step in the right direction for some of the more extensive changes that have been proposed (e.g. using Python *code* for the Python configuration. If we do get to that point, the PEP contains a few things that are superfluous, but I'd consider them minor transitionary warts at that point -- if we ever get there). > > Thanks, Victor, for pushing this massive amount of work through; it won't affect most people using Python (at least until the rest of PEP 432 lands) but it's a giant boon to those of us who embed Python :) > > -- > Thomas Wouters > > Hi! I'm an email virus! Think twice before sending your email to help me spread! > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com -- Night gathers, and now my watch begins. It shall not end until my death. From solipsis at pitrou.net Mon May 27 11:05:32 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 27 May 2019 17:05:32 +0200 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules References: <20190524065544.GS4221@ando.pearwood.info> <25c05229-7e8a-c8b7-8445-ccbb5836aecb@python.org> <20190527031625.GD4221@ando.pearwood.info> Message-ID: <20190527170532.0801c945@fsol> On Mon, 27 May 2019 09:27:33 -0400 David Mertz wrote: > On Sun, May 26, 2019 at 11:17 PM Steven D'Aprano > wrote: > > > > Nobody reads warnings. > > > > If nobody reads warnings, we should just remove the warnings module and > > be done with it. That should probably be a PEP. > > > > We'll have to start issuing a PendingDeprecationWarning when folk import > the `warnings` module. :-) And message users that they can `pip install warnings` to get the independently-maintained version ;-) Regards Antoine. From guido at python.org Mon May 27 11:06:25 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 27 May 2019 08:06:25 -0700 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: <5CEB8F2B.6050300@canterbury.ac.nz> Message-ID: No, there's only one locals() dict *per stack frame*. So no worries about concurrency. On Mon, May 27, 2019 at 6:54 AM Richard Damon wrote: > On 5/27/19 9:12 AM, Terry Reedy wrote: > > On 5/27/2019 3:18 AM, Greg Ewing wrote: > >> Chris Angelico wrote: > >>> Except that it does. After calling locals() a second time, the result > >>> of the *first* call will be updated to reflect changes. > >> > >> Yeow. That's *really* unintuitive. There had better be an extremely > >> good reason for this behaviour. > > > > I believe that the situation is or can be thought of as this: there is > > exactly 1 function locals dict. Initially, it is empty and > > inaccessible (unusable) from code. Each locals() call updates the > > dict to a current snapshot and returns it. > > > I had a similar concern, and one BIG issue with it being define this way > is that you get a fundamental re-entrancy problem. If module a uses > locals(), and then calls module b that uses locals(), module a has lost > its usage. One implication of this is that then you really want ALL > modules to define if they use the locals() function or not, then you get > the question, does this 1 of apply across threads? does a call to locals > in another thread make me lose my locals (or does each thread get its > own version), if that is true then if you might possible be in a > situation where threads are in play you MUST make the copy anyway, and > do it fast enough that the GIL isn't released between the snapshot and > the copy (if possible), > > C made this sort of mistake decades ago for some functions, not thinking > about threads or re-entrancy, and had to create solutions to fix it. Let > us not forget history and thus repeat it. > > Is there a fundamental reason that local needs to keep a single dict, as > opposed to creating a new one for each call? The way it is currently > defined, once it is called, the snapshot will stay forever, consuming > resources, while if a new dict was created, the resource would be > reclaimed after use. Yes, if called twice you end up with two copies > instead of both being updated to the current, but if you WANTED to just > update the current copy, you could just rebind it to the new version, > otherwise you are just forcing the programmer to be making the copies > explicitly. > > -- > Richard Damon > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon May 27 11:15:01 2019 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 27 May 2019 08:15:01 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics Message-ID: First, I want to say: I'm very happy with PEP 558's changes to f_locals. It solves the weird threading bugs, and exposes the fundamental operations you need for debugging in a simple and clean way, while leaving a lot of implementation flexibility for future Python VMs. It's a huge improvement over what we had before. I'm not as sure about the locals() parts of the proposal. It might be fine, but there are some complex trade-offs here that I'm still trying to wrap my head around. The rest of this document is me thinking out loud to try to clarify these issues. ##### What are we trying to solve? There are two major questions, which are somewhat distinct: - What should the behavior of locals() be in CPython? - How much of that should be part of the language definition, vs CPython implementation details? The status quo is that for locals() inside function scope, the behavior is quite complex and subtle, and it's entirely implementation defined. In the current PEP draft, there are some small changes to the semantics, and also it promotes them becoming part of the official language semantics. I think the first question, about semantics, is the more important one. If we're promoting them to the language definition, the main effect is just to make it more important we get the semantics right. ##### What are the PEP's proposed semantics for locals()? They're kinda subtle. [Nick: please double-check this section, both for errors and because I think it includes some edge cases that the PEP currently doesn't mention.] For module/class scopes, locals() has always returned a mapping object which acts as a "source of truth" for the actual local environment ? mutating the environment directly changes the mapping object, and vice-versa. That's not going to change. In function scopes, things are more complicated. The *local environment* is conceptually well-defined, and includes: - local variables (current source of truth: "fast locals" array) - closed-over variables (current source of truth: cell objects) - any arbitrary key/values written to frame.f_locals that don't correspond to local or closed-over variables, e.g. you can do frame.f_locals[object()] = 10, and then later read it out again. However, the mapping returned by locals() does not directly reflect this local environment. Instead, each function frame has a dict associated with it. locals() returns this dict. The dict always holds any non-local/non-closed-over variables, and also, in certain circumstances, we write a snapshot of local and closed-over variables back into the dict. Specifically, we write back: - Whenever locals() is called - Whenever exec() or eval() is called without passing an explicit locals argument - After every trace/profile event, if a Python-level tracing/profiling function is registered. (Note: in CPython, the use of Python-level tracing/profiling functions is extremely rare. It's more common in alternative implementations like PyPy. For example, the coverage package uses a C-level tracing function on CPython, which does not trigger locals updates, but on PyPy it uses a Python-level tracing function, which does trigger updates.) In addition, the PEP doesn't say, but I think that any writes to f_locals immediately update both the environment and the locals dict. These semantics have some surprising consequences. Most obviously, in function scope (unlike other scopes), mutating locals() does not affect the actual local environment: def f(): a = 1 locals()["a"] = 2 assert a == 1 The writeback rules can also produce surprising results: def f(): loc1 = locals() # Since it's a snapshot created at the time of the call # to locals(), it doesn't contain 'loc1': assert "loc1" not in loc1 loc2 = locals() # Now loc1 has changed: assert "loc1" in loc1 However, the results here are totally different if a Python-level tracing/profiling function is installed ? in particular, the first assertion fails. The interaction between f_locals and and locals() is also subtle: def f(): a = 1 loc = locals() assert "loc" not in loc # Regular variable updates don't affect 'loc' a = 2 assert loc["a"] == 1 # But debugging updates do: sys._getframe().f_locals["a"] = 3 assert a == 3 assert loc["a"] == 3 # But it's not a full writeback assert "loc" not in loc # Mutating 'loc' doesn't affect f_locals: loc["a"] = 1 assert sys._getframe().f_locals["a"] == 1 # Except when it does: loc["b"] = 3 assert sys._getframe().f_locals["b"] == 3 Again, the results here are totally different if a Python-level tracing/profiling function is installed. And you can also hit these subtleties via 'exec' and 'eval': def f(): a = 1 loc = locals() assert "loc" not in loc # exec() triggers writeback, and then mutates the locals dict exec("a = 2; b = 3") # So now the current environment has been reflected into 'loc' assert "loc" in loc # Also loc["a"] has been changed to reflect the exec'ed assignments assert loc["a"] == 2 # But if we look at the actual environment, directly or via # f_locals, we can see that 'a' has not changed: assert a == 1 assert sys._getframe().f_locals["a"] == 1 # loc["b"] changed as well: assert loc["b"] == 3 # And this *does* show up in f_locals: assert sys._getframe().f_locals["b"] == 3 Of course, many of these edge cases are pretty obscure, so it's not clear how much they matter. But I think we can at least agree that this isn't the one obvious way to do it :-). ##### What's the landscape of possible semantics? I did some brainstorming, and came up with 4 sets of semantics that seem plausible enough to at least consider: - [PEP]: the semantics in the current PEP draft. - [PEP-minus-tracing]: same as [PEP], except dropping the writeback on Python-level trace/profile events. - [snapshot]: in function scope, each call to locals() returns a new, *static* snapshot of the local environment, removing all this writeback stuff. Something like: def locals(): frame = get_caller_frame() if is_function_scope(frame): # make a point-in-time copy of the "live" proxy object return dict(frame.f_locals) else: # in module/class scope, return the actual local environment return frame.f_locals - [proxy]: Simply return the .f_locals object, so in all contexts locals() returns a live mutable view of the actual environment: def locals(): return get_caller_frame().f_locals ##### How to evaluate our options? I can think of a lot of criteria that all-else-being-equal we would like Python to meet. (Of course, in practice they conflict.) Consistency across APIs: it's surprising if locals() and frame.f_locals do different things. This argues for [proxy]. Consistency across contexts: it's surprising if locals() has acts differently in module/class scope versus function scope. This argues for [proxy]. Consistent behavior when the environment shifts, or small maintenance changes are made: it's nice if code that works today keeps working tomorrow. On this criterion, I think [snapshot] > [proxy] > [PEP-minus-tracing] >>> [PEP]. [PEP] is particularly bad here because a very rare environmental change that almost no-one tests and mostly only happens when debugging (i.e., enabling tracing) causes a radical change in semantics. Simplicity of explaining to users: all else being equal, it's nice if our docs are short and clear and the language fits in your head. On this criterion, I think: [proxy] > [snapshot] > [PEP-minus-tracing] > [PEP]. As evidence that the current behavior is confusing, see: - Ned gets confused and writes a long blog post after he figures it out: https://nedbatchelder.com/blog/201211/tricky_locals.html - A linter that warns against mutating locals(): https://lgtm.com/rules/10030096/ Simplicity of implementation: "If the implementation is easy to explain, it may be a good idea." Since we need the proxy code anyway to implement f_locals, I think it's: [proxy] (free) > [snapshot] (one 'if' statement) > ([PEP] = [PEP-minus-tracing]). Impact on other interpreter implementations: all else being equal, we'd like to give new interpreters maximal freedom to do clever things. (And local variables are a place where language VMs often expend a lot of cleverness.) [proxy] and [snapshot] are both easily implemented in terms of f_locals, so they basically don't constrain alternative implementations at all. I'm not as sure about [PEP] and [PEP-minus-tracing]. I originally thought they must be horrible. On further thought, I'm not convinced they're *that* bad, since the need to support people doing silly stuff like frame.f_locals[object()] = 10 means that implementations will already need to sometimes attach something like a dict object to their function frames. But perhaps alternative implementations would like to disallow this, or are OK with making it really slow but care about locals() performance more. Anyway, it's definitely ([proxy] = [snapshot]) > ([PEP] = [PEP-minus-tracing]), but I'm not sure whether the '>' is large or small. Backwards compatibility: help(locals) says: NOTE: Whether or not updates to this dictionary will affect name lookups in the local scope and vice-versa is *implementation dependent* and not covered by any backwards compatibility guarantees. So that claims that there are ~no backwards compatibility issues here. I'm going to ignore that; no matter what the docs say, we still don't want to break everyone's code. And unfortunately, I can't think of any realistic way to do a gradual transition, with like deprecation warnings and all that (can anyone else?), so whatever we do will be a flag-day change. Of our four options, [PEP] is intuitively the closest to what CPython has traditionally done. But what exactly breaks under the different approaches? I'll split this off into its own section. ##### Backwards compatibility I'll split this into three parts: code that treats locals() as read-only, exec()/eval(), and code that mutates locals(). I believe (but haven't checked) that the majority of uses of locals() are in simple cases like: def f(): .... print("{a} {b}".format(**locals())) Luckily, this code remains totally fine under all four of our candidates. exec() and eval() are an interesting case. In Python 2, exec'ing some assignments actually *did* mutate the local environment, e.g. you could do: # Python 2 def f(): exec "a = 1" assert a == 1 In Python 3, this was changed, so now exec() inside a function cannot mutate the enclosing scope. We got some bug reports about this change, and there are a number of questions on stackoverflow about it, e.g.: - https://bugs.python.org/issue4831 - https://stackoverflow.com/questions/52217525/how-can-i-change-the-value-of-variable-in-a-exec-function - https://stackoverflow.com/questions/50995581/eval-exec-with-assigning-variable-python In all released versions of Python, eval() was syntactically unable to rebind variables, so eval()'s interaction with the local environment was undefined. However, in 3.8, eval() *will* be able to rebind variables using the ':=' operator, so this interaction will become user-visible. Presumably we'll want eval() to match exec(). OK, with that background out of the way, let's look at our candidates. If we adopt [proxy], then that will mean exec()-inside-functions will go back to the Python 2 behavior, where executing assignments in the enclosing scope actually changes the enclosing scope. This will likely break some code out there that's relying on the Python 3 behavior, though I don't know how common that is. (I'm guessing not too common? Using the same variable inside and outside an 'exec' and trusting that they *won't* be the same seems like an unusual thing to do. But I don't know.) With [PEP] and [PEP-minus-tracing], exec() is totally unchanged. With [snapshot], there's technically a small difference: if you call locals() and then exec(), the exec() no longer triggers an implicit writeback to the dict that locals() returned. I think we can ignore this, and say that for all three of these, exec() is unlikely to produce backwards compatibility issues. OK, finally, let's talk about code that calls locals() and then mutates the return value. The main difference between our candidates is how they handle mutation, so this seems like the most important case to focus on. Conveniently, Mark Shannon and friends have statically analyzed a large corpus of Python code and made a list of cases where people do this: https://lgtm.com/rules/10030096/alerts/ Thanks! I haven't gone through the whole list, but I read through the first few in the hopes of getting a better sense of what kind of code does this in the real world and how it would be impacted by our different options. https://lgtm.com/projects/g/pydata/xarray/snapshot/a2ac6af744584c8afed3d56d00c7d6ace85341d9/files/xarray/plot/plot.py?sort=name&dir=ASC&mode=heatmap#L701 Current: raises if a Python-level trace/profile function is set [PEP]: raises if a Python-level trace/profile function is set [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: always raises Comment: uses locals() to capture a bunch of passed in kwargs so it can pass them as **kwargs to another function, and treats them like a snapshot. The authors were clearly aware of the dangers, because this pattern appears multiple times in this file, and all the other places make an explicit copy of locals() before using it, but this place apparently got missed. Fix is trivial: just do that here too. https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-codegen/src/main/resources/python/api.mustache Current: ok [PEP]: ok [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: ok Comment: this is inside a tool used to generate Python wrappers for REST APIs. The vast majority of entries in the lgtm database are from code generated by this tool. This was tricky to analyze, because it's complex templated code, and it does mutate locals(). But I'm pretty confident that the mutations end up just... not mattering, and all possible generated code works under all of our candidates. Which is lucky, because if we did have to fix this it wouldn't be trivial: fixing up the code generator itself wouldn't be too hard, but it'll take years for everyone to regenerate their old wrappers. https://lgtm.com/projects/g/saltstack/salt/snapshot/bb0950e5eafbb897c8e969e3f20fd297d8ba2006/files/salt/utils/thin.py?sort=name&dir=ASC&mode=heatmap#L258 Current: ok [PEP]: ok [PEP-minus-tracing]: ok [snapshot]: raises [proxy]: ok Comment: the use of locals() here is totally superfluous ? it repeatedly reads and writes to locals()[mod], and in all cases this could be replaced by a simple variable, or any other dict. And it's careful not to assign to any name that matches an actual local variable, so it works fine with [proxy] too. But it does assume that multiple calls to locals() return the same dict, so [snapshot] breaks it. In this case the fix is trivial: just use a variable. https://lgtm.com/projects/g/materialsproject/pymatgen/snapshot/fd6900ed1040a4d35f2cf2b3506e6e3d7cdf77db/files/pymatgen/ext/jhu.py?sort=name&dir=ASC&mode=heatmap#L52 Current: buggy if a trace/profile function is set [PEP]: buggy if a trace/profile function is set [PEP-minus-tracing]: ok [snapshot]: ok [proxy]: raises Comment: Another example of collecting kwargs into a dict. Actually this code is always buggy, because they seem to think that dict.pop("a", "b") removes both "a" and "b" from the dict... but it would raise an exception on [proxy], because they use one of those local variables after popping it from locals(). Fix is trivial: take an explicit snapshot of locals() before modifying it. ##### Conclusion so far [PEP-minus-tracing] seems to strictly dominate [PEP]. It's equal or better on all criteria, and actually *more* compatible with all the legacy code I looked at, even though it's technically less consistent with what CPython used to do. Unless someone points out some new argument, I think we can reject the writeback-when-tracing part of the PEP draft. Choosing between the remaining three is more of a judgement call. I'm leaning towards saying that on net, [snapshot] beats [PEP-minus-tracing]: it's dramatically simpler, and the backwards incompatibilities that we've found so far seem pretty minor, on par with what we do in every point release. (In fact, in 3/4 of the cases I looked at, [snapshot] is actually what users seemed to trying to use in the first place.) For [proxy] versus [snapshot], a lot depends on what we think of changing the semantics of exec(). [proxy] is definitely more consistent and elegant, and if we could go back in time I think it's what we'd have done from the start. Its compatibility is maybe a bit worse than [snapshot] on non-exec() cases, but this seems pretty minor overall (it often doesn't matter, and if it does just write dict(locals()) instead of locals(), like you would in non-function scope). But the change in exec() semantics is an actual language change, even though it may not affect much real code, so that's what stands out for me. I'd very much like to hear about any considerations I've missed, and any opinions on the "judgement call" part. -- Nathaniel J. Smith -- https://vorpus.org From steve at holdenweb.com Mon May 27 11:25:55 2019 From: steve at holdenweb.com (Steve Holden) Date: Mon, 27 May 2019 16:25:55 +0100 Subject: [Python-Dev] PEP 594 - a proposal for unmaintained modules In-Reply-To: <20190527170532.0801c945@fsol> References: <20190524065544.GS4221@ando.pearwood.info> <25c05229-7e8a-c8b7-8445-ccbb5836aecb@python.org> <20190527031625.GD4221@ando.pearwood.info> <20190527170532.0801c945@fsol> Message-ID: This whole vexing issue isn't going to be solved with any simple fix. A tool that could identify upcoming trouble spots might or might not be helpful. Or perhaps it could be implemented as a __future__ feature, so that those who choose not to use it during development see no change. The primary effect of the import would be to put the import system into "PEP-594 advice mode," where any imports of potentially troublesome modules would at least give assiduous developers some idea of how much trouble to expect by emitting warnings (unless explicitly silenced). Even though only intended for development, it would inevitably make its way into production code, so there would still be plenty of room for bikeshedding about that ;-). On Mon, May 27, 2019 at 4:07 PM Antoine Pitrou wrote: > On Mon, 27 May 2019 09:27:33 -0400 > David Mertz wrote: > > On Sun, May 26, 2019 at 11:17 PM Steven D'Aprano > > wrote: > > > > > > Nobody reads warnings. > > > > > > If nobody reads warnings, we should just remove the warnings module and > > > be done with it. That should probably be a PEP. > > > > > > > We'll have to start issuing a PendingDeprecationWarning when folk import > > the `warnings` module. :-) > > And message users that they can `pip install warnings` to get the > independently-maintained version ;-) > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon May 27 12:12:25 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 27 May 2019 17:12:25 +0100 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: <5CEB8F2B.6050300@canterbury.ac.nz> Message-ID: On Mon, 27 May 2019 at 16:11, Guido van Rossum wrote: > > No, there's only one locals() dict *per stack frame*. So no worries about concurrency. Again, if the PEP is about formalising the behaviour as language mandated, this is worth documenting explicitly. Paul From guido at python.org Mon May 27 12:15:37 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 27 May 2019 09:15:37 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: Message-ID: I re-ran your examples and found that some of them fail. On Mon, May 27, 2019 at 8:17 AM Nathaniel Smith wrote: > First, I want to say: I'm very happy with PEP 558's changes to > f_locals. It solves the weird threading bugs, and exposes the > fundamental operations you need for debugging in a simple and clean > way, while leaving a lot of implementation flexibility for future > Python VMs. It's a huge improvement over what we had before. > > I'm not as sure about the locals() parts of the proposal. It might be > fine, but there are some complex trade-offs here that I'm still trying > to wrap my head around. The rest of this document is me thinking out > loud to try to clarify these issues. > > > ##### What are we trying to solve? > > There are two major questions, which are somewhat distinct: > - What should the behavior of locals() be in CPython? > - How much of that should be part of the language definition, vs > CPython implementation details? > > The status quo is that for locals() inside function scope, the > behavior is quite complex and subtle, and it's entirely implementation > defined. In the current PEP draft, there are some small changes to the > semantics, and also it promotes them becoming part of the official > language semantics. > > I think the first question, about semantics, is the more important > one. If we're promoting them to the language definition, the main > effect is just to make it more important we get the semantics right. > > > ##### What are the PEP's proposed semantics for locals()? > > They're kinda subtle. [Nick: please double-check this section, both > for errors and because I think it includes some edge cases that the > PEP currently doesn't mention.] > > For module/class scopes, locals() has always returned a mapping object > which acts as a "source of truth" for the actual local environment ? > mutating the environment directly changes the mapping object, and > vice-versa. That's not going to change. > > In function scopes, things are more complicated. The *local > environment* is conceptually well-defined, and includes: > - local variables (current source of truth: "fast locals" array) > - closed-over variables (current source of truth: cell objects) > - any arbitrary key/values written to frame.f_locals that don't > correspond to local or closed-over variables, e.g. you can do > frame.f_locals[object()] = 10, and then later read it out again. > > However, the mapping returned by locals() does not directly reflect > this local environment. Instead, each function frame has a dict > associated with it. locals() returns this dict. The dict always holds > any non-local/non-closed-over variables, and also, in certain > circumstances, we write a snapshot of local and closed-over variables > back into the dict. > > Specifically, we write back: > > - Whenever locals() is called > - Whenever exec() or eval() is called without passing an explicit > locals argument > - After every trace/profile event, if a Python-level tracing/profiling > function is registered. > > (Note: in CPython, the use of Python-level tracing/profiling functions > is extremely rare. It's more common in alternative implementations > like PyPy. For example, the coverage package uses a C-level tracing > function on CPython, which does not trigger locals updates, but on > PyPy it uses a Python-level tracing function, which does trigger > updates.) > > In addition, the PEP doesn't say, but I think that any writes to > f_locals immediately update both the environment and the locals dict. > > These semantics have some surprising consequences. Most obviously, in > function scope (unlike other scopes), mutating locals() does not > affect the actual local environment: > > def f(): > a = 1 > locals()["a"] = 2 > assert a == 1 > > The writeback rules can also produce surprising results: > > def f(): > loc1 = locals() > # Since it's a snapshot created at the time of the call > # to locals(), it doesn't contain 'loc1': > assert "loc1" not in loc1 > loc2 = locals() > # Now loc1 has changed: > assert "loc1" in loc1 > > However, the results here are totally different if a Python-level > tracing/profiling function is installed ? in particular, the first > assertion fails. > > The interaction between f_locals and and locals() is also subtle: > > def f(): > a = 1 > loc = locals() > assert "loc" not in loc > # Regular variable updates don't affect 'loc' > a = 2 > assert loc["a"] == 1 > # But debugging updates do: > sys._getframe().f_locals["a"] = 3 > assert a == 3 > That assert fails; `a` is still 2 here for me. > assert loc["a"] == 3 > # But it's not a full writeback > assert "loc" not in loc > # Mutating 'loc' doesn't affect f_locals: > loc["a"] = 1 > assert sys._getframe().f_locals["a"] == 1 > # Except when it does: > loc["b"] = 3 > assert sys._getframe().f_locals["b"] == 3 > All of this can be explained by realizing `loc is sys._getframe().f_locals`. IOW locals() always returns the dict in f_locals. > Again, the results here are totally different if a Python-level > tracing/profiling function is installed. > > And you can also hit these subtleties via 'exec' and 'eval': > > def f(): > a = 1 > loc = locals() > assert "loc" not in loc > # exec() triggers writeback, and then mutates the locals dict > exec("a = 2; b = 3") > # So now the current environment has been reflected into 'loc' > assert "loc" in loc > # Also loc["a"] has been changed to reflect the exec'ed assignments > assert loc["a"] == 2 > # But if we look at the actual environment, directly or via > # f_locals, we can see that 'a' has not changed: > assert a == 1 > assert sys._getframe().f_locals["a"] == 1 > # loc["b"] changed as well: > assert loc["b"] == 3 > # And this *does* show up in f_locals: > assert sys._getframe().f_locals["b"] == 3 > This works indeed. My understanding is that the bytecode interpreter, when accessing the value of a local variable, ignores f_locals and always uses the "fast" array. But exec() and eval() don't use fast locals, their code is always compiled as if it appears in a module-level scope. While the interpreter is running and no debugger is active, in a function scope f_locals is not used at all, the interpreter only interacts with the fast array and the cells. It is initialized by the first locals() call for a function scope, and locals() copies the fast array and the cells into it. Subsequent calls in the same function scope keep the same value for f_locals and re-copy fast and cells into it. This also clears out deleted local variables and emptied cells, but leaves "strange" keys (like "b" in the examples) unchanged. The truly weird case happen when Python-level tracers are present, then the contents of f_locals is written back to the fast array and cells at certain points. This is intended for use by pdb (am I the only user of pdb left in the world?), so one can step through a function and mutate local variables. I find this essential in some cases. > Of course, many of these edge cases are pretty obscure, so it's not > clear how much they matter. But I think we can at least agree that > this isn't the one obvious way to do it :-). > > > ##### What's the landscape of possible semantics? > > I did some brainstorming, and came up with 4 sets of semantics that > seem plausible enough to at least consider: > > - [PEP]: the semantics in the current PEP draft. > To be absolutely clear this copies the fast array and cells to f_locals when locals() is called, but never copies back, except when Python-level tracing/profiling is on. > - [PEP-minus-tracing]: same as [PEP], except dropping the writeback on > Python-level trace/profile events. > But this still copies the fast array and cells to f_locals when a Python trace function is called, right? It just doesn't write back. > - [snapshot]: in function scope, each call to locals() returns a new, > *static* snapshot of the local environment, removing all this > writeback stuff. Something like: > > def locals(): > frame = get_caller_frame() > if is_function_scope(frame): > # make a point-in-time copy of the "live" proxy object > return dict(frame.f_locals) > else: > # in module/class scope, return the actual local environment > return frame.f_locals > This is the most extreme variant, and in this case there is no point in having f_locals at all for a function scope (since nothing uses it). I'm not 100% sure that you understand this. > - [proxy]: Simply return the .f_locals object, so in all contexts > locals() returns a live mutable view of the actual environment: > > def locals(): > return get_caller_frame().f_locals > So this is PEP without any writeback. But there is still copying from the fast array and cells to f_locals. Does that only happen when locals() is called? Or also when a Python-level trace/profiling function is called? My problem with all variants except what's in the PEP is that it would leave pdb *no* way (short of calling into the C API using ctypes) of writing back local variables. > > ##### How to evaluate our options? > > I can think of a lot of criteria that all-else-being-equal we would > like Python to meet. (Of course, in practice they conflict.) > > Consistency across APIs: it's surprising if locals() and > frame.f_locals do different things. This argues for [proxy]. > > Consistency across contexts: it's surprising if locals() has acts > differently in module/class scope versus function scope. This argues > for [proxy]. > > Consistent behavior when the environment shifts, or small maintenance > changes are made: it's nice if code that works today keeps working > tomorrow. On this criterion, I think [snapshot] > [proxy] > > [PEP-minus-tracing] >>> [PEP]. [PEP] is particularly bad here because > a very rare environmental change that almost no-one tests and mostly > only happens when debugging (i.e., enabling tracing) causes a radical > change in semantics. > > Simplicity of explaining to users: all else being equal, it's nice if > our docs are short and clear and the language fits in your head. On > this criterion, I think: [proxy] > [snapshot] > [PEP-minus-tracing] > > [PEP]. As evidence that the current behavior is confusing, see: > > - Ned gets confused and writes a long blog post after he figures it > out: https://nedbatchelder.com/blog/201211/tricky_locals.html > - A linter that warns against mutating locals(): > https://lgtm.com/rules/10030096/ > > Simplicity of implementation: "If the implementation is easy to > explain, it may be a good idea." Since we need the proxy code anyway > to implement f_locals, I think it's: [proxy] (free) > [snapshot] (one > 'if' statement) > ([PEP] = [PEP-minus-tracing]). > > Impact on other interpreter implementations: all else being equal, > we'd like to give new interpreters maximal freedom to do clever > things. (And local variables are a place where language VMs often > expend a lot of cleverness.) [proxy] and [snapshot] are both easily > implemented in terms of f_locals, so they basically don't constrain > alternative implementations at all. I'm not as sure about [PEP] and > [PEP-minus-tracing]. I originally thought they must be horrible. On > further thought, I'm not convinced they're *that* bad, since the need > to support people doing silly stuff like frame.f_locals[object()] = 10 > means that implementations will already need to sometimes attach > something like a dict object to their function frames. But perhaps > alternative implementations would like to disallow this, or are OK > with making it really slow but care about locals() performance more. > Anyway, it's definitely ([proxy] = [snapshot]) > ([PEP] = > [PEP-minus-tracing]), but I'm not sure whether the '>' is large or > small. > > Backwards compatibility: help(locals) says: > > NOTE: Whether or not updates to this dictionary will affect > name lookups in the local scope and vice-versa is > *implementation dependent* and not covered by any backwards > compatibility guarantees. > > So that claims that there are ~no backwards compatibility issues here. > I'm going to ignore that; no matter what the docs say, we still don't > want to break everyone's code. And unfortunately, I can't think of any > realistic way to do a gradual transition, with like deprecation > warnings and all that (can anyone else?), so whatever we do will be a > flag-day change. > > Of our four options, [PEP] is intuitively the closest to what CPython > has traditionally done. But what exactly breaks under the different > approaches? I'll split this off into its own section. > > > ##### Backwards compatibility > > I'll split this into three parts: code that treats locals() as > read-only, exec()/eval(), and code that mutates locals(). > > I believe (but haven't checked) that the majority of uses of locals() > are in simple cases like: > > def f(): > .... > print("{a} {b}".format(**locals())) > > Luckily, this code remains totally fine under all four of our candidates. > > exec() and eval() are an interesting case. In Python 2, exec'ing some > assignments actually *did* mutate the local environment, e.g. you > could do: > > # Python 2 > def f(): > exec "a = 1" > assert a == 1 > > In Python 3, this was changed, so now exec() inside a function cannot > mutate the enclosing scope. We got some bug reports about this change, > and there are a number of questions on stackoverflow about it, e.g.: > > - https://bugs.python.org/issue4831 > - > https://stackoverflow.com/questions/52217525/how-can-i-change-the-value-of-variable-in-a-exec-function > - > https://stackoverflow.com/questions/50995581/eval-exec-with-assigning-variable-python > > In all released versions of Python, eval() was syntactically unable to > rebind variables, so eval()'s interaction with the local environment > was undefined. However, in 3.8, eval() *will* be able to rebind > variables using the ':=' operator, so this interaction will become > user-visible. Presumably we'll want eval() to match exec(). > > OK, with that background out of the way, let's look at our candidates. > > If we adopt [proxy], then that will mean exec()-inside-functions will > go back to the Python 2 behavior, where executing assignments in the > enclosing scope actually changes the enclosing scope. This will likely > break some code out there that's relying on the Python 3 behavior, > though I don't know how common that is. (I'm guessing not too common? > Using the same variable inside and outside an 'exec' and trusting that > they *won't* be the same seems like an unusual thing to do. But I > don't know.) > > With [PEP] and [PEP-minus-tracing], exec() is totally unchanged. With > [snapshot], there's technically a small difference: if you call > locals() and then exec(), the exec() no longer triggers an implicit > writeback to the dict that locals() returned. I think we can ignore > this, and say that for all three of these, exec() is unlikely to > produce backwards compatibility issues. > > OK, finally, let's talk about code that calls locals() and then > mutates the return value. The main difference between our candidates > is how they handle mutation, so this seems like the most important > case to focus on. > > Conveniently, Mark Shannon and friends have statically analyzed a > large corpus of Python code and made a list of cases where people do > this: https://lgtm.com/rules/10030096/alerts/ > Thanks! I haven't gone through the whole list, but I read through the > first few in the hopes of getting a better sense of what kind of code > does this in the real world and how it would be impacted by our > different options. > > > https://lgtm.com/projects/g/pydata/xarray/snapshot/a2ac6af744584c8afed3d56d00c7d6ace85341d9/files/xarray/plot/plot.py?sort=name&dir=ASC&mode=heatmap#L701 > Current: raises if a Python-level trace/profile function is set > [PEP]: raises if a Python-level trace/profile function is set > [PEP-minus-tracing]: ok > [snapshot]: ok > [proxy]: always raises > Comment: uses locals() to capture a bunch of passed in kwargs so it > can pass them as **kwargs to another function, and treats them like a > snapshot. The authors were clearly aware of the dangers, because this > pattern appears multiple times in this file, and all the other places > make an explicit copy of locals() before using it, but this place > apparently got missed. Fix is trivial: just do that here too. > > > https://github.com/swagger-api/swagger-codegen/blob/master/modules/swagger-codegen/src/main/resources/python/api.mustache > Current: ok > [PEP]: ok > [PEP-minus-tracing]: ok > [snapshot]: ok > [proxy]: ok > Comment: this is inside a tool used to generate Python wrappers for > REST APIs. The vast majority of entries in the lgtm database are from > code generated by this tool. This was tricky to analyze, because it's > complex templated code, and it does mutate locals(). But I'm pretty > confident that the mutations end up just... not mattering, and all > possible generated code works under all of our candidates. Which is > lucky, because if we did have to fix this it wouldn't be trivial: > fixing up the code generator itself wouldn't be too hard, but it'll > take years for everyone to regenerate their old wrappers. > > > https://lgtm.com/projects/g/saltstack/salt/snapshot/bb0950e5eafbb897c8e969e3f20fd297d8ba2006/files/salt/utils/thin.py?sort=name&dir=ASC&mode=heatmap#L258 > Current: ok > [PEP]: ok > [PEP-minus-tracing]: ok > [snapshot]: raises > [proxy]: ok > Comment: the use of locals() here is totally superfluous ? it > repeatedly reads and writes to locals()[mod], and in all cases this > could be replaced by a simple variable, or any other dict. And it's > careful not to assign to any name that matches an actual local > variable, so it works fine with [proxy] too. But it does assume that > multiple calls to locals() return the same dict, so [snapshot] breaks > it. In this case the fix is trivial: just use a variable. > > > https://lgtm.com/projects/g/materialsproject/pymatgen/snapshot/fd6900ed1040a4d35f2cf2b3506e6e3d7cdf77db/files/pymatgen/ext/jhu.py?sort=name&dir=ASC&mode=heatmap#L52 > Current: buggy if a trace/profile function is set > [PEP]: buggy if a trace/profile function is set > [PEP-minus-tracing]: ok > [snapshot]: ok > [proxy]: raises > Comment: Another example of collecting kwargs into a dict. Actually > this code is always buggy, because they seem to think that > dict.pop("a", "b") removes both "a" and "b" from the dict... but it > would raise an exception on [proxy], because they use one of those > local variables after popping it from locals(). Fix is trivial: take > an explicit snapshot of locals() before modifying it. > > > ##### Conclusion so far > > [PEP-minus-tracing] seems to strictly dominate [PEP]. It's equal or > better on all criteria, and actually *more* compatible with all the > legacy code I looked at, even though it's technically less consistent > with what CPython used to do. Unless someone points out some new > argument, I think we can reject the writeback-when-tracing part of the > PEP draft. > > Choosing between the remaining three is more of a judgement call. > > I'm leaning towards saying that on net, [snapshot] beats > [PEP-minus-tracing]: it's dramatically simpler, and the backwards > incompatibilities that we've found so far seem pretty minor, on par > with what we do in every point release. (In fact, in 3/4 of the cases > I looked at, [snapshot] is actually what users seemed to trying to use > in the first place.) > > For [proxy] versus [snapshot], a lot depends on what we think of > changing the semantics of exec(). [proxy] is definitely more > consistent and elegant, and if we could go back in time I think it's > what we'd have done from the start. Its compatibility is maybe a bit > worse than [snapshot] on non-exec() cases, but this seems pretty minor > overall (it often doesn't matter, and if it does just write > dict(locals()) instead of locals(), like you would in non-function > scope). But the change in exec() semantics is an actual language > change, even though it may not affect much real code, so that's what > stands out for me. > > I'd very much like to hear about any considerations I've missed, and > any opinions on the "judgement call" part. > To me it looks as if you're working with the wrong mental model of how it currently works. Unfortunately there's no way to find out how it currently works without reading the source code, since from Python code you cannot access the fast array and cells directly, and by just observing f_locals you get a slightly wrong picture. Perhaps we should give the two operations that copy from/to the fast array and cells names, so we can talk about them more easily. Sorry I didn't snip stuff. -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Mon May 27 12:41:36 2019 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 27 May 2019 09:41:36 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: Message-ID: On Mon, May 27, 2019 at 9:16 AM Guido van Rossum wrote: > > I re-ran your examples and found that some of them fail. > > On Mon, May 27, 2019 at 8:17 AM Nathaniel Smith wrote: [...] >> The interaction between f_locals and and locals() is also subtle: >> >> def f(): >> a = 1 >> loc = locals() >> assert "loc" not in loc >> # Regular variable updates don't affect 'loc' >> a = 2 >> assert loc["a"] == 1 >> # But debugging updates do: >> sys._getframe().f_locals["a"] = 3 >> assert a == 3 > > > That assert fails; `a` is still 2 here for me. I think you're running on current Python, and I'm talking about the semantics in the current PEP 558 draft, which redefines f_locals so that the assert passes. Nick has a branch here if you want to try it: https://github.com/python/cpython/pull/3640 (Though I admit I was lazy, and haven't tried running my examples at all -- they're just based on the text.) >> >> assert loc["a"] == 3 >> # But it's not a full writeback >> assert "loc" not in loc >> # Mutating 'loc' doesn't affect f_locals: >> loc["a"] = 1 >> assert sys._getframe().f_locals["a"] == 1 >> # Except when it does: >> loc["b"] = 3 >> assert sys._getframe().f_locals["b"] == 3 > > > All of this can be explained by realizing `loc is sys._getframe().f_locals`. IOW locals() always returns the dict in f_locals. That's not true in the PEP version of things. locals() and frame.f_locals become radically different. locals() is still a dict stored in the frame object, but f_locals is a magic proxy object that reads/writes to the fast locals array directly. >> >> Again, the results here are totally different if a Python-level >> tracing/profiling function is installed. >> >> And you can also hit these subtleties via 'exec' and 'eval': >> >> def f(): >> a = 1 >> loc = locals() >> assert "loc" not in loc >> # exec() triggers writeback, and then mutates the locals dict >> exec("a = 2; b = 3") >> # So now the current environment has been reflected into 'loc' >> assert "loc" in loc >> # Also loc["a"] has been changed to reflect the exec'ed assignments >> assert loc["a"] == 2 >> # But if we look at the actual environment, directly or via >> # f_locals, we can see that 'a' has not changed: >> assert a == 1 >> assert sys._getframe().f_locals["a"] == 1 >> # loc["b"] changed as well: >> assert loc["b"] == 3 >> # And this *does* show up in f_locals: >> assert sys._getframe().f_locals["b"] == 3 > > > This works indeed. My understanding is that the bytecode interpreter, when accessing the value of a local variable, ignores f_locals and always uses the "fast" array. But exec() and eval() don't use fast locals, their code is always compiled as if it appears in a module-level scope. > > While the interpreter is running and no debugger is active, in a function scope f_locals is not used at all, the interpreter only interacts with the fast array and the cells. It is initialized by the first locals() call for a function scope, and locals() copies the fast array and the cells into it. Subsequent calls in the same function scope keep the same value for f_locals and re-copy fast and cells into it. This also clears out deleted local variables and emptied cells, but leaves "strange" keys (like "b" in the examples) unchanged. > > The truly weird case happen when Python-level tracers are present, then the contents of f_locals is written back to the fast array and cells at certain points. This is intended for use by pdb (am I the only user of pdb left in the world?), so one can step through a function and mutate local variables. I find this essential in some cases. Right, the original goal for the PEP was to remove the "truly weird case" but keep pdb working >> >> Of course, many of these edge cases are pretty obscure, so it's not >> clear how much they matter. But I think we can at least agree that >> this isn't the one obvious way to do it :-). >> >> >> ##### What's the landscape of possible semantics? >> >> I did some brainstorming, and came up with 4 sets of semantics that >> seem plausible enough to at least consider: >> >> - [PEP]: the semantics in the current PEP draft. > > > To be absolutely clear this copies the fast array and cells to f_locals when locals() is called, but never copies back, except when Python-level tracing/profiling is on. In the PEP draft, it never copies back at all, under any circumstance. >> >> - [PEP-minus-tracing]: same as [PEP], except dropping the writeback on >> Python-level trace/profile events. > > > But this still copies the fast array and cells to f_locals when a Python trace function is called, right? It just doesn't write back. No, when I say "writeback" in this email I always mean PyFrame_FastToLocals. The PEP removes PyFrame_LocalsToFast entirely. >> - [snapshot]: in function scope, each call to locals() returns a new, >> *static* snapshot of the local environment, removing all this >> writeback stuff. Something like: >> >> def locals(): >> frame = get_caller_frame() >> if is_function_scope(frame): >> # make a point-in-time copy of the "live" proxy object >> return dict(frame.f_locals) >> else: >> # in module/class scope, return the actual local environment >> return frame.f_locals > > > This is the most extreme variant, and in this case there is no point in having f_locals at all for a function scope (since nothing uses it). I'm not 100% sure that you understand this. Yes, this does suggest an optimization: you should be able to skip allocating a dict for every frame in most cases. I'm not sure how much of a difference it makes. In principle we could implement that optimization right now by delaying the dict allocation until the first time f_locals or locals() is used, but we currently don't bother. And even if we adopt this, we'll still need to keep a slot in the frame struct to allocate the dict if we need to, because people can still be obnoxious and do frame.f_locals["unique never before seen name"] = blah and expect to be able to read it back later, which means we need somewhere to store that. (In fact Trio does do this right now, as part of its control-C handling stuff, because there's literally no other place where you can store information that a signal handler can see when it's walking the stack.) We could deprecate writing new names to f_locals like this, but that's a longer-term thing. >> >> - [proxy]: Simply return the .f_locals object, so in all contexts >> locals() returns a live mutable view of the actual environment: >> >> def locals(): >> return get_caller_frame().f_locals > > > So this is PEP without any writeback. But there is still copying from the fast array and cells to f_locals. Does that only happen when locals() is called? Or also when a Python-level trace/profiling function is called? > My problem with all variants except what's in the PEP is that it would leave pdb *no* way (short of calling into the C API using ctypes) of writing back local variables. No, this option is called [proxy] because this is the version where locals() and f_locals *both* give you magic proxy objects where __getitem__ and __setitem__ access the fast locals array directly, as compared to the PEP where only f_locals gives you that magic object. -n -- Nathaniel J. Smith -- https://vorpus.org From tjreedy at udel.edu Mon May 27 14:05:32 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 27 May 2019 14:05:32 -0400 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: <5CEB8F2B.6050300@canterbury.ac.nz> Message-ID: On 5/27/2019 9:52 AM, Richard Damon wrote: > On 5/27/19 9:12 AM, Terry Reedy wrote: >> I believe that the situation is or can be thought of as this: there is >> exactly 1 function locals dict. per function invocation, or more generally, as Guido said, per stack frame. This part is obvious to me, but I should have been clearer. >>? Initially, it is empty and >> inaccessible (unusable) from code.? Each locals() call updates the >> dict to a current snapshot and returns it. >> > I had a similar concern, and one BIG issue with it being define this way > is that you get a fundamental re-entrancy problem. If module a uses > locals(), and then calls module b that uses locals(), module a has lost > its usage. No. Sorry about being unclear. -- Terry Jan Reedy From Richard at Damon-Family.org Mon May 27 15:08:55 2019 From: Richard at Damon-Family.org (Richard Damon) Date: Mon, 27 May 2019 15:08:55 -0400 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: <5CEB8F2B.6050300@canterbury.ac.nz> Message-ID: <723d1f0a-1b36-0e09-6e1c-f0b1df4f1410@Damon-Family.org> On 5/27/19 2:05 PM, Terry Reedy wrote: > On 5/27/2019 9:52 AM, Richard Damon wrote: >> On 5/27/19 9:12 AM, Terry Reedy wrote: > >>> I believe that the situation is or can be thought of as this: there is >>> exactly 1 function locals dict. > > per function invocation, or more generally, as Guido said, per stack > frame.? This part is obvious to me, but I should have been clearer. > >>> ? Initially, it is empty and >>> inaccessible (unusable) from code.? Each locals() call updates the >>> dict to a current snapshot and returns it. >>> >> I had a similar concern, and one BIG issue with it being define this way >> is that you get a fundamental re-entrancy problem. If module a uses >> locals(), and then calls module b that uses locals(), module a has lost >> its usage. > > No. Sorry about being unclear. > Ok, if each function invocation gets its own dict, then the reentrancy issues go away. The fact that it isn't the 'active' dict, so you can't use it to modify the current state, but also you don't get a fresh copy each time (there is a single mutable dict that gets updated) seems to be an odd behavior and I can't see where it is an advantage to the user of the function, or where that makes it easier on the implementation. (But I could easy be missing something). -- Richard Damon From guido at python.org Mon May 27 15:30:21 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 27 May 2019 12:30:21 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: Message-ID: OK, I apologize for not catching on to the changed semantics of f_locals (which in the proposal is always a proxy for the fast locals and the cells). I don't know if I just skimmed that part of the PEP or that it needs calling out more. I'm assuming that there are backwards compatibility concerns, and the PEP is worried that more code will break if one of the simpler options is chosen. In particular I guess that the choice of returning the same object is meant to make locals() in a function frame more similar to locals() in a module frame. And the choice of returning a plain dict rather than a proxy is meant to make life easier for code that assumes it's getting a plain dict. Other than that I agree that returning a proxy (i.e. just a reference to f_locals) seems to be the most attractive option... On Mon, May 27, 2019 at 9:41 AM Nathaniel Smith wrote: > On Mon, May 27, 2019 at 9:16 AM Guido van Rossum wrote: > > > > I re-ran your examples and found that some of them fail. > > > > On Mon, May 27, 2019 at 8:17 AM Nathaniel Smith wrote: > [...] > >> The interaction between f_locals and and locals() is also subtle: > >> > >> def f(): > >> a = 1 > >> loc = locals() > >> assert "loc" not in loc > >> # Regular variable updates don't affect 'loc' > >> a = 2 > >> assert loc["a"] == 1 > >> # But debugging updates do: > >> sys._getframe().f_locals["a"] = 3 > >> assert a == 3 > > > > > > That assert fails; `a` is still 2 here for me. > > I think you're running on current Python, and I'm talking about the > semantics in the current PEP 558 draft, which redefines f_locals so > that the assert passes. Nick has a branch here if you want to try it: > https://github.com/python/cpython/pull/3640 > > (Though I admit I was lazy, and haven't tried running my examples at > all -- they're just based on the text.) > > >> > >> assert loc["a"] == 3 > >> # But it's not a full writeback > >> assert "loc" not in loc > >> # Mutating 'loc' doesn't affect f_locals: > >> loc["a"] = 1 > >> assert sys._getframe().f_locals["a"] == 1 > >> # Except when it does: > >> loc["b"] = 3 > >> assert sys._getframe().f_locals["b"] == 3 > > > > > > All of this can be explained by realizing `loc is > sys._getframe().f_locals`. IOW locals() always returns the dict in f_locals. > > That's not true in the PEP version of things. locals() and > frame.f_locals become radically different. locals() is still a dict > stored in the frame object, but f_locals is a magic proxy object that > reads/writes to the fast locals array directly. > > >> > >> Again, the results here are totally different if a Python-level > >> tracing/profiling function is installed. > >> > >> And you can also hit these subtleties via 'exec' and 'eval': > >> > >> def f(): > >> a = 1 > >> loc = locals() > >> assert "loc" not in loc > >> # exec() triggers writeback, and then mutates the locals dict > >> exec("a = 2; b = 3") > >> # So now the current environment has been reflected into 'loc' > >> assert "loc" in loc > >> # Also loc["a"] has been changed to reflect the exec'ed > assignments > >> assert loc["a"] == 2 > >> # But if we look at the actual environment, directly or via > >> # f_locals, we can see that 'a' has not changed: > >> assert a == 1 > >> assert sys._getframe().f_locals["a"] == 1 > >> # loc["b"] changed as well: > >> assert loc["b"] == 3 > >> # And this *does* show up in f_locals: > >> assert sys._getframe().f_locals["b"] == 3 > > > > > > This works indeed. My understanding is that the bytecode interpreter, > when accessing the value of a local variable, ignores f_locals and always > uses the "fast" array. But exec() and eval() don't use fast locals, their > code is always compiled as if it appears in a module-level scope. > > > > While the interpreter is running and no debugger is active, in a > function scope f_locals is not used at all, the interpreter only interacts > with the fast array and the cells. It is initialized by the first locals() > call for a function scope, and locals() copies the fast array and the cells > into it. Subsequent calls in the same function scope keep the same value > for f_locals and re-copy fast and cells into it. This also clears out > deleted local variables and emptied cells, but leaves "strange" keys (like > "b" in the examples) unchanged. > > > > The truly weird case happen when Python-level tracers are present, then > the contents of f_locals is written back to the fast array and cells at > certain points. This is intended for use by pdb (am I the only user of pdb > left in the world?), so one can step through a function and mutate local > variables. I find this essential in some cases. > > Right, the original goal for the PEP was to remove the "truly weird > case" but keep pdb working > > >> > >> Of course, many of these edge cases are pretty obscure, so it's not > >> clear how much they matter. But I think we can at least agree that > >> this isn't the one obvious way to do it :-). > >> > >> > >> ##### What's the landscape of possible semantics? > >> > >> I did some brainstorming, and came up with 4 sets of semantics that > >> seem plausible enough to at least consider: > >> > >> - [PEP]: the semantics in the current PEP draft. > > > > > > To be absolutely clear this copies the fast array and cells to f_locals > when locals() is called, but never copies back, except when Python-level > tracing/profiling is on. > > In the PEP draft, it never copies back at all, under any circumstance. > > >> > >> - [PEP-minus-tracing]: same as [PEP], except dropping the writeback on > >> Python-level trace/profile events. > > > > > > But this still copies the fast array and cells to f_locals when a Python > trace function is called, right? It just doesn't write back. > > No, when I say "writeback" in this email I always mean > PyFrame_FastToLocals. The PEP removes PyFrame_LocalsToFast entirely. > > >> - [snapshot]: in function scope, each call to locals() returns a new, > >> *static* snapshot of the local environment, removing all this > >> writeback stuff. Something like: > >> > >> def locals(): > >> frame = get_caller_frame() > >> if is_function_scope(frame): > >> # make a point-in-time copy of the "live" proxy object > >> return dict(frame.f_locals) > >> else: > >> # in module/class scope, return the actual local environment > >> return frame.f_locals > > > > > > This is the most extreme variant, and in this case there is no point in > having f_locals at all for a function scope (since nothing uses it). I'm > not 100% sure that you understand this. > > Yes, this does suggest an optimization: you should be able to skip > allocating a dict for every frame in most cases. I'm not sure how much > of a difference it makes. In principle we could implement that > optimization right now by delaying the dict allocation until the first > time f_locals or locals() is used, but we currently don't bother. And > even if we adopt this, we'll still need to keep a slot in the frame > struct to allocate the dict if we need to, because people can still be > obnoxious and do frame.f_locals["unique never before seen name"] = > blah and expect to be able to read it back later, which means we need > somewhere to store that. (In fact Trio does do this right now, as part > of its control-C handling stuff, because there's literally no other > place where you can store information that a signal handler can see > when it's walking the stack.) We could deprecate writing new names to > f_locals like this, but that's a longer-term thing. > > >> > >> - [proxy]: Simply return the .f_locals object, so in all contexts > >> locals() returns a live mutable view of the actual environment: > >> > >> def locals(): > >> return get_caller_frame().f_locals > > > > > > So this is PEP without any writeback. But there is still copying from > the fast array and cells to f_locals. Does that only happen when locals() > is called? Or also when a Python-level trace/profiling function is called? > > My problem with all variants except what's in the PEP is that it would > leave pdb *no* way (short of calling into the C API using ctypes) of > writing back local variables. > > No, this option is called [proxy] because this is the version where > locals() and f_locals *both* give you magic proxy objects where > __getitem__ and __setitem__ access the fast locals array directly, as > compared to the PEP where only f_locals gives you that magic object. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon May 27 22:28:48 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 May 2019 12:28:48 +1000 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: Message-ID: <20190528022848.GF4221@ando.pearwood.info> On Mon, May 27, 2019 at 08:15:01AM -0700, Nathaniel Smith wrote: [...] > I'm not as sure about the locals() parts of the proposal. It might be > fine, but there are some complex trade-offs here that I'm still trying > to wrap my head around. The rest of this document is me thinking out > loud to try to clarify these issues. Wow. Thanks for the detail on this, I think the PEP should link to this thread, you've done some great work here. [...] > In function scopes, things are more complicated. The *local > environment* is conceptually well-defined, and includes: > - local variables (current source of truth: "fast locals" array) > - closed-over variables (current source of truth: cell objects) I don't think closed-over variables are *local* variables. They're "nonlocal", and you need a special keyword to write to them. > - any arbitrary key/values written to frame.f_locals that don't > correspond to local or closed-over variables, e.g. you can do > frame.f_locals[object()] = 10, and then later read it out again. Today I learned something new. > However, the mapping returned by locals() does not directly reflect > this local environment. Instead, each function frame has a dict > associated with it. locals() returns this dict. The dict always holds > any non-local/non-closed-over variables, and also, in certain > circumstances, we write a snapshot of local and closed-over variables > back into the dict. I'm going to try to make a case for your "snapshot" scenario. The locals dict inside a function is rather weird: - unlike in the global or class scope, writing to the dict does not update the variables - writing to it is discouraged, but its not a read-only proxy - it seems to be a snapshot of the state of variables when you called locals(): # inside a function x = 1 d = locals() x = 2 assert d['x'] == 1 - but it's not a proper, static, snapshot, because sometimes it will mutate without you touching it. That last point is Action At A Distance, and while it is explicable ("there's only one locals dict, and calling locals() updates it") its also rather unintuitive and surprising and violates the Principle Of Least Surprise. [Usual disclaimers about "surprising to whom?" applies.] Unless I missed something, it doesn't seem that any of the code you (Nathan) analysed is making use of this AAAD behaviour, at least not deliberately. At least one of the examples took steps to avoid it by making an explicit copy after calling locals(), but missed one leaving that function possibly buggy. Given how weirdly the locals dict behaves, and how tricky it is to explain all the corner cases, I'm going to +1 your "snapshot" idea: - we keep the current behaviour for locals() in the global and class scopes; - we keep the PEP's behaviour for writebacks when locals() or exec() (and eval with walrus operator) are called, for the frame dict; - but we change locals() to return a copy of that dict, rather than the dict itself. (I think I've got the details right... please correct me if I've misunderstood anything.) Being a backwards-incompatible change, that means that folks who were relying on that automagical refresh of the snapshot will need to change their code to explicitly refresh: # update in place: d.update(locals()) # or get a new snapshot d = locals() Or they explicitly grab a reference to the frame dict instead of calling locals(). Either way is likely to be less surprising than the status quo and less likely to lead to accidental, unexpected updates of the local dictionary without your knowledge. [...] > Of course, many of these edge cases are pretty obscure, so it's not > clear how much they matter. But I think we can at least agree that > this isn't the one obvious way to do it :-). Indeed. I thought I was doing well to know that writing to locals() inside a function didn't necessarily update the variable, but I had no idea of the levels of complexity actually involved! > I can think of a lot of criteria that all-else-being-equal we would > like Python to meet. (Of course, in practice they conflict.) > > Consistency across APIs: it's surprising if locals() and > frame.f_locals do different things. This argues for [proxy]. I don't think it is that surprising, since frame.f_locals is kinda obscure (the average Python coder wouldn't know a frame if one fell on them) and locals() has been documented as weird for decades. In any case, at least "its a copy of ..." is simple and understandable. > Consistency across contexts: it's surprising if locals() has acts > differently in module/class scope versus function scope. This argues > for [proxy]. True as far as it goes, but its also true that for the longest time, in most implementations, locals() has acted differently. So no change there. On the other hand, locals() currently returns a dict everywhere. It might be surprising for it to start returning a proxy object inside functions instead of a dict. -- Steven From v+python at g.nevcal.com Mon May 27 23:53:55 2019 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 27 May 2019 20:53:55 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: <20190528022848.GF4221@ando.pearwood.info> References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: <34234c7a-c616-2508-7d4a-7a435d92ed5e@g.nevcal.com> On 5/27/2019 7:28 PM, Steven D'Aprano wrote: > On the other hand, locals() currently returns a dict everywhere. It > might be surprising for it to start returning a proxy object inside > functions instead of a dict. I thought the proxy object sounded more useful... how different is it in use from a dict? "proxy" sounds like it should quack like a dict, as a general term, but maybe a more specific "proxy" is meant here, that doesn't quite quack like a dict? -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue May 28 00:17:23 2019 From: guido at python.org (Guido van Rossum) Date: Mon, 27 May 2019 21:17:23 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: <20190528022848.GF4221@ando.pearwood.info> References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: Note that the weird, Action At A Distance behavior is also visible for locals() called at module scope (since there, locals() is globals(), which returns the actual dict that's the module's __dict__, i.e. the Source Of Truth. So I think it's unavoidable in general, and we would do wise not to try and "fix" it just for function locals. (And I certainly don't want to mess with globals().) This is another case for the proposed [proxy] semantics, assuming we can get over our worry about backwards incompatibility. On Mon, May 27, 2019 at 7:31 PM Steven D'Aprano wrote: > On Mon, May 27, 2019 at 08:15:01AM -0700, Nathaniel Smith wrote: > [...] > > I'm not as sure about the locals() parts of the proposal. It might be > > fine, but there are some complex trade-offs here that I'm still trying > > to wrap my head around. The rest of this document is me thinking out > > loud to try to clarify these issues. > > Wow. Thanks for the detail on this, I think the PEP should link to this > thread, you've done some great work here. > > > [...] > > In function scopes, things are more complicated. The *local > > environment* is conceptually well-defined, and includes: > > - local variables (current source of truth: "fast locals" array) > > - closed-over variables (current source of truth: cell objects) > > I don't think closed-over variables are *local* variables. They're > "nonlocal", and you need a special keyword to write to them. > > > > - any arbitrary key/values written to frame.f_locals that don't > > correspond to local or closed-over variables, e.g. you can do > > frame.f_locals[object()] = 10, and then later read it out again. > > Today I learned something new. > > > > However, the mapping returned by locals() does not directly reflect > > this local environment. Instead, each function frame has a dict > > associated with it. locals() returns this dict. The dict always holds > > any non-local/non-closed-over variables, and also, in certain > > circumstances, we write a snapshot of local and closed-over variables > > back into the dict. > > I'm going to try to make a case for your "snapshot" scenario. > > The locals dict inside a function is rather weird: > > - unlike in the global or class scope, writing to the dict does not > update the variables > > - writing to it is discouraged, but its not a read-only proxy > > - it seems to be a snapshot of the state of variables when you > called locals(): > > # inside a function > x = 1 > d = locals() > x = 2 > assert d['x'] == 1 > > - but it's not a proper, static, snapshot, because sometimes it > will mutate without you touching it. > > That last point is Action At A Distance, and while it is explicable > ("there's only one locals dict, and calling locals() updates it") its > also rather unintuitive and surprising and violates the Principle Of > Least Surprise. > > [Usual disclaimers about "surprising to whom?" applies.] > > Unless I missed something, it doesn't seem that any of the code you > (Nathan) analysed is making use of this AAAD behaviour, at least not > deliberately. At least one of the examples took steps to avoid it by > making an explicit copy after calling locals(), but missed one leaving > that function possibly buggy. > > Given how weirdly the locals dict behaves, and how tricky it is to > explain all the corner cases, I'm going to +1 your "snapshot" idea: > > - we keep the current behaviour for locals() in the global and class > scopes; > > - we keep the PEP's behaviour for writebacks when locals() or exec() > (and eval with walrus operator) are called, for the frame dict; > > - but we change locals() to return a copy of that dict, rather than > the dict itself. > > (I think I've got the details right... please correct me if I've > misunderstood anything.) > > Being a backwards-incompatible change, that means that folks who were > relying on that automagical refresh of the snapshot will need to change > their code to explicitly refresh: > > # update in place: > d.update(locals()) > > # or get a new snapshot > d = locals() > > > Or they explicitly grab a reference to the frame dict instead of > calling locals(). Either way is likely to be less surprising than the > status quo and less likely to lead to accidental, unexpected updates of > the local dictionary without your knowledge. > > > [...] > > Of course, many of these edge cases are pretty obscure, so it's not > > clear how much they matter. But I think we can at least agree that > > this isn't the one obvious way to do it :-). > > Indeed. I thought I was doing well to know that writing to locals() > inside a function didn't necessarily update the variable, but I had no > idea of the levels of complexity actually involved! > > > > I can think of a lot of criteria that all-else-being-equal we would > > like Python to meet. (Of course, in practice they conflict.) > > > > Consistency across APIs: it's surprising if locals() and > > frame.f_locals do different things. This argues for [proxy]. > > I don't think it is that surprising, since frame.f_locals is kinda > obscure (the average Python coder wouldn't know a frame if one fell > on them) and locals() has been documented as weird for decades. > > In any case, at least "its a copy of ..." is simple and understandable. > > > > Consistency across contexts: it's surprising if locals() has acts > > differently in module/class scope versus function scope. This argues > > for [proxy]. > > True as far as it goes, but its also true that for the longest time, in > most implementations, locals() has acted differently. So no change there. > > On the other hand, locals() currently returns a dict everywhere. It > might be surprising for it to start returning a proxy object inside > functions instead of a dict. > > > -- > Steven > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue May 28 00:57:34 2019 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 27 May 2019 21:57:34 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: On Mon, May 27, 2019 at 9:18 PM Guido van Rossum wrote: > > Note that the weird, Action At A Distance behavior is also visible for locals() called at module scope (since there, locals() is globals(), which returns the actual dict that's the module's __dict__, i.e. the Source Of Truth. So I think it's unavoidable in general, and we would do wise not to try and "fix" it just for function locals. (And I certainly don't want to mess with globals().) I think it's worth distinguishing between two different types of weird Action At A Distance here: - There's the "justified" action-at-a-distance that currently happens at module scope, where locals().__setitem__ affects variable lookup, and variable mutation affects locals().__getitem__. This can produce surprising results if you pass locals() into something that's expecting a regular dict, but it's also arguably the point of an environment introspection API, and like you say, it's unavoidable and expected at module scope and when using globals(). - And then there's the "spooky" action-at-a-distance that currently happens at function scope, where calling locals() has the side-effect of mutating the return value from previous calls to locals(), and the objects returned from locals may or may not spontaneously mutate themselves depending on whether some other code registered a trace function. This is traditional, but extremely surprising if you aren't deeply familiar with internals of CPython's implementation. Of the four designs: [PEP] and [PEP-minus-tracing] both have "spooky" action-at-a-distance (worse in [PEP]), but they don't have "justified" action-at-a-distance. [proxy] adds "justified" action-at-a-distance, and removes "spooky" action at a distance. [snapshot] gets rid of both kinds of action-at-a-distance (at least in function scope). -n -- Nathaniel J. Smith -- https://vorpus.org From p.f.moore at gmail.com Tue May 28 03:37:17 2019 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 28 May 2019 08:37:17 +0100 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: On Tue, 28 May 2019 at 06:00, Nathaniel Smith wrote: > - There's the "justified" action-at-a-distance that currently happens > at module scope, where locals().__setitem__ affects variable lookup, > and variable mutation affects locals().__getitem__. This can produce > surprising results if you pass locals() into something that's > expecting a regular dict, but it's also arguably the point of an > environment introspection API, and like you say, it's unavoidable and > expected at module scope and when using globals(). If I understand you, this "justified" action at a distance is exactly similar to how os.environ works, making it very reasonable that locals and globals work the same (indeed, explaining that locals *doesn't* do this at the moment is why (IMO) people consider locals() as "a bit weird"). > - And then there's the "spooky" action-at-a-distance that currently > happens at function scope, where calling locals() has the side-effect > of mutating the return value from previous calls to locals(), and the > objects returned from locals may or may not spontaneously mutate > themselves depending on whether some other code registered a trace > function. This is traditional, but extremely surprising if you aren't > deeply familiar with internals of CPython's implementation. Yep, this is the one that most non-experts commenting on this thread (including me) seem to have been surprised by. > Of the four designs: > > [PEP] and [PEP-minus-tracing] both have "spooky" action-at-a-distance > (worse in [PEP]), but they don't have "justified" > action-at-a-distance. > > [proxy] adds "justified" action-at-a-distance, and removes "spooky" > action at a distance. > > [snapshot] gets rid of both kinds of action-at-a-distance (at least in > function scope). This summary leaves me very strongly feeling that I prefer [proxy] first, then [snapshot], with the two [PEP] variants a distant third and fourth. Of course, all of this is only if we have decided to formalise the semantics and change CPython to conform. I've never personally been affected by any of the edge cases with locals(), so on a purely personal basis, I'm equally happy with "do nothing" :-) Paul From steve at holdenweb.com Tue May 28 04:30:11 2019 From: steve at holdenweb.com (Steve Holden) Date: Tue, 28 May 2019 09:30:11 +0100 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: <723d1f0a-1b36-0e09-6e1c-f0b1df4f1410@Damon-Family.org> References: <5CEB8F2B.6050300@canterbury.ac.nz> <723d1f0a-1b36-0e09-6e1c-f0b1df4f1410@Damon-Family.org> Message-ID: I'm guessing the reason is to remove the overhead of keeping the dictionary up to date during function execution when no Python code needs access to it. On Mon, May 27, 2019 at 8:10 PM Richard Damon wrote: > On 5/27/19 2:05 PM, Terry Reedy wrote: > > On 5/27/2019 9:52 AM, Richard Damon wrote: > >> On 5/27/19 9:12 AM, Terry Reedy wrote: > > > >>> I believe that the situation is or can be thought of as this: there is > >>> exactly 1 function locals dict. > > > > per function invocation, or more generally, as Guido said, per stack > > frame. This part is obvious to me, but I should have been clearer. > > > >>> Initially, it is empty and > >>> inaccessible (unusable) from code. Each locals() call updates the > >>> dict to a current snapshot and returns it. > >>> > >> I had a similar concern, and one BIG issue with it being define this way > >> is that you get a fundamental re-entrancy problem. If module a uses > >> locals(), and then calls module b that uses locals(), module a has lost > >> its usage. > > > > No. Sorry about being unclear. > > > Ok, if each function invocation gets its own dict, then the reentrancy > issues go away. > > The fact that it isn't the 'active' dict, so you can't use it to modify > the current state, but also you don't get a fresh copy each time (there > is a single mutable dict that gets updated) seems to be an odd behavior > and I can't see where it is an advantage to the user of the function, or > where that makes it easier on the implementation. (But I could easy be > missing something). > > -- > Richard Damon > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue May 28 06:03:09 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 28 May 2019 20:03:09 +1000 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: <20190528100309.GI4221@ando.pearwood.info> On Tue, May 28, 2019 at 08:37:17AM +0100, Paul Moore wrote: > Of course, all of this is only if we have decided to formalise the > semantics and change CPython to conform. I've never personally been > affected by any of the edge cases with locals(), so on a purely > personal basis, I'm equally happy with "do nothing" :-) I don't think "Do Nothing" is a good option, because (as I understand it) the status quo has some weird bugs when you have tracing functions written in Python (but not in C?). So something has to change, one way or another. Hence Nick's PEP. -- Steven From encukou at gmail.com Tue May 28 07:52:53 2019 From: encukou at gmail.com (Petr Viktorin) Date: Tue, 28 May 2019 13:52:53 +0200 Subject: [Python-Dev] Accepting PEP 590 (Vectorcall) Message-ID: As BDFL-Delegate, I am accepting PEP 590 -- Vectorcall: a fast calling protocol for CPython. https://www.python.org/dev/peps/pep-0590/ There was a major last-minute change to the text (though one that was discussed since PyCon US): The API is not finalized yet, and it's provided using private names (with leading underscores). We intend to use the Python 3.8 cycle to get feedback from consumers, and finalize it in 3.9. Thanks to Jeroen Demeyer, who spent a lot of time pushing this forward, and probably has some new gray hair from interactions with me and other core devs. Without his PEP 580 (and its implementation), the simpler PEP 590 would not be possible. It might not be apparent if some of his interactions are taken out of context, but throughout the process Jeroen always worked to make Python better, and he acted professionally even as we made decisions he didn't agree with. Thanks to Mark Shannon for providing major insights, patiently explaining that they actually are good ideas, and for the initial implementation. Thanks to Stefan Behnel, Nick Coghlan, Victor Stinner and everyone else involved in PEPs 576, 579, 580 and 590. I learned a lot in this process. This is my first time as BDFL-delegate. If you have any criticism of the way I handled things, please share! (Assume I have tough skin, but keep to the code of conduct.) I'd be happy to know how I can improve. And now, I'm off to get the implementation reviewed! From guido at python.org Tue May 28 10:47:19 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 28 May 2019 07:47:19 -0700 Subject: [Python-Dev] Accepting PEP 590 (Vectorcall) In-Reply-To: References: Message-ID: Thank you Petr for being a great BDFL-Delegate, and thanks to Mark and Jeroen for working together. And thanks to everyone else who participated in the discussion. Good luck with the implementation and the follow-up steps! --Guido On Tue, May 28, 2019 at 4:55 AM Petr Viktorin wrote: > As BDFL-Delegate, I am accepting PEP 590 -- Vectorcall: a fast calling > protocol for CPython. > https://www.python.org/dev/peps/pep-0590/ > > There was a major last-minute change to the text (though one that was > discussed since PyCon US): The API is not finalized yet, and it's > provided using private names (with leading underscores). We intend to > use the Python 3.8 cycle to get feedback from consumers, and finalize it > in 3.9. > > > > Thanks to Jeroen Demeyer, who spent a lot of time pushing this forward, > and probably has some new gray hair from interactions with me and other > core devs. Without his PEP 580 (and its implementation), the simpler PEP > 590 would not be possible. > It might not be apparent if some of his interactions are taken out of > context, but throughout the process Jeroen always worked to make Python > better, and he acted professionally even as we made decisions he didn't > agree with. > > Thanks to Mark Shannon for providing major insights, patiently > explaining that they actually are good ideas, and for the initial > implementation. > > Thanks to Stefan Behnel, Nick Coghlan, Victor Stinner and everyone else > involved in PEPs 576, 579, 580 and 590. > > I learned a lot in this process. > > This is my first time as BDFL-delegate. If you have any criticism of the > way I handled things, please share! (Assume I have tough skin, but keep > to the code of conduct.) I'd be happy to know how I can improve. > > > And now, I'm off to get the implementation reviewed! > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Tue May 28 11:15:32 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 May 2019 01:15:32 +1000 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: On Sun, 26 May 2019 at 00:37, Guido van Rossum wrote: > > This looks great. > > I only have two nits with the text. > > First, why is the snapshot called a "dynamic snapshot"? What exactly is dynamic about it? The dynamic part comes in if you call locals() twice: because it returns the same mapping object every time, the second call to locals() will update the snapshot with the updated values of the local variables. To get a static snapshot that won't be implicitly updated by subsequent calls to locals() (or accesses to frame.f_locals, or implicit updates prior to trace function invocation) you have to do "locals().copy()". As others have noted, this isn't the behaviour we would choose if we were designing this API from scratch, but the current CPython behaviour has ~18 years of de facto standardisation behind it (as near as I can tell it has worked this way since nested scopes and fast locals were introduced in Python 2.1), so I'd be pretty hesitant about changing it at this point. In particular, I want to preserve the existing behaviour of Python 3 code that runs exec() and eval() at function scope, which means ensuring that locals(): 1. Continues to behave like a normal dict instance when passed to another API 2. Continues to leave the actual local variables of the calling frame alone > Second, you use the word "mapping" a lot. Would you mind changing that to "mapping object" in most places? Especially in the phrase "each call to ``locals()`` returns the *same* mapping". To me, without the word "object" added, this *could* be interpreted as "a dict with the same key/value pairs" (since "mapping" is also an abstract mathematical concept describing anything that maps keys to values). That makes sense - I'll update the text. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From solipsis at pitrou.net Tue May 28 11:37:29 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 28 May 2019 17:37:29 +0200 Subject: [Python-Dev] Accepting PEP 590 (Vectorcall) References: Message-ID: <20190528173729.3d23914b@fsol> Congrats to everyone who participated! Regards Antoine. On Tue, 28 May 2019 13:52:53 +0200 Petr Viktorin wrote: > As BDFL-Delegate, I am accepting PEP 590 -- Vectorcall: a fast calling > protocol for CPython. > https://www.python.org/dev/peps/pep-0590/ > > There was a major last-minute change to the text (though one that was > discussed since PyCon US): The API is not finalized yet, and it's > provided using private names (with leading underscores). We intend to > use the Python 3.8 cycle to get feedback from consumers, and finalize it > in 3.9. > > > > Thanks to Jeroen Demeyer, who spent a lot of time pushing this forward, > and probably has some new gray hair from interactions with me and other > core devs. Without his PEP 580 (and its implementation), the simpler PEP > 590 would not be possible. > It might not be apparent if some of his interactions are taken out of > context, but throughout the process Jeroen always worked to make Python > better, and he acted professionally even as we made decisions he didn't > agree with. > > Thanks to Mark Shannon for providing major insights, patiently > explaining that they actually are good ideas, and for the initial > implementation. > > Thanks to Stefan Behnel, Nick Coghlan, Victor Stinner and everyone else > involved in PEPs 576, 579, 580 and 590. > > I learned a lot in this process. > > This is my first time as BDFL-delegate. If you have any criticism of the > way I handled things, please share! (Assume I have tough skin, but keep > to the code of conduct.) I'd be happy to know how I can improve. > > > And now, I'm off to get the implementation reviewed! From tim.peters at gmail.com Tue May 28 14:23:53 2019 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 28 May 2019 13:23:53 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: I made a pull request for this that appears to work fine for my 10x smaller test case (reduces tear-down time from over 45 seconds to over 7). It implements my second earlier sketch (add a vector of search fingers, to eliminate searches): https://github.com/python/cpython/pull/13612 It would be good to know how it works on the OP's 10x larger test case, which requires a machine with over 80 GB of RAM. Hoping it will reduce tear-down time from hours to minutes (they have about a billion `Node` objects, so tear-down time will never be trivial). It would also be good to get a code review, if you have the time and inclination. Thanks! From ezio.melotti at gmail.com Tue May 28 15:05:10 2019 From: ezio.melotti at gmail.com (Ezio Melotti) Date: Tue, 28 May 2019 21:05:10 +0200 Subject: [Python-Dev] PEP 595: Improving bugs.python.org In-Reply-To: References: Message-ID: On Thu, May 23, 2019 at 10:17 PM Ezio Melotti wrote: > > Hello, > Berker and I have been working on a PEP that suggests we keep using > and improving bugs.python.org and Roundup instead of switching to > GitHub Issues as proposed by PEP 581. > > The PEP covers: > * What are the advantages of Roundup over GitHub issues; > * What features are missing in Roundup and how can we add them; > * Issues with PEP 581; > * Issues with the migration plan proposed by PEP 588; > > The rendered version of PEP 595 is available at > https://www.python.org/dev/peps/pep-0595/ > > For reference, you can consult PEP 581 and 588 at > https://www.python.org/dev/peps/pep-0581/ and > https://www.python.org/dev/peps/pep-0588/ > > The full text of the PEP is include below. We are planning to update > the PEP to include the feedback we receive and to update the status of > features as we implement them (we also have a Google Summer of Code > students working on it). > > Best Regards, > Ezio Melotti > > Hello, earlier today I expanded and reworded the "Migration considerations" section and added the feedback I got from the replies. You can find the rendered version of that section (and the rest of the PEP) at https://www.python.org/dev/peps/pep-0595/#migration-considerations The changeset can be found at https://github.com/python/peps/commit/b3f4c8eb09d1987d00785cad385adf7e0394776e The full text of the latest version of the PEP is included below. Best Regards, Ezio Melotti ================ PEP: 595 Title: Improving bugs.python.org Author: Ezio Melotti , Berker Peksag Status: Draft Type: Process Content-Type: text/x-rst Created: 12-May-2019 Abstract ======== This PEP proposes a list of improvements to make bugs.python.org more usable for contributors and core developers. This PEP also discusses why remaining on Roundup should be preferred over switching to GitHub Issues, as proposed by :pep:`581`. Motivation ========== On May 14th, 2019 :pep:`581` has been accepted [#]_ without much public discussion and without a clear consensus [#]_. The PEP contains factual errors and doesn't address some of the issues that the migration to GitHub Issues might present. Given the scope of the migration, the amount of work required, and how it will negatively affect the workflow during the transition phase, this decision should be re-evaluated. Roundup advantages over GitHub Issues ===================================== This section discusses reasons why Roundup should be preferred over GitHub Issues and Roundup features that are not available on GitHub Issues. * **Roundup is the status quo.** Roundup has been an integral part of the CPython workflow for years. It is a stable product that has been tested and customized to adapt to our needs as the workflow evolved. It is possible to gradually improve it and avoid the disruption that a switch to a different system would inevitabily bring to the workflow. * **Open-source and Python powered.** Roundup is an open-source project and is written in Python. By using it and supporting it, we also support the Python ecosystem. Several features developed for bpo have also been ported to upstream Roundup over the years. * **Fully customizable.** Roundup can be (and has been) fully customized to fit our needs. * **Finer-grained access control.** Roundup allows the creation of different roles with different permissions (e.g. create, view, edit, etc.) for each individual property, and users can have multiple roles. * **Flexible UI.** While Roundup UI might look dated, it is convenient and flexible. For example, on the issue page, each field (e.g. title, type, versions, status, linked files and PRs, etc.) have appropriate UI elements (input boxes, dropdowns, tables, etc.) that are easy to set and also provide a convenient way to get info about the issue at a glance. The number of fields, their values, and the UI element they use is also fully customizable. GitHub only provides labels. The issue list page presents the issues in a compact and easy to read table with separate columns for different fields. For comparison, Roundup lists 50 issues in a screen, whereas GitHub takes two screens to shows 25 issues. * **Advanced search.** Roundup provides an accurate way to search and filter by using any combination of issue fields. It is also possible to customize the number of results and the fields displayed in the table, and the sorting and grouping (up to two levels). bpo also provides predefined summaries (e.g. "Created by you", "Assigned to you", etc.) and allows the creation of custom search queries that can be conveniently accessed from the sidebar. * **Nosy list autocomplete.** The nosy list has an autocomplete feature that suggests maintainers and experts. The suggestions are automatically updated when the experts index [#]_ changes. * **Dependencies and Superseders.** Roundup allows to specify dependencies that must be addressed before the current issues can be closed and a superseder issue to easily mark duplicates [#]_. The list of dependencies can also be used to create meta-issues that references several other sub-issues [#]_. Improving Roundup ================= This section lists some of the issues mentioned by :pep:`581` and other desired features and discusses how they can be implemented by improving Roundup and/or our instance. * **REST API support.** A REST API will make integration with other services and the development of new tools and applications easiers. Upstream Roundup now supports a REST API. Updating the tracker will make the REST API available. * **GitHub login support.** This will allow users to login to bugs.python.org (bpo) without having to create a new account. It will also solve issues with confirmation emails being marked as spam, and provide two-factor authentication. A patch to add this functionality is already available and is being integrated at the time of writing [#]_. * **Markdown support and message preview and editing.** This feature will allow the use of Markdown in messages and the ability to preview the message before the submission and edit it afterward. This can be done, but it will take some work. Possible solutions have been proposed on the roundup-devel mailing list [#]_. * **"Remove me from nosy list" button.** Add a button on issue pages to remove self from the nosy list. This feature will be added during GSoC 2019. * **Mobile friendly theme.** Current theme of bugs.python.org looks dated and it doesn't work well with mobile browsers. A mobile-friendly theme that is more modern but still familiar will be added. * **Move reply box close to the last message.** The reply box is located at the top of the page, whereas the last message is at the bottom. The reply box can be moved or duplicated after the last message. * **Real-time updates.** When another users submits changes to an issue, they should show up in real time. This can be accomplished by using the REST API. * **Add PR link to BPO emails.** Currently bpo emails don't include links to the corresponding PRs. A patch [#]_ is available to change the content of the bpo emails from:: components: +Tkinter versions: +Python 3.4 pull_requests: +42 to:: components: +Tkinter versions: +Python 3.4 pull_request: https://github.com/python/cpython/pull/341 * **Python 3 support.** Using Python 3 will make maintenance easier. Upstream Roundup now supports Python 3. Updating the tracker will allow us to switch to Python 3. The instances will need to be updated as well. * **Use upstream Roundup.** We currently use a fork of Roundup with a few modifications, most notably the GitHub integration. If this is ported upstream, we can start using upstream Roundup without having to maintain our fork. PEP 581 issues ============== This section addresses some errors and inaccuracies found in :pep:`581`. The "Why GitHub?" section of PEP 581 lists features currently available on GitHub Issues but not on Roundup. Some of this features are currently supported: * "Ability to reply to issue and pull request conversations via email." * Being able to reply by email has been one of the core features of Roundup since the beginning. It is also possible to create new issues or close existing ones, set or modify fields, and add attachments. * "Email notifications containing metadata, integrated with Gmail, allowing systematic filtering of emails." * Emails sent by Roundup contains metadata that can be used for filtering. * "Additional privacy, such as offering the user a choice to hide an email address, while still allowing communication with the user through @-mentions." * Email addresses are hidden by default to users that are not registered. Registered users can see other users' addresses because we configured the tracker to show them. It can easily be changed if desired. Users can still be added to the nosy list by using their username even if their address is hidden. * "Ability to automatically close issues when a PR has been merged." * The GitHub integration of Roundup automatically closes issues when a commit that contains "fixes issue " is merged. (Alternative spellings such as "closes" or "bug" are also supported.) See [#]_ for a recent example of this feature. * "Support for permalinks, allowing easy quoting and copying & pasting of source code." * Roundup has permalinks for issues, messages, attachments, etc. In addition, Roundup allows to easily rewrite broken URLs in messages (e.g. if the code hosting changes). * "Core developers, volunteers, and the PSF don't have to maintain the issue infrastructure/site, giving us more time and resources to focus on the development of Python." * While this is partially true, additional resources are required to write and maintain bots. In some cases, bots are required to workaround GitHub's lack of features rather than expanding. [#]_ was written specifically to workaround GitHub's email integration. Updating our bots to stay up-to-date with changes in the GitHub API has also maintenance cost. [#]_ took two days to be fixed. In addition, we will still need to maintain Roundup for bpo (even if it becomes read-only) and for the other trackers we currently host/maintain (Jython [#]_ and Roundup [#]_). The "Issues with Roundup / bpo" section of :pep:`581` lists some issues that have already been fixed: * "The upstream Roundup code is in Mercurial. Without any CI available, it puts heavy burden on the few existing maintainers in terms of reviewing, testing, and applying patches." * While Roundup uses Mercurial by default, there is a git clone available on GitHub [#]_. Roundup also has CI available [#]_ [#]_. * "There is no REST API available. There is an open issue in Roundup for adding REST API. Last activity was in 2016." * The REST API has been integrated and it's now available in Roundup. * "Users email addresses are exposed. There is no option to mask it." * Exposing addresses to registered and logged in users was a decision taken when our instance was set up. This has now been changed to make the email addresses hidden for regular users too (Developers and Coordinators can still see them). The "Email address"" column from the user listing page [#]_ has been removed too. * "It sends a number of unnecessary emails and notifications, and it is difficult, if not impossible, to configure." * This can be configured. * "Creating an account has been a hassle. There have been reports of people having trouble creating accounts or logging in." * The main issue is confirmation emails being marked as spam. Work has been done to resolve the issue. Migration considerations ======================== This section describes issues with the migrations that might not have been addressed by :pep:`581` and :pep:`588`. :pep:`588` suggests to add a button to migrate issues to GitHub only when someone wants to keep working on them. This approach has several issues, but there are also other issues that will need to be addressed regardless of the approach used: * **Vendor lock-in.** GitHub is properietary and there is risk of vendor lock-in. Their business model might change and they could shut down altogether. For example, several projects decided to move away from GitHub after Microsoft acquisition. If/when the repository is no longer available on GitHub, we will be forced to migrate again and all the links to the issues won't work anymore. * **Required bpo updates.** bpo will need to be updated in order to add a button that, once pressed, creates a new issue on GitHub, copies over all the messages, attachments, and creates/adds labels for the existing fields. Permissions will also need to be tweaked to make individual issues read-only once they are migrated, and to prevent users to create new accounts. It might be necessary to set up redirects (see below). * **Two trackers.** If issues are migrated on demand, the issues will be split between two trackers. Referencing and searching issues will take significant more effort. * **Lossy conversion.** GitHub only mechanism to add custom metadata is through labels. bpo uses a number of fields to specify several different metadata. Preserving all fields and values will result in too many labels. If only some fields and values are preserved the others will be lost (unless there is a way to preserve them elsewhere). * **Issue IDs preservation.** GitHub doesn't provide a way to set and preserve the ID of migrated issues. Some projects managed to preserve the IDs by contacting the GitHub staff and migrating the issues *en masse*. However, this is no longer possible, since PRs and issues share the same namespace and PRs already use existing bpo issue IDs. * **Internal issue links preservation.** Existing issues might contain references to other issues in messages and fields (e.g. dependencies or superseder). Since the issue ID will change during the migration, these will need to be updated. If the issues are migrated on demand, all the existing internal references to the migrated issues (on both bpo and GitHub issues) will have to be updated. Setting up a redirect for each migrated issue on bpo might mitigate the issue, however -- if references in migrated messages are not updated -- it will cause confusion (e.g. if bpo issue `#1234` becomes GitHub issue `#4321`, a reference to `#1234` in a migrated message could link to bpo `#1234` and bpo can redirect to GitHub issue `#4321`, but new references to `#1234` will link to GitHub PR `#1234` rather than GitHub issue `#4321`). Manually having to specify a `bpo-` or `gh-` prefix is error prone. * **External issue links preservation.** A number of websites, mails, etc. link to bpo issues. If bpo is shut down, these links will break. If we don't want to break the links, we will have to keep bpo alive and set up a redirect system that links to the corresponding GitHub issue. In addition, if GitHub shuts down, we won't have any way to setup redirects and preserve external links to GitHub issues. * **References preservation and updating.** In addition to issue references, bpo converts a number of other references into links, including message and PR IDs, changeset numbers, legacy SVN revision numbers, paths to files in the repo, files in tracebacks (detecting the correct branch), and links to devguide pages and sections [#]_. Since Roundup converts references to links when messages are requested, it is possible to update the target and generate the correct link. This need already arised several times, for example: files and HG changesets moved from `hg.python.org` to GitHub and the devguide moved from `docs.python.org/devguide` to `devguide.python.org`. Since messages on GitHub are static, the links will need to be generated and hardcoded during the migration or they will be lost. In order to update them, a tool to find all references and regenerate the links will need to be written. * **Roundup and bpo maintenance.** On top of the aforementioned changes to bpo and development of tools required to migrate to GitHub issues, we will still need to keep running and maintaining Roundup, both for our bpo instance (read-only) and for the Jython and Roundup trackers (read-write). Even if eventually we migrate all bpo issues to GitHub and we stop maintaining Jython and Roundup, bpo will need to be maintained and redirect to the corresponding GitHub issues. * **Bots maintenance.** Since it's not possible to customize GitHub directly, it's also necessary to write, maintain, and host bots. Even if eventually we stop maintaining Roundup, the maintenance burden simply shifted from Roundup to the bots. Hosting each different bot also has a monetary cost. * **Using issue templates.** Manually editing issue templates to "remove texts that don't apply to [the] issue" is cumbersome and error-prone. * **Signal to noise ratio.** Switching to GitHub Issues will likely increase the number of invalid reports and increase the triaging effort. This concern has been raised in the past in a Zulip topic [#]_. There have been already cases where people posted comments on PRs that required moderators to mark them as off-topic or disruptive, delete them altogether, and even lock the conversation [#]_. * **Weekly tracker reports and stats.** Roundup sends weekly reports to python-dev with a summary that includes new issues, recent issues with no replies, recent issues waiting for review, most discussed issues, closed issues, and deltas for open/closed/total issue counts [#]_. The report provides an easy way to keep track of the tracker activity and to make sure that issues that require attention are noticed. The data collect by the weekly report is also use to generate statistics and graphs that can be used to gain new insights [#]_. * **bpo-related MLs.** There are currently two mailing lists where bpo posts new tracker issues and all messages respectively: `new-bugs-announce` [#]_ and `python-bugs-list` [#]_. A new system will need to be developed to preserve this functionality. These MLs offer additional ways to keep track of the tracker activity. References ========== .. [#] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted https://mail.python.org/pipermail/python-dev/2019-May/157399.html .. [#] [python-committers] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted https://mail.python.org/pipermail/python-committers/2019-May/006755.html .. [#] Experts Index -- Python Devguide https://devguide.python.org/experts/ .. [#] An example of superseded issues: "re.sub() replaces only several matches" https://bugs.python.org/issue12078 .. [#] An example of meta issue using dependencies to track sub-issues: "Meta-issue: support of the android platform"" https://bugs.python.org/issue26865 .. [#] Support logging in with GitHub https://github.com/python/bugs.python.org/issues/7 .. [#] Re: [Roundup-devel] PEP 581 and Google Summer of Code https://sourceforge.net/p/roundup/mailman/message/36667828/ .. [#] [Tracker-discuss] [issue624] bpo emails contain useless non-github pull_request number - users want a link to actual github PR https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html .. [#] The commit reported in msg342882 closes the issue (see the history below) https://bugs.python.org/issue36951#msg342882 .. [#] The cpython-emailer-webhook project https://github.com/berkerpeksag/cpython-emailer-webhook .. [#] A recent incident caused by GitHub https://github.com/python/bedevere/pull/163 .. [#] Jython issue tracker https://bugs.jython.org/ .. [#] Roundup issue tracker https://issues.roundup-tracker.org/ .. [#] GitHub clone of Roundup https://github.com/roundup-tracker/roundup .. [#] Travis-CI for Roundup https://travis-ci.org/roundup-tracker/roundup) and codecov .. [#] Codecov for Roundup https://codecov.io/gh/roundup-tracker/roundup/commits .. [#] User listing -- Python tracker https://bugs.python.org/user?@sort=username .. [#] Generating Special Links in a Comment -- Python Devguide https://devguide.python.org/triaging/#generating-special-links-in-a-comment .. [#] The New-bugs-announce mailing list https://mail.python.org/mailman/listinfo/new-bugs-announce .. [#] The Python-bugs-list mailing list https://mail.python.org/mailman/listinfo/python-bugs-list .. [#] An example of [Python-Dev] Summary of Python tracker Issues https://mail.python.org/pipermail/python-dev/2019-May/157483.html .. [#] Issues stats -- Python tracker https://bugs.python.org/issue?@template=stats .. [#] s/n ratio -- Python -- Zulip https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20ratio .. [#] For example this and other related PRs: https://github.com/python/cpython/pull/9099 Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From greg.ewing at canterbury.ac.nz Tue May 28 20:23:14 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 29 May 2019 12:23:14 +1200 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: References: <5CEB8F2B.6050300@canterbury.ac.nz> Message-ID: <5CEDD0F2.20107@canterbury.ac.nz> Terry Reedy wrote: > I believe that the situation is or can be thought of as this: there is > exactly 1 function locals dict. Initially, it is empty and inaccessible > (unusable) from code. Each locals() call updates the dict to a current > snapshot and returns it. Yes, I understand *what's* happening, but not *why* it was designed that way. Would it really be probihitively expensive to create a fresh dict each time? -- Greg From guido at python.org Tue May 28 21:02:19 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 28 May 2019 18:02:19 -0700 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: <5CEDD0F2.20107@canterbury.ac.nz> References: <5CEB8F2B.6050300@canterbury.ac.nz> <5CEDD0F2.20107@canterbury.ac.nz> Message-ID: On Tue, May 28, 2019 at 5:25 PM Greg Ewing wrote: > Terry Reedy wrote: > > I believe that the situation is or can be thought of as this: there is > > exactly 1 function locals dict. Initially, it is empty and inaccessible > > (unusable) from code. Each locals() call updates the dict to a current > > snapshot and returns it. > > Yes, I understand *what's* happening, but not *why* it was designed > that way. Would it really be prohibitively expensive to create a > fresh dict each time? > No. But it would be inconsistent with the behavior at module level. FWIW I am leaning more and more to the [proxy] model, where locals() and frame.f_locals are the same object, which *proxies* the fast locals and cells. That only has one downside: it no longer returns a dict, but merely a MutableMapping. But why would code care about the difference? (There used to be some relevant builtins that took dicts but not general MutableMappings -- but that has been fixed long ago.) -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue May 28 21:45:41 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 29 May 2019 13:45:41 +1200 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: Message-ID: <5CEDE445.80107@canterbury.ac.nz> Nathaniel Smith wrote: > - [proxy]: Simply return the .f_locals object, so in all contexts > locals() returns a live mutable view of the actual environment: > > def locals(): > return get_caller_frame().f_locals Not sure I quite follow this -- as far as I can see, f_locals currently has the same snapshot behaviour as locals(). I'm assuming you mean to change things so that locals() returns a mutable view tracking the environment in both directions. That sounds like a much better idea all round to me. No weird shared-snapshot behaviour, and no need for anything to behave differently when tracing. If the change to the behaviour of exec() and eval() is a concern, then perhaps there should be a new localsview() function that returns a mutable view, with locals() redefined as dict(localsview()). -- Greg From njs at pobox.com Tue May 28 21:50:04 2019 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 28 May 2019 18:50:04 -0700 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: References: <5CEB8F2B.6050300@canterbury.ac.nz> <5CEDD0F2.20107@canterbury.ac.nz> Message-ID: On Tue, May 28, 2019 at 6:02 PM Guido van Rossum wrote: > > On Tue, May 28, 2019 at 5:25 PM Greg Ewing wrote: >> >> Terry Reedy wrote: >> > I believe that the situation is or can be thought of as this: there is >> > exactly 1 function locals dict. Initially, it is empty and inaccessible >> > (unusable) from code. Each locals() call updates the dict to a current >> > snapshot and returns it. >> >> Yes, I understand *what's* happening, but not *why* it was designed >> that way. Would it really be prohibitively expensive to create a >> fresh dict each time? > > No. But it would be inconsistent with the behavior at module level. > > FWIW I am leaning more and more to the [proxy] model, where locals() and frame.f_locals are the same object, which *proxies* the fast locals and cells. That only has one downside: it no longer returns a dict, but merely a MutableMapping. But why would code care about the difference? (There used to be some relevant builtins that took dicts but not general MutableMappings -- but that has been fixed long ago.) Related trivia: the exec() and eval() builtins still mandate that their 'globals' argument be an actual no-fooling dict, but their 'locals' argument is allowed to be any kind of mapping object. This is an intentional, documented feature [1]. And inside the exec/eval, calls to locals() return whatever object was passed. For example: >>> exec("print(type(locals()))", {}, collections.ChainMap()) So technically speaking, it's already possible for locals() to return a non-dict. Of course this is incredibly uncommon in practice, so existing code doesn't necessarily take it into account. But it's some kind of conceptual precedent, anyway. -n [1] See https://docs.python.org/3/library/functions.html#eval and the 'exec' docs right below it. I think the motivation is that in the current CPython implementation, every time you access a global it does a direct lookup in the globals object, so it's important that we do this lookup as fast as possible, and forcing the globals object to be a actual dict allows some optimizations. For locals, though, we usually use the "fast locals" mechanism and the mapping object is mostly vestigial, so it doesn't matter how fast lookups are, so we can support any mapping. -- Nathaniel J. Smith -- https://vorpus.org From rob.cliffe at btinternet.com Tue May 28 21:52:09 2019 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 29 May 2019 02:52:09 +0100 Subject: [Python-Dev] Overloading comparison operator for lists In-Reply-To: References: Message-ID: On 27/05/2019 04:52:17, Montana Burr wrote: > NumPy arrays have this awesome feature, where array == 3 does an > element-wise comparison and returns a list. For example: > > np.array([1,2,3,4,5])==3 > > returns > > [False,False,True,False,False] > > It would be cool if Python had similar functionality for lists. > Well, it does have: >>>? [x==3 for x in [1,2,3,4,5]] [False, False, True, False, False] This is IMHO much more intuitive than your construct overloading "==". It is also more flexible (any operation can be performed on x, not just an equality comparison). So sorry, but I can see no justification for changing the Python language to do something, which can already be done, to be done in a more obscure way. Also ??? [1,2,3,4,5]==3 is already legal syntax (it evaluates to False, since the operands are not equal), so you are proposing a code-breaking change. Apologies if someone has already pointed all this out, as is very likely (I have only just (re-)joined python-ideas and may have missed the relevant posts). Best wishes Rob Cliffe From njs at pobox.com Tue May 28 22:27:59 2019 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 28 May 2019 19:27:59 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: <5CEDE445.80107@canterbury.ac.nz> References: <5CEDE445.80107@canterbury.ac.nz> Message-ID: On Tue, May 28, 2019 at 6:48 PM Greg Ewing wrote: > > Nathaniel Smith wrote: > > - [proxy]: Simply return the .f_locals object, so in all contexts > > locals() returns a live mutable view of the actual environment: > > > > def locals(): > > return get_caller_frame().f_locals > > Not sure I quite follow this -- as far as I can see, f_locals > currently has the same snapshot behaviour as locals(). > > I'm assuming you mean to change things so that locals() returns a > mutable view tracking the environment in both directions. That > sounds like a much better idea all round to me. No weird > shared-snapshot behaviour, and no need for anything to behave > differently when tracing. Yeah, I made the classic mistake and forgot that my audience isn't as immersed in this as I am :-). Throughout the email I'm assuming we're going to adopt PEP 558's proposal about replacing f_locals with a new kind of mutable view object, and then given that, asking what we should do about locals(). -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Wed May 29 00:02:51 2019 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 28 May 2019 21:02:51 -0700 Subject: [Python-Dev] [SPAM?] Re: PEP 558: Defined semantics for locals() In-Reply-To: <5CEDD0F2.20107@canterbury.ac.nz> References: <5CEB8F2B.6050300@canterbury.ac.nz> <5CEDD0F2.20107@canterbury.ac.nz> Message-ID: On Tue, May 28, 2019 at 5:24 PM Greg Ewing wrote: > > Terry Reedy wrote: > > I believe that the situation is or can be thought of as this: there is > > exactly 1 function locals dict. Initially, it is empty and inaccessible > > (unusable) from code. Each locals() call updates the dict to a current > > snapshot and returns it. > > Yes, I understand *what's* happening, but not *why* it was designed > that way. I'm not sure of the exact history, but I think it's something like: In the Beginning, CPython was Simple, but Slow: every frame struct had an f_locals field, it was always a dict, the bytecode accessed the dict, locals() returned the dict, that was that. Then one day the serpent of Performance Optimization came, whispering of static analysis of function scope and LOAD_FAST bytecodes. And we were seduced by the serpent's vision, and made CPython Faster, with semantics that were Almost The Same, and we shipped it to our users. But now the sin of Cache Inconsistency had entered our hearts, and we were condemned to labor endlessly: again and again, users discovered a leak in our abstraction, and again and again we covered our sin with new patches, until Simplicity was obscured. (The current design does makes sense, but you really have to look at it as a hard-fought compromise between the elegant original design versus ~30 years of real-world demands. And hey, it could be worse ? look at the fun Intel's been having with their caches.) -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Wed May 29 00:12:17 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 May 2019 14:12:17 +1000 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: (I'll likely write a more detailed reply once I'm back on an actual computer, but wanted to send an initial response while folks in the US are still awake, as the detailed reply may not be needed) Thanks for this write-up Nathaniel - I think you've done a good job of capturing the available design alternatives. The one implicit function locals() update case that you missed is in the accessor for "frame.f_locals", so I think dropping CPython's implicit update for all Python trace functions would be a clear win (I actually almost changed the PEP to do that once I realized it was already pretty redundant given the frame accessor behaviour). I'm OK with either [PEP-minus-tracing] or [snapshot], with a slight preference for [snapshot] (since it's easier to explain). The only design option I wouldn't be OK with is [proxy], as I think that poses a significant potential backwards compatibility problem, and I trust Armin Rigo's perspective that it would be hellish for JIT-compiled Python implementations to handle without taking the same kind of performance hit they do when they need to emulate the frame API. By contrast, true snapshot semantics will hopefully make life *easier* for JIT compilers, and folks that actually want the update() behaviour can either rely on frame.f_locals, or do an explicit update. This would also create a possible opportunity to simplify the fast locals proxy semantics: if it doesn't need to emulate the current behaviour of allowing arbitrary keys to be added and preserved between calls to locals(), then it could dispense with its internal dict cache entirely, and instead reject any operations that try to add new keys that aren't defined on the underlying code object (removing keys and then adding them back would still be permitted, and handled like a code level del statement). Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From montana.burr at gmail.com Tue May 28 23:35:51 2019 From: montana.burr at gmail.com (Montana Burr) Date: Tue, 28 May 2019 21:35:51 -0600 Subject: [Python-Dev] Overloading comparison operator for lists In-Reply-To: References: Message-ID: Ok, now I'm mildly curious to knpw: What is the justification for causing list == 3 to evaluate to False, besides the obvious "a list cannot equal a number"? On Tue, May 28, 2019 at 9:34 PM Montana Burr wrote: > Ok, now I'm mildly curious to knpw: > > What is the justification for causing list == 3 to evaluate to False, > besides the obvious "a list cannot equal a number"? > > On Tue, May 28, 2019 at 7:52 PM Rob Cliffe > wrote: > >> >> >> On 27/05/2019 04:52:17, Montana Burr wrote: >> > NumPy arrays have this awesome feature, where array == 3 does an >> > element-wise comparison and returns a list. For example: >> > >> > np.array([1,2,3,4,5])==3 >> > >> > returns >> > >> > [False,False,True,False,False] >> > >> > It would be cool if Python had similar functionality for lists. >> > >> Well, it does have: >> >>> [x==3 for x in [1,2,3,4,5]] >> [False, False, True, False, False] >> >> This is IMHO much more intuitive than your construct overloading "==". >> It is also more flexible (any operation can be performed on x, not just >> an equality comparison). >> So sorry, but I can see no justification for changing the Python >> language to do something, which can already be done, to be done in a >> more obscure way. >> Also >> [1,2,3,4,5]==3 >> is already legal syntax (it evaluates to False, since the operands are >> not equal), so you are proposing a code-breaking change. >> >> Apologies if someone has already pointed all this out, as is very likely >> (I have only just (re-)joined python-ideas and may have missed the >> relevant posts). >> Best wishes >> Rob Cliffe >> > > > -- > > WebRep > Overall rating > > -- WebRep Overall rating -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed May 29 00:29:15 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 28 May 2019 21:29:15 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: So why is it ?hellish? for JITs if locals() returns a proxy, while frame.f_locals being a proxy is okay? On Tue, May 28, 2019 at 9:12 PM Nick Coghlan wrote: > (I'll likely write a more detailed reply once I'm back on an actual > computer, but wanted to send an initial response while folks in the US are > still awake, as the detailed reply may not be needed) > > Thanks for this write-up Nathaniel - I think you've done a good job of > capturing the available design alternatives. > > The one implicit function locals() update case that you missed is in the > accessor for "frame.f_locals", so I think dropping CPython's implicit > update for all Python trace functions would be a clear win (I actually > almost changed the PEP to do that once I realized it was already pretty > redundant given the frame accessor behaviour). > > I'm OK with either [PEP-minus-tracing] or [snapshot], with a slight > preference for [snapshot] (since it's easier to explain). > > The only design option I wouldn't be OK with is [proxy], as I think that > poses a significant potential backwards compatibility problem, and I trust > Armin Rigo's perspective that it would be hellish for JIT-compiled Python > implementations to handle without taking the same kind of performance hit > they do when they need to emulate the frame API. > > By contrast, true snapshot semantics will hopefully make life *easier* for > JIT compilers, and folks that actually want the update() behaviour can > either rely on frame.f_locals, or do an explicit update. > > This would also create a possible opportunity to simplify the fast locals > proxy semantics: if it doesn't need to emulate the current behaviour of > allowing arbitrary keys to be added and preserved between calls to > locals(), then it could dispense with its internal dict cache entirely, and > instead reject any operations that try to add new keys that aren't defined > on the underlying code object (removing keys and then adding them back > would still be permitted, and handled like a code level del statement). > > Cheers, > Nick. > > > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed May 29 01:22:55 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 29 May 2019 15:22:55 +1000 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: On Wed., 29 May 2019, 2:29 pm Guido van Rossum, wrote: > So why is it ?hellish? for JITs if locals() returns a proxy, while > frame.f_locals being a proxy is okay? > As I understand it, they already drop out of compiled mode if they detect that the code is tinkering with frame objects. Having a single locals() call de-optimize an entire function would be far from ideal. Cheers, Nick. ) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed May 29 01:43:07 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 29 May 2019 01:43:07 -0400 Subject: [Python-Dev] Overloading comparison operator for lists In-Reply-To: References: Message-ID: On 5/28/2019 11:35 PM, Montana Burr wrote: > What is the justification for causing list == 3 to evaluate to False, > besides the obvious "a list cannot equal a number"? That is completely sufficient. The default comparison is by id, and a class must override to get something different. The 'default' override is compare by class and if equal, compare by value. Numbers are odd in allowing instances of different classes compare equal. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Wed May 29 02:05:24 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 29 May 2019 18:05:24 +1200 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> Message-ID: <5CEE2124.4020409@canterbury.ac.nz> Nick Coghlan wrote: > Having a single locals() call de-optimize an entire function would be > far from ideal. I don't see what would be so bad about that. The vast majority of functions have no need for locals(). -- Greg From guido at python.org Wed May 29 02:09:35 2019 From: guido at python.org (Guido van Rossum) Date: Tue, 28 May 2019 23:09:35 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: <5CEE2124.4020409@canterbury.ac.nz> References: <20190528022848.GF4221@ando.pearwood.info> <5CEE2124.4020409@canterbury.ac.nz> Message-ID: Indeed. On Tue, May 28, 2019 at 11:07 PM Greg Ewing wrote: > Nick Coghlan wrote: > > Having a single locals() call de-optimize an entire function would be > > far from ideal. > > I don't see what would be so bad about that. The vast majority > of functions have no need for locals(). > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Wed May 29 08:25:25 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 29 May 2019 14:25:25 +0200 Subject: [Python-Dev] PEP 590: vectorcall without tp_call Message-ID: <5CEE7A35.3000106@UGent.be> Hello, I have one implementation question about vectorcall which is not specified in PEP 590: what should happen if a type implements vectorcall (i.e. _Py_TPFLAGS_HAVE_VECTORCALL is set) but doesn't set tp_call (i.e. tp_call == NULL)? I see the following possibilities: 1. Ignore this problem/assume that it won't happen. This would be bad, since callable(obj) would be False even though obj() would succeed. 2. Raise SystemError. 3. Automatically set tp_call to PyVectorcall_Call. I would vote for 3 since it's the most user-friendly option. There is also no way how it could be wrong: it ensures that tp_call and vectorcall are consistent. Any opinions? Jeroen. From steve at pearwood.info Wed May 29 08:44:25 2019 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 29 May 2019 22:44:25 +1000 Subject: [Python-Dev] Overloading comparison operator for lists In-Reply-To: References: Message-ID: <20190529124425.GK4221@ando.pearwood.info> Hi Montana, As Cameron Simpson already pointed out, your query is off-topic for the Python-Dev mailing list and should be taken to the Python-Ideas mailing list, which is for speculative discussion of new designs. Like Cameron, I've CCed Python-Ideas. Please send any follow-ups to that list and not Python-Dev. You asked this question: On Tue, May 28, 2019 at 09:35:51PM -0600, Montana Burr wrote: > Ok, now I'm mildly curious to knpw: > > What is the justification for causing list == 3 to evaluate to False, > besides the obvious "a list cannot equal a number"? I concur with Terry Reedy -- what more justification is needed? A list cannot equal a number, so the default behaviour ought to be to return False. What would you have the default behaviour be? People have already suggested that getting the numpy-style behaviour is simple with a list comprehension, but the other technique is to subclass list, override ``__eq__`` and give it the behaviour you want. -- Steven From songofacandy at gmail.com Wed May 29 09:19:25 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Wed, 29 May 2019 22:19:25 +0900 Subject: [Python-Dev] Per opcode cache for LOAD_GLOBAL Message-ID: Hi, all. Yury implemented per opcode cache for LOAD_GLOBAL, LOAD_ATTR, and LOAD_METHOD. [1] I update the patch for current master branch, but only for LOAD_GLOBAL for now. [2] It sped up LOAD_GLOBAL about 40%. [3] It is attractive optimization. Now 3.8b1 will be coming soon, but the pull request is not reviewed well yet. For example, should we add some switch to disable the cache? May I merge it before beta1 and polish (or revert) it by rc1? Or should I postpone it for 3.9? Regards, [1]: https://bugs.python.org/issue26219 [2]: https://github.com/python/cpython/pull/12884 [3]: https://github.com/methane/sandbox/tree/master/2019/opcache_load_global#opcache-for-load_global -- Inada Naoki From encukou at gmail.com Wed May 29 09:29:35 2019 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 29 May 2019 15:29:35 +0200 Subject: [Python-Dev] PEP 590: vectorcall without tp_call In-Reply-To: <5CEE7A35.3000106@UGent.be> References: <5CEE7A35.3000106@UGent.be> Message-ID: <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> On 5/29/19 2:25 PM, Jeroen Demeyer wrote: > Hello, > > I have one implementation question about vectorcall which is not > specified in PEP 590: what should happen if a type implements vectorcall > (i.e. _Py_TPFLAGS_HAVE_VECTORCALL is set) but doesn't set tp_call (i.e. > tp_call == NULL)? I see the following possibilities: > > 1. Ignore this problem/assume that it won't happen. This would be bad, > since callable(obj) would be False even though obj() would succeed. > > 2. Raise SystemError. > > 3. Automatically set tp_call to PyVectorcall_Call. > > I would vote for 3 since it's the most user-friendly option. There is > also no way how it could be wrong: it ensures that tp_call and > vectorcall are consistent. That sounds like a good idea for PyType_FromSpec. For static types I either wouldn't bother at all, or only check in debug builds and fail with Py_FatalError. From J.Demeyer at UGent.be Wed May 29 09:36:59 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 29 May 2019 15:36:59 +0200 Subject: [Python-Dev] PEP 590: vectorcall without tp_call In-Reply-To: <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> References: <5CEE7A35.3000106@UGent.be> <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> Message-ID: <5CEE8AFB.7010808@UGent.be> On 2019-05-29 15:29, Petr Viktorin wrote: > That sounds like a good idea for PyType_FromSpec. I don't think we're planning to support vectorcall in PyType_FromSpec for now. That's maybe for 3.9 when vectorcall is no longer provisional. > For static types I either wouldn't bother at all, or only check in debug > builds and fail with Py_FatalError. So basically an assert(...)? From christian at cheimes.de Wed May 29 10:00:40 2019 From: christian at cheimes.de (Christian Heimes) Date: Wed, 29 May 2019 16:00:40 +0200 Subject: [Python-Dev] PEP 590: vectorcall without tp_call In-Reply-To: <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> References: <5CEE7A35.3000106@UGent.be> <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> Message-ID: <2b8badf1-e98b-e4cd-1087-ab6b722c3641@cheimes.de> On 29/05/2019 15.29, Petr Viktorin wrote: > On 5/29/19 2:25 PM, Jeroen Demeyer wrote: >> Hello, >> >> I have one implementation question about vectorcall which is not >> specified in PEP 590: what should happen if a type implements >> vectorcall (i.e. _Py_TPFLAGS_HAVE_VECTORCALL is set) but doesn't set >> tp_call (i.e. tp_call == NULL)? I see the following possibilities: >> >> 1. Ignore this problem/assume that it won't happen. This would be bad, >> since callable(obj) would be False even though obj() would succeed. >> >> 2. Raise SystemError. >> >> 3. Automatically set tp_call to PyVectorcall_Call. >> >> I would vote for 3 since it's the most user-friendly option. There is >> also no way how it could be wrong: it ensures that tp_call and >> vectorcall are consistent. > > That sounds like a good idea for PyType_FromSpec. > > For static types I either wouldn't bother at all, or only check in debug > ?builds and fail with Py_FatalError. You could add a check to PyType_Ready() and have it either return an error or fix tp_call. Christian From J.Demeyer at UGent.be Wed May 29 10:54:11 2019 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 29 May 2019 16:54:11 +0200 Subject: [Python-Dev] PEP 590: vectorcall without tp_call In-Reply-To: <2b8badf1-e98b-e4cd-1087-ab6b722c3641@cheimes.de> References: <5CEE7A35.3000106@UGent.be> <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> <2b8badf1-e98b-e4cd-1087-ab6b722c3641@cheimes.de> Message-ID: <5CEE9D13.7070402@UGent.be> On 2019-05-29 16:00, Christian Heimes wrote: > You could add a check to PyType_Ready() and have it either return an > error or fix tp_call. Yes, but the question is: which of these two alternatives? I would vote for fixing tp_call but Petr voted for an error. From ncoghlan at gmail.com Wed May 29 10:57:42 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 May 2019 00:57:42 +1000 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: <5CEE2124.4020409@canterbury.ac.nz> References: <20190528022848.GF4221@ando.pearwood.info> <5CEE2124.4020409@canterbury.ac.nz> Message-ID: On Wed, 29 May 2019 at 16:08, Greg Ewing wrote: > > Nick Coghlan wrote: > > Having a single locals() call de-optimize an entire function would be > > far from ideal. > > I don't see what would be so bad about that. The vast majority > of functions have no need for locals(). If there was a compelling use case for letting "a = 1; exec(src); print(a)" print something other than "1" at function scope, then I'd be more amenable to the idea of the associated compatibility break and potential performance regression in other implementations. However, there isn't any such use case - if there were, we wouldn't have deliberately changed the semantics from the old Python 2 ones to the current Python 3 ones in PEP 3100 [1]. It's also worth noting that the "no backwards compatibility guarantees" wording is only in the help() text, and not in https://docs.python.org/3/library/functions.html#locals - the latter just notes that writing back to it may not work, not that the semantics may arbitrarily change between CPython versions. I think the [snapshot] approach is a solid improvement over my initial proposal, though, since removing the "locals() must always return the same mapping object" requirement also makes it possible to remove some oddities in the fastlocalsproxy implementation, and Nathaniel makes a compelling case that in the areas where the status quo and the snapshot proposal differ, those differences mostly either don't matter, or else they will serve make code otherwise subject to subtle bugs in tracing mode more correct. Cheers, Nick. [1] "exec as a statement is not worth it -- make it a function" in https://www.python.org/dev/peps/pep-3100/#core-language -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From brett at python.org Wed May 29 12:45:33 2019 From: brett at python.org (Brett Cannon) Date: Wed, 29 May 2019 09:45:33 -0700 Subject: [Python-Dev] Per opcode cache for LOAD_GLOBAL In-Reply-To: References: Message-ID: On Wed, May 29, 2019 at 6:21 AM Inada Naoki wrote: > Hi, all. > > Yury implemented per opcode cache for LOAD_GLOBAL, > LOAD_ATTR, and LOAD_METHOD. [1] > > I update the patch for current master branch, but only for > LOAD_GLOBAL for now. [2] It sped up LOAD_GLOBAL > about 40%. [3] It is attractive optimization. > > Now 3.8b1 will be coming soon, but the pull request is not > reviewed well yet. > For example, should we add some switch to disable the cache? > Why would we want disable it? > > May I merge it before beta1 and polish (or revert) it by rc1? > I think that's an RM call. > Or should I postpone it for 3.9? > That's obviously the simplest. :) -Brett > > Regards, > > [1]: https://bugs.python.org/issue26219 > [2]: https://github.com/python/cpython/pull/12884 > [3]: > https://github.com/methane/sandbox/tree/master/2019/opcache_load_global#opcache-for-load_global > > -- > Inada Naoki > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed May 29 12:49:00 2019 From: brett at python.org (Brett Cannon) Date: Wed, 29 May 2019 09:49:00 -0700 Subject: [Python-Dev] PEP 590: vectorcall without tp_call In-Reply-To: <5CEE9D13.7070402@UGent.be> References: <5CEE7A35.3000106@UGent.be> <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> <2b8badf1-e98b-e4cd-1087-ab6b722c3641@cheimes.de> <5CEE9D13.7070402@UGent.be> Message-ID: On Wed, May 29, 2019 at 7:55 AM Jeroen Demeyer wrote: > On 2019-05-29 16:00, Christian Heimes wrote: > > You could add a check to PyType_Ready() and have it either return an > > error or fix tp_call. > > Yes, but the question is: which of these two alternatives? I would vote > for fixing tp_call but Petr voted for an error. > I also vote for the error so that people are sure they get the semantics they want/expect. If not defining tp_call is a reasonable thing then I would expect it to never have to be set to begin with and that doesn't sound like what anyone wants if we're talking about automatically filling it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Wed May 29 16:11:01 2019 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 29 May 2019 22:11:01 +0200 Subject: [Python-Dev] PEP 590: vectorcall without tp_call In-Reply-To: <5CEE8AFB.7010808@UGent.be> References: <5CEE7A35.3000106@UGent.be> <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> <5CEE8AFB.7010808@UGent.be> Message-ID: On 5/29/19 3:36 PM, Jeroen Demeyer wrote: > On 2019-05-29 15:29, Petr Viktorin wrote: >> That sounds like a good idea for PyType_FromSpec. > > I don't think we're planning to support vectorcall in PyType_FromSpec > for now. That's maybe for 3.9 when vectorcall is no longer provisional. > >> For static types I either wouldn't bother at all, or only check in debug >> builds and fail with Py_FatalError. > > So basically an assert(...)? Yes. That's the usual way to let C extension authors know they did something wrong. From greg.ewing at canterbury.ac.nz Wed May 29 19:09:42 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 30 May 2019 11:09:42 +1200 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> <5CEE2124.4020409@canterbury.ac.nz> Message-ID: <5CEF1136.2040607@canterbury.ac.nz> Nick Coghlan wrote: > If there was a compelling use case for letting "a = 1; exec(src); > print(a)" print something other than "1" at function scope, then I'd > be more amenable to the idea of the associated compatibility break and > potential performance regression in other implementations. > > However, there isn't any such use case - if there were, we wouldn't > have deliberately changed the semantics from the old Python 2 ones to > the current Python 3 ones in PEP 3100 [1]. I get the impression that was done because everyone involved thought it wasn't worth the ugliness of maintaining all the fast/locals swapping stuff, not because of any principle that the current behaviour is right or better in any way. Given a locals proxy object, it would be much easier to support the old behaviour (which seems obvious and correct to me) without eval or exec having to be anything special. -- Greg From ncoghlan at gmail.com Thu May 30 07:31:12 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 30 May 2019 21:31:12 +1000 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: <5CEF1136.2040607@canterbury.ac.nz> References: <20190528022848.GF4221@ando.pearwood.info> <5CEE2124.4020409@canterbury.ac.nz> <5CEF1136.2040607@canterbury.ac.nz> Message-ID: On Thu, 30 May 2019 at 09:12, Greg Ewing wrote: > > Nick Coghlan wrote: > > If there was a compelling use case for letting "a = 1; exec(src); > > print(a)" print something other than "1" at function scope, then I'd > > be more amenable to the idea of the associated compatibility break and > > potential performance regression in other implementations. > > > > However, there isn't any such use case - if there were, we wouldn't > > have deliberately changed the semantics from the old Python 2 ones to > > the current Python 3 ones in PEP 3100 [1]. > > I get the impression that was done because everyone involved > thought it wasn't worth the ugliness of maintaining all the > fast/locals swapping stuff, not because of any principle that > the current behaviour is right or better in any way. You may have felt that way, but I certainly don't - routinely hiding function local rebinding from the compiler (and type checkers, and human readers) is awful, and a privilege that should be reserved to debuggers and other tools operating at a similar level of "able to interfere with the normal runtime execution of a program". (Module and class namespaces are different - they're inherently mutable shared namespaces as far as the Python runtime is occurred, so locals() providing one more way of getting a reference to them isn't that big of a deal - if it's a problem, you can just push the code where you need to avoid those semantics matters down into a helper function) So for me, getting rid of write backs via exec and "import *" was a matter of "Yay, we finally closed those unfortunate loopholes" rather than being any kind of regrettable necessity. It's likely also a factor that Python 2.2.2 was the first version of Python that I ever used extensively, so the existing snapshot-like behaviour of locals() is the behaviour that feels normal to me, and I'm somewhat mystified by the notion that anyone might actually *want* it to behave differently (except to close the remaining loopholes that still allowed mysterious implicit mutation of previously created snapshots). The only rationale for considering that possibility seems to be "It would make function namespaces behave more like class and module namespaces", to which my response is "Why would you even want that, and what practical benefit would it bring to justify the otherwise gratuitous compatibility break?" Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From xdegaye at gmail.com Thu May 30 15:20:15 2019 From: xdegaye at gmail.com (Xavier de Gaye) Date: Thu, 30 May 2019 21:20:15 +0200 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: References: Message-ID: <2d43c69f-49ab-6de5-a5df-a86d4d2ac7c3@gmail.com> Currently f_locals is documented as readonly [1]. The PEP says: > * "Don't change what isn't broken": the current tracing mode problems are caused > by a requirement that's specific to tracing mode (support for external > rebinding of function local variable references), so it made sense to also > restrict any related fixes to tracing mode > > However, actually attempting to implement and document that dynamic approach > highlighted the fact that it makes for a really subtle runtime state dependent > behaviour distinction in how ``frame.f_locals`` works, and creates several > new edge cases around how ``f_locals`` behaves as trace functions are added > and removed. > > Accordingly, the design was switched to the current one, where > ``frame.f_locals`` is always a write-through proxy, and ``locals()`` is always > a dynamic snapshot, which is both simpler to implement and easier to explain. Do these edge cases still exist when f_locals write access is restricted to code executed by the tracing function (which is more restrictive than 'tracing mode') ? We can use the condition frame->f_trace not NULL and tstate->tracing true (tstate being a pointer to the PyThreadState structure) to know when code is executed by the tracing function [2]: * The condition on tstate->tracing allows to figure out if we are running a frame executed by the trace function as opposed to a frame that is being traced or a frame executed in 'regular operation'. * The condition on frame->f_trace removes the ambiguity whether tstate->tracing is set by a tracing function or by a profiling function. [1] In section 'Frame objects' at https://github.com/python/cpython/blob/master/Doc/reference/datamodel.rst#the-standard-type-hierarchy [2] Except that frame->f_trace is NULL in a 'PyTrace_CALL' trace event, so f_locals would remain readonly in that case. But this event is always followed by a 'PyTrace_LINE' event anyway so this limitation is not important IMO. Xavier From ncoghlan at gmail.com Thu May 30 19:06:37 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 May 2019 09:06:37 +1000 Subject: [Python-Dev] PEP 558: Defined semantics for locals() In-Reply-To: <2d43c69f-49ab-6de5-a5df-a86d4d2ac7c3@gmail.com> References: <2d43c69f-49ab-6de5-a5df-a86d4d2ac7c3@gmail.com> Message-ID: On Fri., 31 May 2019, 5:20 am Xavier de Gaye, wrote: > Currently f_locals is documented as readonly [1]. > Read-only in the sense that you can't rebind it to point to a different object - the dict it points to is mutable. > The PEP says: > > * "Don't change what isn't broken": the current tracing mode problems > are caused > > by a requirement that's specific to tracing mode (support for external > > rebinding of function local variable references), so it made sense to > also > > restrict any related fixes to tracing mode > > > > However, actually attempting to implement and document that dynamic > approach > > highlighted the fact that it makes for a really subtle runtime state > dependent > > behaviour distinction in how ``frame.f_locals`` works, and creates > several > > new edge cases around how ``f_locals`` behaves as trace functions are > added > > and removed. > > > > Accordingly, the design was switched to the current one, where > > ``frame.f_locals`` is always a write-through proxy, and ``locals()`` is > always > > a dynamic snapshot, which is both simpler to implement and easier to > explain. > > Do these edge cases still exist when f_locals write access is restricted > to code executed by the tracing function (which is more restrictive than > 'tracing mode') ? We can use the condition frame->f_trace not NULL and tstate->tracing true > (tstate being a pointer to the PyThreadState structure) to know when code > is executed by the tracing function [2]: > * The condition on tstate->tracing allows to figure out if we are running > a frame executed by the trace function as opposed to a frame that is being > traced or a frame executed in 'regular operation'. > * The condition on frame->f_trace removes the ambiguity whether > tstate->tracing is set by a tracing function or by a profiling function. > Always creating the proxy and sometimes bypassing it and returning the snapshot instead would indeed have fewer edge cases than sometimes storing the snapshot directly on the frame object without creating the proxy at all. It's still significantly harder to document than "frame.f_locals references a proxy, locals() creates a snapshot", though. Cheers, Nick. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu May 30 19:22:05 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 May 2019 09:22:05 +1000 Subject: [Python-Dev] Should I postpone PEP 558 (locals() semantics) to Python 3.9? Message-ID: Hi folks, The reference implementation for PEP 558 (my attempt to fix the interaction between tracing functions and closure variables) is currently segfaulting somewhere deep in the garbage collector, and I've found that there's an issue with the PyEval_GetLocals() API returning a borrowed reference that means I need to tweak the proposed C API a bit such that PyEval_GetLocals() returns the proxy at function scope, and we add a new PyEval_GetPyLocals() that matches the locals() builtin. I don't *want* to postpone this to Python 3.9, but there turned out to be more remaining work than I thought there was to get this ready for inclusion in beta 1. I'll try to get the C API design details sorted today, but the segfault is mystifying me, and prevents the option of putting the core implementation in place for b1, and tidying up the documentation and comments for b2. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu May 30 19:26:03 2019 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 31 May 2019 11:26:03 +1200 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: References: <20190528022848.GF4221@ando.pearwood.info> <5CEE2124.4020409@canterbury.ac.nz> <5CEF1136.2040607@canterbury.ac.nz> Message-ID: <5CF0668B.2060209@canterbury.ac.nz> Nick Coghlan wrote: > So for me, getting rid of write backs via exec and "import *" was a > matter of "Yay, we finally closed those unfortunate loopholes" rather > than being any kind of regrettable necessity. If that were the reasoning, the principled thing to do would be to raise an exception if an eval or exec tries to write to a local, rather than mostly ignore it. In any case, I don't really agree with that philosophy. Python is at its essence a dynamic language. Things like JIT and static type analysis are only possible to the extent that you refrain from using some of its dynamic features. Removing features entirely just because they *can* interfere with these things goes against the spirit of the language, IMO. -- Greg From guido at python.org Thu May 30 20:17:44 2019 From: guido at python.org (Guido van Rossum) Date: Thu, 30 May 2019 17:17:44 -0700 Subject: [Python-Dev] Should I postpone PEP 558 (locals() semantics) to Python 3.9? In-Reply-To: References: Message-ID: Given the flurry of discussion that happened this week, and that the PR switching the PEP to use independent snapshots hasn't even been landed, I'm skeptical that it's wise to attempt to get this in before beta 1 even if you were to resolve the segfaults. On Thu, May 30, 2019 at 4:24 PM Nick Coghlan wrote: > Hi folks, > > The reference implementation for PEP 558 (my attempt to fix the > interaction between tracing functions and closure variables) is currently > segfaulting somewhere deep in the garbage collector, and I've found that > there's an issue with the PyEval_GetLocals() API returning a borrowed > reference that means I need to tweak the proposed C API a bit such that > PyEval_GetLocals() returns the proxy at function scope, and we add a new > PyEval_GetPyLocals() that matches the locals() builtin. > > I don't *want* to postpone this to Python 3.9, but there turned out to be > more remaining work than I thought there was to get this ready for > inclusion in beta 1. > > I'll try to get the C API design details sorted today, but the segfault is > mystifying me, and prevents the option of putting the core implementation > in place for b1, and tidying up the documentation and comments for b2. > > Cheers, > Nick. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu May 30 20:22:57 2019 From: guido at python.org (Guido van Rossum) Date: Thu, 30 May 2019 17:22:57 -0700 Subject: [Python-Dev] [PEP 558] thinking through locals() semantics In-Reply-To: <5CF0668B.2060209@canterbury.ac.nz> References: <20190528022848.GF4221@ando.pearwood.info> <5CEE2124.4020409@canterbury.ac.nz> <5CEF1136.2040607@canterbury.ac.nz> <5CF0668B.2060209@canterbury.ac.nz> Message-ID: On Thu, May 30, 2019 at 4:28 PM Greg Ewing wrote: > Nick Coghlan wrote: > > So for me, getting rid of write backs via exec and "import *" was a > > matter of "Yay, we finally closed those unfortunate loopholes" rather > > than being any kind of regrettable necessity. > > If that were the reasoning, the principled thing to do would be > to raise an exception if an eval or exec tries to write to a > local, rather than mostly ignore it. > > In any case, I don't really agree with that philosophy. Python > is at its essence a dynamic language. Things like JIT and static > type analysis are only possible to the extent that you refrain > from using some of its dynamic features. Removing features > entirely just because they *can* interfere with these things goes > against the spirit of the language, IMO. > Right. And static analysis should also be able to detect most uses of locals() in a frame. I believe I've heard of some alternate Python implementations that detect usage of some functions to disable some optimizations (IIRC IronPython did this to sys._getframe()). -- --Guido van Rossum (python.org/~guido) *Pronouns: he/him/his **(why is my pronoun here?)* -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Fri May 31 01:24:06 2019 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 31 May 2019 00:24:06 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: The PR for this looks good to go: https://github.com/python/cpython/pull/13612 But, I still have no idea how it works for the OP's original test case. So, if you have at least 80 GB of RAM to try it, I added `arena.py` to the BPO report: https://bugs.python.org/issue37029 That adds code to the OP's test case to display the times needed to build the tree and to tear it down (& to display some obmalloc stats). So there's no need for you to think about anything ;-) I'm keen to get feedback on this before merging the PR, because this case is so very much larger than anything I've ever tried that I'm wary that there may be more than one "surprise" lurking here. The PR certainly addresses "an obvious" (with hindsight) problem - but is that the _only_ gross design flaw here? From solipsis at pitrou.net Fri May 31 04:22:24 2019 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 31 May 2019 10:22:24 +0200 Subject: [Python-Dev] PEP 595: Improving bugs.python.org References: Message-ID: <20190531102224.0bf2229f@fsol> I second this. There are currently ~7000 bugs open on bugs.python.org. The Web UI makes a good job of actually being able to navigate through these bugs, search through them, etc. Did the Steering Council conduct a usability study of Github Issues with those ~7000 bugs open? If not, then I think the acceptance of migrating to Github is a rushed job. Please reconsider. Regards Antoine. On Thu, 23 May 2019 22:17:24 +0200 Ezio Melotti wrote: > Hello, > Berker and I have been working on a PEP that suggests we keep using > and improving bugs.python.org and Roundup instead of switching to > GitHub Issues as proposed by PEP 581. > > The PEP covers: > * What are the advantages of Roundup over GitHub issues; > * What features are missing in Roundup and how can we add them; > * Issues with PEP 581; > * Issues with the migration plan proposed by PEP 588; > > The rendered version of PEP 595 is available at > https://www.python.org/dev/peps/pep-0595/ > > For reference, you can consult PEP 581 and 588 at > https://www.python.org/dev/peps/pep-0581/ and > https://www.python.org/dev/peps/pep-0588/ > > The full text of the PEP is include below. We are planning to update > the PEP to include the feedback we receive and to update the status of > features as we implement them (we also have a Google Summer of Code > students working on it). > > Best Regards, > Ezio Melotti > > > ================ > PEP: 595 > Title: Improving bugs.python.org > Author: Ezio Melotti , Berker Peksag > > Status: Draft > Type: Process > Content-Type: text/x-rst > Created: 12-May-2019 > > > Abstract > ======== > > This PEP proposes a list of improvements to make bugs.python.org > more usable for contributors and core developers. This PEP also > discusses why remaining on Roundup should be preferred over > switching to GitHub Issues, as proposed by :pep:`581`. > > > Motivation > ========== > > On May 14th, 2019 :pep:`581` has been accepted [#]_ without much > public discussion and without a clear consensus [#]_. The PEP > contains factual errors and doesn't address some of the > issues that the migration to GitHub Issues might present. > > Given the scope of the migration, the amount of work required, > and how it will negatively affect the workflow during the > transition phase, this decision should be re-evaluated. > > > Roundup advantages over GitHub Issues > ===================================== > > This section discusses reasons why Roundup should be preferred > over GitHub Issues and Roundup features that are not available > on GitHub Issues. > > * **Roundup is the status quo.** Roundup has been an integral > part of the CPython workflow for years. It is a stable product > that has been tested and customized to adapt to our needs as the > workflow evolved. > > It is possible to gradually improve it and avoid the disruption > that a switch to a different system would inevitabily bring to > the workflow. > > * **Open-source and Python powered.** Roundup is an open-source > project and is written in Python. By using it and supporting > it, we also support the Python ecosystem. Several features > developed for bpo have also been ported to upstream Roundup > over the years. > > * **Fully customizable.** Roundup can be (and has been) fully > customized to fit our needs. > > * **Finer-grained access control.** Roundup allows the creation > of different roles with different permissions (e.g. create, > view, edit, etc.) for each individual property, and users can > have multiple roles. > > * **Flexible UI.** While Roundup UI might look dated, it is > convenient and flexible. > > For example, on the issue page, each field (e.g. title, type, > versions, status, linked files and PRs, etc.) have appropriate > UI elements (input boxes, dropdowns, tables, etc.) that are > easy to set and also provide a convenient way to get info about > the issue at a glance. The number of fields, their values, and > the UI element they use is also fully customizable. > GitHub only provides labels. > > The issue list page presents the issues in a compact and easy > to read table with separate columns for different fields. For > comparison, Roundup lists 50 issues in a screen, whereas GitHub > takes two screens to shows 25 issues. > > * **Advanced search.** Roundup provides an accurate way to search > and filter by using any combination of issue fields. > It is also possible to customize the number of results and the > fields displayed in the table, and the sorting and grouping > (up to two levels). > > bpo also provides predefined summaries (e.g. "Created by you", > "Assigned to you", etc.) and allows the creation of custom > search queries that can be conveniently accessed from the sidebar. > > * **Nosy list autocomplete.** The nosy list has an autocomplete > feature that suggests maintainers and experts. The suggestions > are automatically updated when the experts index [#]_ changes. > > * **Dependencies and Superseders.** Roundup allows to specify > dependencies that must be addressed before the current issues > can be closed and a superseder issue to easily mark duplicates > [#]_. The list of dependencies can also be used to create > meta-issues that references several other sub-issues [#]_. > > > Improving Roundup > ================= > > This section lists some of the issues mentioned by :pep:`581` > and other desired features and discusses how they can be implemented > by improving Roundup and/or our instance. > > * **REST API support.** A REST API will make integration with other > services and the development of new tools and applications easiers. > > Upstream Roundup now supports a REST API. Updating the tracker will > make the REST API available. > > * **GitHub login support.** This will allow users to login > to bugs.python.org (bpo) without having to create a new account. > It will also solve issues with confirmation emails being marked > as spam, and provide two-factor authentication. > > A patch to add this functionality is already available and is > being integrated at the time of writing [#]_. > > * **Markdown support and message preview and editing.** This feature > will allow the use of Markdown in messages and the ability to > preview the message before the submission and edit it afterward. > > This can be done, but it will take some work. Possible solutions > have been proposed on the roundup-devel mailing list [#]_. > > * **"Remove me from nosy list" button.** Add a button on issue pages > to remove self from the nosy list. > > This feature will be added during GSoC 2019. > > * **Mobile friendly theme.** Current theme of bugs.python.org looks > dated and it doesn't work well with mobile browsers. > > A mobile-friendly theme that is more modern but still familiar > will be added. > > * **Add PR link to BPO emails.** Currently bpo emails don't include > links to the corresponding PRs. > > A patch [#]_ is available to change the content of the bpo emails > from:: > > components: +Tkinter > versions: +Python 3.4 > pull_requests: +42 > > to:: > > components: +Tkinter > versions: +Python 3.4 > pull_request: https://github.com/python/cpython/pull/341 > > * **Python 3 support.** Using Python 3 will make maintenance easier. > > Upstream Roundup now supports Python 3. Updating the tracker will > allow us to switch to Python 3. The instances will need to be > updated as well. > > * **Use upstream Roundup.** We currently use a fork of Roundup with > a few modifications, most notably the GitHub integration. If this > is ported upstream, we can start using upstream Roundup without > having to maintain our fork. > > > PEP 581 issues > ============== > > This section addresses some errors and inaccuracies found in :pep:`581`. > > The "Why GitHub?" section of PEP 581 lists features currently > available on GitHub Issues but not on Roundup. Some of this features > are currently supported: > > * "Ability to reply to issue and pull request conversations via email." > > * Being able to reply by email has been one of the core features of > Roundup since the beginning. It is also possible to create new > issues or close existing ones, set or modify fields, and add > attachments. > > * "Email notifications containing metadata, integrated with Gmail, > allowing systematic filtering of emails." > > * Emails sent by Roundup contains metadata that can be used for > filtering. > > * "Additional privacy, such as offering the user a choice to hide an > email address, while still allowing communication with the user > through @-mentions." > > * Email addresses are hidden by default to users that are not > registered. Registered users can see other users' addresses > because we configured the tracker to show them. It can easily > be changed if desired. Users can still be added to the nosy > list by using their username even if their address is hidden. > > * "Ability to automatically close issues when a PR has been merged." > > * The GitHub integration of Roundup automatically closes issues > when a commit that contains "fixes issue " is merged. > (Alternative spellings such as "closes" or "bug" are also supported.) > See [#]_ for a recent example of this feature. > > * "Support for permalinks, allowing easy quoting and copying & > pasting of source code." > > * Roundup has permalinks for issues, messages, attachments, etc. > In addition, Roundup allows to easily rewrite broken URLs in > messages (e.g. if the code hosting changes). > > * "Core developers, volunteers, and the PSF don't have to maintain the > issue infrastructure/site, giving us more time and resources to focus > on the development of Python." > > * While this is partially true, additional resources are required to > write and maintain bots. > > In some cases, bots are required to workaround GitHub's lack of > features rather than expanding. [#]_ was written > specifically to workaround GitHub's email integration. > > Updating our bots to stay up-to-date with changes in the GitHub API > has also maintenance cost. [#]_ took two days to be fixed. > > In addition, we will still need to maintain Roundup for bpo (even > if it becomes read-only) and for the other trackers > we currently host/maintain (Jython [#]_ and Roundup [#]_). > > The "Issues with Roundup / bpo" section of :pep:`581` lists some issues > that have already been fixed: > > * "The upstream Roundup code is in Mercurial. Without any CI available, > it puts heavy burden on the few existing maintainers in terms of > reviewing, testing, and applying patches." > > * While Roundup uses Mercurial by default, there is a git clone > available on GitHub [#]_. Roundup also has CI available [#]_ [#]_. > > * "There is no REST API available. There is an open issue in Roundup for > adding REST API. Last activity was in 2016." > > * The REST API has been integrated and it's now available in Roundup. > > * "Users email addresses are exposed. There is no option to mask it." > > * Exposing addresses to registered and logged in users was a decision > taken when our instance was set up. > > This has now been changed to make the email addresses hidden for > regular users too (Developers and Coordinators can still see them). > The "Email address"" column from the user listing page [#]_ has been > removed too. > > * "It sends a number of unnecessary emails and notifications, and it is > difficult, if not impossible, to configure." > > * This can be configured. > > * "Creating an account has been a hassle. There have been reports of people > having trouble creating accounts or logging in." > > * The main issue is confirmation emails being marked as spam. Work has > been done to resolve the issue. > > Migration considerations > ======================== > > This section describes issues with the migrations that might not > have been addressed by :pep:`581` and :pep:`588`. > > :pep:`588` suggests to add a button to migrate issues to GitHub > only when someone wants to keep working on them. This approach > has several issues: > > * bpo will need to be updated in order to add a button that, > once pressed, creates a new issue on GitHub, copies over all > the messages, attachments, and creates/adds label for the > existing fields. Permissions will also need to be tweaked > to make individual issues read-only once they are migrated, > and to prevent users to create new accounts. > > * The issues will be split between two trackers; searching issues > will take significant more effort. > > * The conversion from Roundup to GitHub is lossy, unless all > the bpo fields are converted into labels or preserved somewhere > else. > > * bpo converts a number of references into links, including > issue, message, and PR IDs, changeset numbers, legacy SVN > revision numbers, paths to files in the repo, files in > tracebacks (detecting the correct branch), links to devguide > pages and sections [#]_. This happens when messages are > requested so it is possible to create the correct link (e.g. > all the file links used to point to hg.python.org and now > point to GitHub). > > If the links are hardcoded during the migration, it will be > difficult (if not impossible) to change them later. If they > aren't, they will either be lost, or a tool to generate the > links and updating them will need to be written. > > * GitHub doesn't provide a way to set and preserve issue IDs > if they are migrated automatically with the use of a button. > (Some projects managed to preserve the IDs by contaacting > the GitHub staff and migrating the issues *en masse*.) > > * On top of the work and changes required to migrate to GitHub > issues, we will still need to keep running and maintaining > Roundup, for both our instance (read-only) and for the Jython > and Roundup trackers (read-write). > > > In addition to the issues listed in the "Open issues" section of > :pep:`588`, this issues will need to be addressed: > > * GitHub is properietary and there is risk of vendor lock-in. > Their business model might change and they could shut down > altogether. > > * Switching to GitHub Issues will likely increase the number of > invalid reports and increase the triaging effort. This concern > has been raised in the past in a Zulip topic [#]_. > > There have been already cases where people posted comments on > PRs that required moderators to mark them as off-topic or > disruptive, delete them altogether, and even lock the > conversation [#]_. > > * Roundup sends weekly reports to python-dev with a summary that > includes new issues, recent issues with no replies, recent > issues waiting for review, most discussed issues, closed issues, > and deltas for open/closed/total issue counts [#]_. The report > provides an easy way to keep track of the tracker activity and > to make sure that issues that require attention are noticed. > > The data collect by the weekly report is also use to generate > statistics and graphs that can be used to gain new insights [#]_. > > * There are currently two mailing lists where Roundup posts new > tracker issues and all messages respectively: new-bugs-announce > [#]_ and python-bugs-list [#]_. A new system will need to be > developed to preserve this functionality. These MLs offer > additional ways to keep track of the tracker activity. > > > References > ========== > > .. [#] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted > > https://mail.python.org/pipermail/python-dev/2019-May/157399.html > > .. [#] [python-committers] [Python-Dev] PEP 581 (Using GitHub issues > for CPython) is accepted > > https://mail.python.org/pipermail/python-committers/2019-May/006755.html > > .. [#] Experts Index -- Python Devguide > > https://devguide.python.org/experts/ > > .. [#] An example of superseded issues: > "re.sub() replaces only several matches" > > https://bugs.python.org/issue12078 > > .. [#] An example of meta issue using dependencies to track sub-issues: > "Meta-issue: support of the android platform"" > > https://bugs.python.org/issue26865 > > .. [#] Support logging in with GitHub > > https://github.com/python/bugs.python.org/issues/7 > > .. [#] Re: [Roundup-devel] PEP 581 and Google Summer of Code > > https://sourceforge.net/p/roundup/mailman/message/36667828/ > > .. [#] [Tracker-discuss] [issue624] bpo emails contain useless non-github > pull_request number - users want a link to actual github PR > > https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html > > .. [#] The commit reported in msg342882 closes the issue (see the history below) > > https://bugs.python.org/issue36951#msg342882 > > .. [#] The cpython-emailer-webhook project > > https://github.com/berkerpeksag/cpython-emailer-webhook > > .. [#] A recent incident caused by GitHub > > https://github.com/python/bedevere/pull/163 > > .. [#] Jython issue tracker > > https://bugs.jython.org/ > > .. [#] Roundup issue tracker > > https://issues.roundup-tracker.org/ > > .. [#] GitHub clone of Roundup > > https://github.com/roundup-tracker/roundup > > .. [#] Travis-CI for Roundup > > https://travis-ci.org/roundup-tracker/roundup) and codecov > > .. [#] Codecov for Roundup > > https://codecov.io/gh/roundup-tracker/roundup/commits > > .. [#] User listing -- Python tracker > > https://bugs.python.org/user?@sort=username > > .. [#] Generating Special Links in a Comment -- Python Devguide > > https://devguide.python.org/triaging/#generating-special-links-in-a-comment > > .. [#] The New-bugs-announce mailing list > > https://mail.python.org/mailman/listinfo/new-bugs-announce > > .. [#] The Python-bugs-list mailing list > > https://mail.python.org/mailman/listinfo/python-bugs-list > > .. [#] An example of [Python-Dev] Summary of Python tracker Issues > > https://mail.python.org/pipermail/python-dev/2019-May/157483.html > > .. [#] Issues stats -- Python tracker > > https://bugs.python.org/issue?@template=stats > > .. [#] s/n ratio -- Python -- Zulip > > https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20ratio > > .. [#] For example this and other related PRs: > > https://github.com/python/cpython/pull/9099 > > > Copyright > ========= > > This document has been placed in the public domain. > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: From njs at pobox.com Fri May 31 04:34:08 2019 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 31 May 2019 01:34:08 -0700 Subject: [Python-Dev] Should I postpone PEP 558 (locals() semantics) to Python 3.9? In-Reply-To: References: Message-ID: I wouldn't mind having a little more breathing room. It's frustrating to miss the train, but these bugs are several decades old so I guess nothing terrible will happen if their fixes get delayed to 3.9. On Thu, May 30, 2019 at 4:23 PM Nick Coghlan wrote: > > Hi folks, > > The reference implementation for PEP 558 (my attempt to fix the interaction between tracing functions and closure variables) is currently segfaulting somewhere deep in the garbage collector, and I've found that there's an issue with the PyEval_GetLocals() API returning a borrowed reference that means I need to tweak the proposed C API a bit such that PyEval_GetLocals() returns the proxy at function scope, and we add a new PyEval_GetPyLocals() that matches the locals() builtin. > > I don't *want* to postpone this to Python 3.9, but there turned out to be more remaining work than I thought there was to get this ready for inclusion in beta 1. > > I'll try to get the C API design details sorted today, but the segfault is mystifying me, and prevents the option of putting the core implementation in place for b1, and tidying up the documentation and comments for b2. > > Cheers, > Nick. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com -- Nathaniel J. Smith -- https://vorpus.org From encukou at gmail.com Fri May 31 04:46:58 2019 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 31 May 2019 10:46:58 +0200 Subject: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures Message-ID: <67fbae0d-0155-a21d-cd4b-6e16628debb6@gmail.com> Hello, PEP 570 (Positional-Only Parameters) changed the signatures of PyCode_New() and types.CodeType(), adding a new argument for "posargcount". Our policy for such changes seems to be fragmented tribal knowledge. I'm writing to check if my understanding is reasonable, so I can apply it and document it explicitly. There is a surprisingly large ecosystem of tools that create code objects. The expectation seems to be that these tools will need to be adapted for each minor version of Python. But that's not the same as saying that if you use types.CodeType(), you're on your own. To me [PR 13271], seems to go too far when it adds: > These types are not supposed to be instantiated outside of > CPython internals and constructor signatures will vary > between Python versions. This kind of "washing our hands" doesn't do justice to tools like Cython, whose use of PyCode_New is (IMO) perfectly justified. As is adapting to Python minor versions. So, we should document the changes as any backwards-incompatible change. Specifically, changes should be (or should have been, in hindsight) mentioned in: * The "API changes" and/or "Porting" sections of the What's New document * Version history of the documentation (e.g. versionchanged blocks) * Any relevant PEPs Also, the expected level of API stability should be documented in the docs. Does that sound right? PEP 570: https://www.python.org/dev/peps/pep-0570/ PyCode_New: https://docs.python.org/3.8/c-api/code.html#c.PyCode_New [PR 13271]: https://github.com/python/cpython/pull/13271/files From songofacandy at gmail.com Fri May 31 06:07:51 2019 From: songofacandy at gmail.com (Inada Naoki) Date: Fri, 31 May 2019 19:07:51 +0900 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: > I'm keen to get feedback on this before merging the PR, because this > case is so very much larger than anything I've ever tried that I'm > wary that there may be more than one "surprise" lurking here. The PR > certainly addresses "an obvious" (with hindsight) problem - but is > that the _only_ gross design flaw here? I started r5a.4xlarge EC2 instance and started arena.py. I will post the result in next 12 hours. Regards, -- Inada Naoki From ncoghlan at gmail.com Fri May 31 06:20:59 2019 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 31 May 2019 20:20:59 +1000 Subject: [Python-Dev] Should I postpone PEP 558 (locals() semantics) to Python 3.9? In-Reply-To: References: Message-ID: On Fri., 31 May 2019, 6:34 pm Nathaniel Smith, wrote: > I wouldn't mind having a little more breathing room. It's frustrating > to miss the train, but these bugs are several decades old so I guess > nothing terrible will happen if their fixes get delayed to 3.9. > And I could put that extra time to good use, as starting to flesh out the proxy implementation showed that we're missing a lot of scaffolding to help make it easier to define new low level mapping types without duplicating a lot of code. I'll update the PEP headers accordingly. Cheers, Nick. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri May 31 12:37:01 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 31 May 2019 12:37:01 -0400 Subject: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures In-Reply-To: <67fbae0d-0155-a21d-cd4b-6e16628debb6@gmail.com> References: <67fbae0d-0155-a21d-cd4b-6e16628debb6@gmail.com> Message-ID: On 5/31/2019 4:46 AM, Petr Viktorin wrote: > Hello, > PEP 570 (Positional-Only Parameters) changed the signatures of > PyCode_New() and types.CodeType(), adding a new argument for "posargcount". > Our policy for such changes seems to be fragmented tribal knowledge. I'm > writing to check if my understanding is reasonable, so I can apply it > and document it explicitly. > > There is a surprisingly large ecosystem of tools that create code objects. > The expectation seems to be that these tools will need to be adapted for > each minor version of Python. > > But that's not the same as saying that if you use types.CodeType(), > you're on your own. To me [PR 13271], seems to go too far when it adds: > > These types are not supposed to be instantiated outside of > > CPython internals and constructor signatures will vary > > between Python versions. > > This kind of "washing our hands" doesn't do justice to tools like > Cython, whose use of PyCode_New is (IMO) perfectly justified. As is > adapting to Python minor versions. I summarized the above on the issue https://bugs.python.org/issue36896 and suggested specific revisions on the PR. https://github.com/python/cpython/pull/13271 > So, we should document the changes as any backwards-incompatible change. > Specifically, changes should be (or should have been, in hindsight) > mentioned in: > * The "API changes" and/or "Porting" sections of the What's New document The second PR for Document changes ... added something to What's New. https://bugs.python.org/issue36886 https://github.com/python/cpython/commit/5d23e282af69d404a3430bb95aefe371112817b3 If you think something more is needed, say so on the issue. > * Version history of the documentation (e.g. versionchanged blocks) We put version history of the code (but not of the docs) in the docs. https://docs.python.org/3/reference/compound_stmts.html#function-definitions should have a version-changed note for the added '/' syntax. > * Any relevant PEPs If you mean PEPs other than 570, we don't do that. > Also, the expected level of API stability should be documented in the docs. Yes. I think that part of PR 13271, rewritten, is good. > PEP 570: https://www.python.org/dev/peps/pep-0570/ > PyCode_New: https://docs.python.org/3.8/c-api/code.html#c.PyCode_New > [PR 13271]: https://github.com/python/cpython/pull/13271/files -- Terry Jan Reedy From tjreedy at udel.edu Fri May 31 12:43:20 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 31 May 2019 12:43:20 -0400 Subject: [Python-Dev] Should I postpone PEP 558 (locals() semantics) to Python 3.9? In-Reply-To: References: Message-ID: On 5/31/2019 6:20 AM, Nick Coghlan wrote: > On Fri., 31 May 2019, 6:34 pm Nathaniel Smith, > wrote: > > I wouldn't mind having a little more breathing room. It's frustrating > to miss the train, but these bugs are several decades old so I guess > nothing terrible will happen if their fixes get delayed to 3.9. Agreed. > And I could put that extra time to good use, as starting to flesh out > the proxy implementation showed that we're missing a lot of scaffolding > to help make it easier to define new low level mapping types without > duplicating a lot of code. > > I'll update the PEP headers accordingly. I believe some of your suggested doc change is true now, will remain true, and improves on the current locals entry. No PEP approval is needed for this much. -- Terry Jan Reedy From status at bugs.python.org Fri May 31 14:08:08 2019 From: status at bugs.python.org (Python tracker) Date: Fri, 31 May 2019 18:08:08 +0000 (UTC) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20190531180808.52ABF52B006@bugs.ams1.psf.io> ACTIVITY SUMMARY (2019-05-24 - 2019-05-31) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 7028 (-39) closed 41839 (+118) total 48867 (+79) Open issues with patches: 2820 Issues opened (50) ================== #26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows https://bugs.python.org/issue26423 reopened by vstinner #34725: Py_GetProgramFullPath() odd behaviour in Windows https://bugs.python.org/issue34725 reopened by vstinner #35753: Importing call from unittest.mock directly causes ValueError https://bugs.python.org/issue35753 reopened by pablogsal #36889: Merge StreamWriter and StreamReader into just asyncio.Stream https://bugs.python.org/issue36889 reopened by koobs #36991: zipfile: AttributeError on extract https://bugs.python.org/issue36991 reopened by berker.peksag #37038: Make idlelib/run.py runnable. https://bugs.python.org/issue37038 opened by terry.reedy #37039: IDLE: Improve zoomheight doc and behavior. https://bugs.python.org/issue37039 opened by terry.reedy #37040: checking for membership in itertools.count enters infinite loo https://bugs.python.org/issue37040 opened by Dan Rose #37041: IDLE: path browser unusable on some displays https://bugs.python.org/issue37041 opened by aroberge #37042: wait_for(coro, timeout=0) memleak https://bugs.python.org/issue37042 opened by ixje #37043: Buildbots fail when new files are added https://bugs.python.org/issue37043 opened by jaraco #37044: Build/test artifacts not ignored for framework build https://bugs.python.org/issue37044 opened by jaraco #37048: ssl module: QUIC support for HTTP/3 https://bugs.python.org/issue37048 opened by christian.heimes #37052: Add examples for mocking async for and async context manager i https://bugs.python.org/issue37052 opened by xtreak #37055: Numerous warnings with blake2 module https://bugs.python.org/issue37055 opened by remi.lapeyre #37060: import ctypes fails with a statically linked interpreter due t https://bugs.python.org/issue37060 opened by indygreg #37062: `AutoNumber` class in enum documentation: support *args in con https://bugs.python.org/issue37062 opened by rrt #37063: Incorrect application of func.__defaults__ by inspect's signat https://bugs.python.org/issue37063 opened by bup #37064: Feature request: option to keep/add flags to pathfix. https://bugs.python.org/issue37064 opened by pkopkan #37068: Emit SyntaxWarning for f-strings without expressions ? https://bugs.python.org/issue37068 opened by mbussonn #37069: regrtest: log unraisable exceptions and uncaught thread except https://bugs.python.org/issue37069 opened by vstinner #37071: HTMLParser mistakenly inventing new tags while parsing https://bugs.python.org/issue37071 opened by htran #37073: clarify functions docs in IO modules and Bytes Objects https://bugs.python.org/issue37073 opened by Windson Yang #37074: os.stat() does not work for NUL and CON https://bugs.python.org/issue37074 opened by serhiy.storchaka #37075: Error message improvement for AsyncMock https://bugs.python.org/issue37075 opened by xtreak #37077: Threading: add builtin TID attribute to Thread objects for AIX https://bugs.python.org/issue37077 opened by Michael.Felt #37080: Cannot compile Python3.7.3 on Alt-F (ARM) https://bugs.python.org/issue37080 opened by Jarl Gullberg #37081: Test with OpenSSL 1.1.1c https://bugs.python.org/issue37081 opened by christian.heimes #37082: Assignment expression operator (walrus) not in built-in help() https://bugs.python.org/issue37082 opened by mbussonn #37083: Document TYPE_COMMENT in documentation reference for compound https://bugs.python.org/issue37083 opened by pablogsal #37084: _ctypes not failing, can't find reason https://bugs.python.org/issue37084 opened by Brian Spratke #37085: Expose additional socket constants for CAN_BCM flags https://bugs.python.org/issue37085 opened by karlding #37086: time.sleep error message misleading https://bugs.python.org/issue37086 opened by Justin Fay #37087: Adding native id support for openbsd https://bugs.python.org/issue37087 opened by David Carlier #37088: Add a way to schedule a function to be called from the main th https://bugs.python.org/issue37088 opened by yselivanov #37089: `import Lib.os` works on windows (but shouldn't) https://bugs.python.org/issue37089 opened by Anthony Sottile #37090: test_gdb's test_pycfunction should test all calling convention https://bugs.python.org/issue37090 opened by petr.viktorin #37091: subprocess - uncaught PermissionError in send_signal can cause https://bugs.python.org/issue37091 opened by wesinator #37093: http.client aborts header parsing upon encountering non-ASCII https://bugs.python.org/issue37093 opened by tburke #37095: [Feature Request]: Add zstd support in tarfile https://bugs.python.org/issue37095 opened by evan0greenup #37096: Add large-file tests for modules using sendfile(2) https://bugs.python.org/issue37096 opened by giampaolo.rodola #37097: python_is_optimized() false negatives https://bugs.python.org/issue37097 opened by jdemeyer #37098: test_memfd_create() test failure https://bugs.python.org/issue37098 opened by jdemeyer #37100: test_coroutine.test_unawaited_warning_when_module_broken fails https://bugs.python.org/issue37100 opened by xtreak #37102: Automatically dedent docstring constants by default https://bugs.python.org/issue37102 opened by gregory.p.smith #37103: Undo deprecation of optparse https://bugs.python.org/issue37103 opened by andersk #37105: Add deprecated-remove information on stream doc https://bugs.python.org/issue37105 opened by eamanu #37107: ensurepip --upgrade doesn't change the version of pip used by https://bugs.python.org/issue37107 opened by cjw296 #37110: Clarify hashability of custom class instances https://bugs.python.org/issue37110 opened by cito #37116: Use PEP 570 syntax for positional-only parameters https://bugs.python.org/issue37116 opened by serhiy.storchaka Most recent 15 issues with no replies (15) ========================================== #37116: Use PEP 570 syntax for positional-only parameters https://bugs.python.org/issue37116 #37110: Clarify hashability of custom class instances https://bugs.python.org/issue37110 #37107: ensurepip --upgrade doesn't change the version of pip used by https://bugs.python.org/issue37107 #37105: Add deprecated-remove information on stream doc https://bugs.python.org/issue37105 #37102: Automatically dedent docstring constants by default https://bugs.python.org/issue37102 #37097: python_is_optimized() false negatives https://bugs.python.org/issue37097 #37096: Add large-file tests for modules using sendfile(2) https://bugs.python.org/issue37096 #37095: [Feature Request]: Add zstd support in tarfile https://bugs.python.org/issue37095 #37093: http.client aborts header parsing upon encountering non-ASCII https://bugs.python.org/issue37093 #37090: test_gdb's test_pycfunction should test all calling convention https://bugs.python.org/issue37090 #37087: Adding native id support for openbsd https://bugs.python.org/issue37087 #37085: Expose additional socket constants for CAN_BCM flags https://bugs.python.org/issue37085 #37084: _ctypes not failing, can't find reason https://bugs.python.org/issue37084 #37080: Cannot compile Python3.7.3 on Alt-F (ARM) https://bugs.python.org/issue37080 #37077: Threading: add builtin TID attribute to Thread objects for AIX https://bugs.python.org/issue37077 Most recent 15 issues waiting for review (15) ============================================= #37116: Use PEP 570 syntax for positional-only parameters https://bugs.python.org/issue37116 #37105: Add deprecated-remove information on stream doc https://bugs.python.org/issue37105 #37098: test_memfd_create() test failure https://bugs.python.org/issue37098 #37096: Add large-file tests for modules using sendfile(2) https://bugs.python.org/issue37096 #37090: test_gdb's test_pycfunction should test all calling convention https://bugs.python.org/issue37090 #37088: Add a way to schedule a function to be called from the main th https://bugs.python.org/issue37088 #37085: Expose additional socket constants for CAN_BCM flags https://bugs.python.org/issue37085 #37081: Test with OpenSSL 1.1.1c https://bugs.python.org/issue37081 #37077: Threading: add builtin TID attribute to Thread objects for AIX https://bugs.python.org/issue37077 #37075: Error message improvement for AsyncMock https://bugs.python.org/issue37075 #37064: Feature request: option to keep/add flags to pathfix. https://bugs.python.org/issue37064 #37039: IDLE: Improve zoomheight doc and behavior. https://bugs.python.org/issue37039 #37038: Make idlelib/run.py runnable. https://bugs.python.org/issue37038 #37034: Argument Clinic omits name of keyword-only parameter on _PyArg https://bugs.python.org/issue37034 #37030: Lib/cmd.py: Hide undocumented commands in help and completenam https://bugs.python.org/issue37030 Top 10 most discussed issues (10) ================================= #37039: IDLE: Improve zoomheight doc and behavior. https://bugs.python.org/issue37039 20 msgs #33725: Python crashes on macOS after fork with no exec https://bugs.python.org/issue33725 16 msgs #36839: Support the buffer protocol in code objects https://bugs.python.org/issue36839 12 msgs #36974: Implement PEP 590 https://bugs.python.org/issue36974 12 msgs #35545: asyncio.base_events.create_connection doesn't handle scoped IP https://bugs.python.org/issue35545 9 msgs #36709: Asyncio SSL keep-alive connections raise errors after loop clo https://bugs.python.org/issue36709 9 msgs #36889: Merge StreamWriter and StreamReader into just asyncio.Stream https://bugs.python.org/issue36889 8 msgs #37011: pdb: restore original tracing function instead of sys.settrace https://bugs.python.org/issue37011 8 msgs #26423: Integer overflow in wrap_lenfunc() on 64-bit build of Windows https://bugs.python.org/issue26423 6 msgs #28866: Type cache is not correctly invalidated on a class defining mr https://bugs.python.org/issue28866 6 msgs Issues closed (115) =================== #5028: tokenize.generate_tokens doesn't always return logical line https://bugs.python.org/issue5028 closed by cheryl.sabella #11320: Usage of API method Py_SetPath causes errors in Py_Initialize( https://bugs.python.org/issue11320 closed by vstinner #13533: Would like Py_Initialize to play friendly with host app https://bugs.python.org/issue13533 closed by vstinner #14656: Add a macro for unreachable code https://bugs.python.org/issue14656 closed by pablogsal #14956: custom PYTHONPATH may break apps embedding Python https://bugs.python.org/issue14956 closed by vstinner #19569: Use __attribute__((deprecated)) to warn usage of deprecated fu https://bugs.python.org/issue19569 closed by vstinner #19983: When interrupted during startup, Python should not call abort( https://bugs.python.org/issue19983 closed by vstinner #22102: Zipfile generates Zipfile error in zip with 0 total number of https://bugs.python.org/issue22102 closed by cheryl.sabella #22454: Adding the opposite function of shlex.split() https://bugs.python.org/issue22454 closed by berker.peksag #22640: Add silent mode for py_compile https://bugs.python.org/issue22640 closed by berker.peksag #24564: shutil.copytree fails when copying NFS to NFS https://bugs.python.org/issue24564 closed by giampaolo.rodola #24732: 3.5.0b3 Windows accept() on unready non-blocking socket raises https://bugs.python.org/issue24732 closed by vstinner #25735: math.factorial doc should mention integer return type https://bugs.python.org/issue25735 closed by cheryl.sabella #27639: UserList.__getitem__ doesn't account for slices https://bugs.python.org/issue27639 closed by SilentGhost #27987: obmalloc's 8-byte alignment causes undefined behavior https://bugs.python.org/issue27987 closed by inada.naoki #29143: Logger should ignore propagate property for disabled handlers. https://bugs.python.org/issue29143 closed by mariocj89 #29262: Provide a way to check for *real* typing.Union instances https://bugs.python.org/issue29262 closed by levkivskyi #29696: Use namedtuple in string.Formatter.parse iterator response https://bugs.python.org/issue29696 closed by rhettinger #29778: _Py_CheckPython3 uses uninitialized dllpath when embedder sets https://bugs.python.org/issue29778 closed by vstinner #29883: asyncio: Windows Proactor Event Loop UDP Support https://bugs.python.org/issue29883 closed by asvetlov #30006: Deadlocks in `concurrent.futures.ProcessPoolExecutor` https://bugs.python.org/issue30006 closed by pitrou #30560: Py_SetFatalErrorAbortFunc: Allow embedding program to handle f https://bugs.python.org/issue30560 closed by vstinner #30969: Docs should say that `x is z or x == z` is used for `x in y` i https://bugs.python.org/issue30969 closed by cheryl.sabella #31745: Overloading "Py_GetPath" does not work https://bugs.python.org/issue31745 closed by vstinner #32299: unittest.mock.patch.dict.__enter__ should return the dict https://bugs.python.org/issue32299 closed by cheryl.sabella #32388: Remove cross-version binary compatibility https://bugs.python.org/issue32388 closed by pitrou #32941: mmap should expose madvise() https://bugs.python.org/issue32941 closed by pitrou #33006: docstring of filter function is incorrect https://bugs.python.org/issue33006 closed by ned.deily #33135: Define field prefixes for the various config structs https://bugs.python.org/issue33135 closed by vstinner #33407: Implement Py_DEPRECATED() macro for Visual Studio https://bugs.python.org/issue33407 closed by vstinner #33918: Hooking into pause/resume of iterators/coroutines https://bugs.python.org/issue33918 closed by asvetlov #33919: Expose _PyCoreConfig structure to Python https://bugs.python.org/issue33919 closed by vstinner #34148: Fatal read error on socket transport https://bugs.python.org/issue34148 closed by asvetlov #34613: asyncio.StreamReader initialization documentation incorrectly https://bugs.python.org/issue34613 closed by asvetlov #34626: PEP 384's PyType_Spec and PyType_Slot are not documented https://bugs.python.org/issue34626 closed by petr.viktorin #34632: Port importlib_metadata to Python 3.8 https://bugs.python.org/issue34632 closed by barry #34655: Support sendfile in asyncio streams API https://bugs.python.org/issue34655 closed by asvetlov #34791: xml package does not obey sys.flags.ignore_environment https://bugs.python.org/issue34791 closed by vstinner #34818: test.test_ssl.ThreadedTests.test_tls1_3 fails in 2.7 with Attr https://bugs.python.org/issue34818 closed by cheryl.sabella #34975: start_tls() difficult when using asyncio.start_server() https://bugs.python.org/issue34975 closed by asvetlov #34993: asyncio.streams.FlowControlMixin should be part of the API https://bugs.python.org/issue34993 closed by asvetlov #35246: asyncio.create_subprocess_exec doesn't accept pathlib.Path lik https://bugs.python.org/issue35246 closed by asvetlov #35279: asyncio uses too many threads by default https://bugs.python.org/issue35279 closed by inada.naoki #35397: Undeprecate and document urllib.parse.unwrap https://bugs.python.org/issue35397 closed by cheryl.sabella #35925: test_httplib test_nntplib test_ssl fail on ARMv7 Debian buster https://bugs.python.org/issue35925 closed by ned.deily #36204: Deprecate calling Py_Main() after Py_Initialize()? Add Py_Init https://bugs.python.org/issue36204 closed by vstinner #36271: '_io.TextIOWrapper' object has no attribute 'mode' https://bugs.python.org/issue36271 closed by berker.peksag #36540: PEP 570: Python Positional-Only Parameters https://bugs.python.org/issue36540 closed by pablogsal #36610: os.sendfile can return EINVAL on Solaris https://bugs.python.org/issue36610 closed by giampaolo.rodola #36686: Docs: asyncio.loop.subprocess_exec documentation is confusing, https://bugs.python.org/issue36686 closed by sbstp #36721: Add pkg-config python-3.8-embed and --embed to python3.8-confi https://bugs.python.org/issue36721 closed by vstinner #36739: "4.6. Defining Functions" should mention nonlocal https://bugs.python.org/issue36739 closed by Mariatta #36772: Let lru_cache be used as a decorator with no arguments https://bugs.python.org/issue36772 closed by rhettinger #36782: Add tests for the datetime C API https://bugs.python.org/issue36782 closed by vstinner #36794: asyncio.Lock documentation in Py3.8 lacks parts presented in d https://bugs.python.org/issue36794 closed by asvetlov #36809: Crash for test test_importlib https://bugs.python.org/issue36809 closed by vstinner #36829: Add sys.unraisablehook() to custom how "unraisable exceptions" https://bugs.python.org/issue36829 closed by vstinner #36840: Add stream.abort() async method https://bugs.python.org/issue36840 closed by asvetlov #36842: Implement PEP 578 https://bugs.python.org/issue36842 closed by steve.dower #36843: AIX build fails with failure to get random numbers https://bugs.python.org/issue36843 closed by vstinner #36856: faulthandler._stack_overflow doesn't work on x86-linux with KP https://bugs.python.org/issue36856 closed by vstinner #36891: Additional startup plugin for vendors https://bugs.python.org/issue36891 closed by inada.naoki #36922: Implement Py_TPFLAGS_METHOD_DESCRIPTOR https://bugs.python.org/issue36922 closed by petr.viktorin #36933: sys.set_coroutine_wrapper documented as to be removed in 3.8 ( https://bugs.python.org/issue36933 closed by asvetlov #36969: pdb.do_args: display keyword-only and positional only argument https://bugs.python.org/issue36969 closed by pablogsal #36983: typing.__all__ has drifted from actual contents https://bugs.python.org/issue36983 closed by gvanrossum #36996: unittest.mock.patch decorator doesn't work with async function https://bugs.python.org/issue36996 closed by asvetlov #36999: Expose the coroutine object in asyncio task objects https://bugs.python.org/issue36999 closed by asvetlov #37001: symtable.symtable doesn't accept bytes which leads to a mismat https://bugs.python.org/issue37001 closed by dino.viehland #37003: ast unparse does not support f-string new debug format. https://bugs.python.org/issue37003 closed by mbussonn #37007: Implement socket.if_{nametoindex,indextoname} for Windows https://bugs.python.org/issue37007 closed by steve.dower #37015: Fix asyncio mock warnings https://bugs.python.org/issue37015 closed by asvetlov #37017: Use LOAD_METHOD optimization in CallMethod C API functions https://bugs.python.org/issue37017 closed by inada.naoki #37021: Can _random.getrandbits() be converted to METH_FASTCALL? https://bugs.python.org/issue37021 closed by pablogsal #37023: test_gdb failed on AMD64 Debian PGO 3.x https://bugs.python.org/issue37023 closed by steve.dower #37027: Return a safe proxy over a socket from get_extra_info('socket' https://bugs.python.org/issue37027 closed by asvetlov #37028: Implement asyncio repl https://bugs.python.org/issue37028 closed by yselivanov #37032: Add CodeType.replace() method https://bugs.python.org/issue37032 closed by vstinner #37035: Don't log OSError exceptions in asyncio transports https://bugs.python.org/issue37035 closed by asvetlov #37045: Implement PEP 591: add Final qualifier and @final decorator to https://bugs.python.org/issue37045 closed by levkivskyi #37046: Implement PEP 586: add Literal type constructor to typing https://bugs.python.org/issue37046 closed by levkivskyi #37047: Refactor AsyncMock setup logic in create_autospec https://bugs.python.org/issue37047 closed by yselivanov #37049: Implement PEP 589: add TypedDict to typing https://bugs.python.org/issue37049 closed by levkivskyi #37050: Remove expr_text from ast node FormattedValue https://bugs.python.org/issue37050 closed by eric.smith #37051: Glossary item "hashable" incorrect https://bugs.python.org/issue37051 closed by rhettinger #37053: Tools/parser/unparse.py does not handle u"bar" correctly https://bugs.python.org/issue37053 closed by pablogsal #37054: Ignored exceptions in test_memoryio https://bugs.python.org/issue37054 closed by vstinner #37056: test_tools is failing on several buildbots https://bugs.python.org/issue37056 closed by levkivskyi #37057: suspicious.py sphinx extension access non-existing attribute. https://bugs.python.org/issue37057 closed by mbussonn #37058: Implement PEP 544: add Protocol to typing https://bugs.python.org/issue37058 closed by levkivskyi #37059: spam https://bugs.python.org/issue37059 closed by Mariatta #37061: The strangest glitch I have ever seen - incorrect indenterror, https://bugs.python.org/issue37061 closed by eric.smith #37065: File and lineno is not reported for syntax error in f-string https://bugs.python.org/issue37065 closed by SilentGhost #37066: os.execl opening a new bash shell doesn't work if initfile/rcf https://bugs.python.org/issue37066 closed by xtreak #37067: os.execl doesn't allow for empty string in mac https://bugs.python.org/issue37067 closed by ned.deily #37070: Clean up f-string debug handling, including potential memory l https://bugs.python.org/issue37070 closed by eric.smith #37072: PyNode_Compile() crashes in Python 3.8. https://bugs.python.org/issue37072 closed by gvanrossum #37076: _thread.start_new_thread(): call sys.unraisablehook() to handl https://bugs.python.org/issue37076 closed by vstinner #37078: spam https://bugs.python.org/issue37078 closed by xtreak #37079: PEM cadata causes ssl.SSLError: nested asn1 error https://bugs.python.org/issue37079 closed by christian.heimes #37092: LoggerAdapter doesn't have fatal() like Logger https://bugs.python.org/issue37092 closed by vinay.sajip #37094: Provide an example for TestCase.skipTest in unittest doc https://bugs.python.org/issue37094 closed by zach.ware #37099: test_inspect generates DeprecationWarning https://bugs.python.org/issue37099 closed by xtreak #37101: Filterer.filter can be rewritten using built-ins just as effic https://bugs.python.org/issue37101 closed by dwvisser #37104: logging.Logger.disabled is not documented https://bugs.python.org/issue37104 closed by vinay.sajip #37106: python re.escape doesn't escape some special characters. https://bugs.python.org/issue37106 closed by eric.smith #37108: Positional-only arguments break super() https://bugs.python.org/issue37108 closed by pablogsal #37109: Inacurrate documentation regarding return type of math.factori https://bugs.python.org/issue37109 closed by Antony.Lee #37111: Logging - Inconsistent behaviour when handling unicode https://bugs.python.org/issue37111 closed by vinay.sajip #37112: Error in compilig the AST for functions with optional position https://bugs.python.org/issue37112 closed by pablogsal #37113: '??'.upper() should return '???' https://bugs.python.org/issue37113 closed by SilentGhost #37114: lstrip remove extra characters in the presence of a matching n https://bugs.python.org/issue37114 closed by christian.heimes #37115: Support annotations in positional-only arguments https://bugs.python.org/issue37115 closed by pablogsal #1230540: Add threading.excepthook() to handle uncaught exceptions raise https://bugs.python.org/issue1230540 closed by vstinner #1195571: simple callback system for Py_FatalError https://bugs.python.org/issue1195571 closed by vstinner From tim.peters at gmail.com Fri May 31 14:11:13 2019 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 31 May 2019 13:11:13 -0500 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: [Tim] >> I'm keen to get feedback on this before merging the PR, because this >> case is so very much larger than anything I've ever tried that I'm >> wary that there may be more than one "surprise" lurking here. ... [Inada Naoki ] > I started r5a.4xlarge EC2 instance and started arena.py. > I will post the result in next 12 hours. Thank you! Wrapping this up, Inada attached the results to the bug report. For the OP's original big case, the time to reclaim the tree storage dropped from nearly 5 hours to about 70 seconds. The time to build the tree didn't materially change (it was a bit faster after the patch, but offhand didn't look significantly faster to me). Since I called this a "design flaw" rather than "a bug", I'm not inclined to backport it. If someone else thinks it should be, that's up to them. I'm _assuming_ Github won't decide to backport it on its own - but maybe I'm wrong (Github workflow is still pretty magical to me). From brett at python.org Fri May 31 14:21:20 2019 From: brett at python.org (Brett Cannon) Date: Fri, 31 May 2019 11:21:20 -0700 Subject: [Python-Dev] PEP 595: Improving bugs.python.org In-Reply-To: <20190531102224.0bf2229f@fsol> References: <20190531102224.0bf2229f@fsol> Message-ID: On Fri, May 31, 2019 at 1:23 AM Antoine Pitrou wrote: > > I second this. > > There are currently ~7000 bugs open on bugs.python.org. The Web UI > makes a good job of actually being able to navigate through these bugs, > search through them, etc. > > Did the Steering Council conduct a usability study of Github Issues > with those ~7000 bugs open? What I can personally speak to is I work on https://github.com/microsoft/vscode-python for a living and we're at 928 issues. I also interact with and know the team behind https://github.com/microsoft/vscode at 5,439 issues. https://github.com/rust-lang/rust/ is as 4,806. https://github.com/golang/go is at 4,467. Whether it's 928 or 5,439, to me both numbers are well beyond being small enough to remember everything in the repo and having to rely on search, labels, etc. to find anything and it's worked out for me and my team at work. The VS Code team has also never told me that they were unhappy with GitHub for their issue tracker at their scale either (I've chatted to them about their automatic classification bot and they just gave an internal talk on how they manage the project). -Brett > If not, then I think the acceptance of > migrating to Github is a rushed job. Please reconsider. > > Regards > > Antoine. > > > > On Thu, 23 May 2019 22:17:24 +0200 > Ezio Melotti wrote: > > Hello, > > Berker and I have been working on a PEP that suggests we keep using > > and improving bugs.python.org and Roundup instead of switching to > > GitHub Issues as proposed by PEP 581. > > > > The PEP covers: > > * What are the advantages of Roundup over GitHub issues; > > * What features are missing in Roundup and how can we add them; > > * Issues with PEP 581; > > * Issues with the migration plan proposed by PEP 588; > > > > The rendered version of PEP 595 is available at > > https://www.python.org/dev/peps/pep-0595/ > > > > For reference, you can consult PEP 581 and 588 at > > https://www.python.org/dev/peps/pep-0581/ and > > https://www.python.org/dev/peps/pep-0588/ > > > > The full text of the PEP is include below. We are planning to update > > the PEP to include the feedback we receive and to update the status of > > features as we implement them (we also have a Google Summer of Code > > students working on it). > > > > Best Regards, > > Ezio Melotti > > > > > > ================ > > PEP: 595 > > Title: Improving bugs.python.org > > Author: Ezio Melotti , Berker Peksag > > > > Status: Draft > > Type: Process > > Content-Type: text/x-rst > > Created: 12-May-2019 > > > > > > Abstract > > ======== > > > > This PEP proposes a list of improvements to make bugs.python.org > > more usable for contributors and core developers. This PEP also > > discusses why remaining on Roundup should be preferred over > > switching to GitHub Issues, as proposed by :pep:`581`. > > > > > > Motivation > > ========== > > > > On May 14th, 2019 :pep:`581` has been accepted [#]_ without much > > public discussion and without a clear consensus [#]_. The PEP > > contains factual errors and doesn't address some of the > > issues that the migration to GitHub Issues might present. > > > > Given the scope of the migration, the amount of work required, > > and how it will negatively affect the workflow during the > > transition phase, this decision should be re-evaluated. > > > > > > Roundup advantages over GitHub Issues > > ===================================== > > > > This section discusses reasons why Roundup should be preferred > > over GitHub Issues and Roundup features that are not available > > on GitHub Issues. > > > > * **Roundup is the status quo.** Roundup has been an integral > > part of the CPython workflow for years. It is a stable product > > that has been tested and customized to adapt to our needs as the > > workflow evolved. > > > > It is possible to gradually improve it and avoid the disruption > > that a switch to a different system would inevitabily bring to > > the workflow. > > > > * **Open-source and Python powered.** Roundup is an open-source > > project and is written in Python. By using it and supporting > > it, we also support the Python ecosystem. Several features > > developed for bpo have also been ported to upstream Roundup > > over the years. > > > > * **Fully customizable.** Roundup can be (and has been) fully > > customized to fit our needs. > > > > * **Finer-grained access control.** Roundup allows the creation > > of different roles with different permissions (e.g. create, > > view, edit, etc.) for each individual property, and users can > > have multiple roles. > > > > * **Flexible UI.** While Roundup UI might look dated, it is > > convenient and flexible. > > > > For example, on the issue page, each field (e.g. title, type, > > versions, status, linked files and PRs, etc.) have appropriate > > UI elements (input boxes, dropdowns, tables, etc.) that are > > easy to set and also provide a convenient way to get info about > > the issue at a glance. The number of fields, their values, and > > the UI element they use is also fully customizable. > > GitHub only provides labels. > > > > The issue list page presents the issues in a compact and easy > > to read table with separate columns for different fields. For > > comparison, Roundup lists 50 issues in a screen, whereas GitHub > > takes two screens to shows 25 issues. > > > > * **Advanced search.** Roundup provides an accurate way to search > > and filter by using any combination of issue fields. > > It is also possible to customize the number of results and the > > fields displayed in the table, and the sorting and grouping > > (up to two levels). > > > > bpo also provides predefined summaries (e.g. "Created by you", > > "Assigned to you", etc.) and allows the creation of custom > > search queries that can be conveniently accessed from the sidebar. > > > > * **Nosy list autocomplete.** The nosy list has an autocomplete > > feature that suggests maintainers and experts. The suggestions > > are automatically updated when the experts index [#]_ changes. > > > > * **Dependencies and Superseders.** Roundup allows to specify > > dependencies that must be addressed before the current issues > > can be closed and a superseder issue to easily mark duplicates > > [#]_. The list of dependencies can also be used to create > > meta-issues that references several other sub-issues [#]_. > > > > > > Improving Roundup > > ================= > > > > This section lists some of the issues mentioned by :pep:`581` > > and other desired features and discusses how they can be implemented > > by improving Roundup and/or our instance. > > > > * **REST API support.** A REST API will make integration with other > > services and the development of new tools and applications easiers. > > > > Upstream Roundup now supports a REST API. Updating the tracker will > > make the REST API available. > > > > * **GitHub login support.** This will allow users to login > > to bugs.python.org (bpo) without having to create a new account. > > It will also solve issues with confirmation emails being marked > > as spam, and provide two-factor authentication. > > > > A patch to add this functionality is already available and is > > being integrated at the time of writing [#]_. > > > > * **Markdown support and message preview and editing.** This feature > > will allow the use of Markdown in messages and the ability to > > preview the message before the submission and edit it afterward. > > > > This can be done, but it will take some work. Possible solutions > > have been proposed on the roundup-devel mailing list [#]_. > > > > * **"Remove me from nosy list" button.** Add a button on issue pages > > to remove self from the nosy list. > > > > This feature will be added during GSoC 2019. > > > > * **Mobile friendly theme.** Current theme of bugs.python.org looks > > dated and it doesn't work well with mobile browsers. > > > > A mobile-friendly theme that is more modern but still familiar > > will be added. > > > > * **Add PR link to BPO emails.** Currently bpo emails don't include > > links to the corresponding PRs. > > > > A patch [#]_ is available to change the content of the bpo emails > > from:: > > > > components: +Tkinter > > versions: +Python 3.4 > > pull_requests: +42 > > > > to:: > > > > components: +Tkinter > > versions: +Python 3.4 > > pull_request: https://github.com/python/cpython/pull/341 > > > > * **Python 3 support.** Using Python 3 will make maintenance easier. > > > > Upstream Roundup now supports Python 3. Updating the tracker will > > allow us to switch to Python 3. The instances will need to be > > updated as well. > > > > * **Use upstream Roundup.** We currently use a fork of Roundup with > > a few modifications, most notably the GitHub integration. If this > > is ported upstream, we can start using upstream Roundup without > > having to maintain our fork. > > > > > > PEP 581 issues > > ============== > > > > This section addresses some errors and inaccuracies found in :pep:`581`. > > > > The "Why GitHub?" section of PEP 581 lists features currently > > available on GitHub Issues but not on Roundup. Some of this features > > are currently supported: > > > > * "Ability to reply to issue and pull request conversations via email." > > > > * Being able to reply by email has been one of the core features of > > Roundup since the beginning. It is also possible to create new > > issues or close existing ones, set or modify fields, and add > > attachments. > > > > * "Email notifications containing metadata, integrated with Gmail, > > allowing systematic filtering of emails." > > > > * Emails sent by Roundup contains metadata that can be used for > > filtering. > > > > * "Additional privacy, such as offering the user a choice to hide an > > email address, while still allowing communication with the user > > through @-mentions." > > > > * Email addresses are hidden by default to users that are not > > registered. Registered users can see other users' addresses > > because we configured the tracker to show them. It can easily > > be changed if desired. Users can still be added to the nosy > > list by using their username even if their address is hidden. > > > > * "Ability to automatically close issues when a PR has been merged." > > > > * The GitHub integration of Roundup automatically closes issues > > when a commit that contains "fixes issue " is merged. > > (Alternative spellings such as "closes" or "bug" are also supported.) > > See [#]_ for a recent example of this feature. > > > > * "Support for permalinks, allowing easy quoting and copying & > > pasting of source code." > > > > * Roundup has permalinks for issues, messages, attachments, etc. > > In addition, Roundup allows to easily rewrite broken URLs in > > messages (e.g. if the code hosting changes). > > > > * "Core developers, volunteers, and the PSF don't have to maintain the > > issue infrastructure/site, giving us more time and resources to focus > > on the development of Python." > > > > * While this is partially true, additional resources are required to > > write and maintain bots. > > > > In some cases, bots are required to workaround GitHub's lack of > > features rather than expanding. [#]_ was written > > specifically to workaround GitHub's email integration. > > > > Updating our bots to stay up-to-date with changes in the GitHub API > > has also maintenance cost. [#]_ took two days to be fixed. > > > > In addition, we will still need to maintain Roundup for bpo (even > > if it becomes read-only) and for the other trackers > > we currently host/maintain (Jython [#]_ and Roundup [#]_). > > > > The "Issues with Roundup / bpo" section of :pep:`581` lists some issues > > that have already been fixed: > > > > * "The upstream Roundup code is in Mercurial. Without any CI available, > > it puts heavy burden on the few existing maintainers in terms of > > reviewing, testing, and applying patches." > > > > * While Roundup uses Mercurial by default, there is a git clone > > available on GitHub [#]_. Roundup also has CI available [#]_ [#]_. > > > > * "There is no REST API available. There is an open issue in Roundup for > > adding REST API. Last activity was in 2016." > > > > * The REST API has been integrated and it's now available in Roundup. > > > > * "Users email addresses are exposed. There is no option to mask it." > > > > * Exposing addresses to registered and logged in users was a decision > > taken when our instance was set up. > > > > This has now been changed to make the email addresses hidden for > > regular users too (Developers and Coordinators can still see them). > > The "Email address"" column from the user listing page [#]_ has been > > removed too. > > > > * "It sends a number of unnecessary emails and notifications, and it is > > difficult, if not impossible, to configure." > > > > * This can be configured. > > > > * "Creating an account has been a hassle. There have been reports of > people > > having trouble creating accounts or logging in." > > > > * The main issue is confirmation emails being marked as spam. Work has > > been done to resolve the issue. > > > > Migration considerations > > ======================== > > > > This section describes issues with the migrations that might not > > have been addressed by :pep:`581` and :pep:`588`. > > > > :pep:`588` suggests to add a button to migrate issues to GitHub > > only when someone wants to keep working on them. This approach > > has several issues: > > > > * bpo will need to be updated in order to add a button that, > > once pressed, creates a new issue on GitHub, copies over all > > the messages, attachments, and creates/adds label for the > > existing fields. Permissions will also need to be tweaked > > to make individual issues read-only once they are migrated, > > and to prevent users to create new accounts. > > > > * The issues will be split between two trackers; searching issues > > will take significant more effort. > > > > * The conversion from Roundup to GitHub is lossy, unless all > > the bpo fields are converted into labels or preserved somewhere > > else. > > > > * bpo converts a number of references into links, including > > issue, message, and PR IDs, changeset numbers, legacy SVN > > revision numbers, paths to files in the repo, files in > > tracebacks (detecting the correct branch), links to devguide > > pages and sections [#]_. This happens when messages are > > requested so it is possible to create the correct link (e.g. > > all the file links used to point to hg.python.org and now > > point to GitHub). > > > > If the links are hardcoded during the migration, it will be > > difficult (if not impossible) to change them later. If they > > aren't, they will either be lost, or a tool to generate the > > links and updating them will need to be written. > > > > * GitHub doesn't provide a way to set and preserve issue IDs > > if they are migrated automatically with the use of a button. > > (Some projects managed to preserve the IDs by contaacting > > the GitHub staff and migrating the issues *en masse*.) > > > > * On top of the work and changes required to migrate to GitHub > > issues, we will still need to keep running and maintaining > > Roundup, for both our instance (read-only) and for the Jython > > and Roundup trackers (read-write). > > > > > > In addition to the issues listed in the "Open issues" section of > > :pep:`588`, this issues will need to be addressed: > > > > * GitHub is properietary and there is risk of vendor lock-in. > > Their business model might change and they could shut down > > altogether. > > > > * Switching to GitHub Issues will likely increase the number of > > invalid reports and increase the triaging effort. This concern > > has been raised in the past in a Zulip topic [#]_. > > > > There have been already cases where people posted comments on > > PRs that required moderators to mark them as off-topic or > > disruptive, delete them altogether, and even lock the > > conversation [#]_. > > > > * Roundup sends weekly reports to python-dev with a summary that > > includes new issues, recent issues with no replies, recent > > issues waiting for review, most discussed issues, closed issues, > > and deltas for open/closed/total issue counts [#]_. The report > > provides an easy way to keep track of the tracker activity and > > to make sure that issues that require attention are noticed. > > > > The data collect by the weekly report is also use to generate > > statistics and graphs that can be used to gain new insights [#]_. > > > > * There are currently two mailing lists where Roundup posts new > > tracker issues and all messages respectively: new-bugs-announce > > [#]_ and python-bugs-list [#]_. A new system will need to be > > developed to preserve this functionality. These MLs offer > > additional ways to keep track of the tracker activity. > > > > > > References > > ========== > > > > .. [#] [Python-Dev] PEP 581 (Using GitHub issues for CPython) is accepted > > > > https://mail.python.org/pipermail/python-dev/2019-May/157399.html > > > > .. [#] [python-committers] [Python-Dev] PEP 581 (Using GitHub issues > > for CPython) is accepted > > > > > https://mail.python.org/pipermail/python-committers/2019-May/006755.html > > > > .. [#] Experts Index -- Python Devguide > > > > https://devguide.python.org/experts/ > > > > .. [#] An example of superseded issues: > > "re.sub() replaces only several matches" > > > > https://bugs.python.org/issue12078 > > > > .. [#] An example of meta issue using dependencies to track sub-issues: > > "Meta-issue: support of the android platform"" > > > > https://bugs.python.org/issue26865 > > > > .. [#] Support logging in with GitHub > > > > https://github.com/python/bugs.python.org/issues/7 > > > > .. [#] Re: [Roundup-devel] PEP 581 and Google Summer of Code > > > > https://sourceforge.net/p/roundup/mailman/message/36667828/ > > > > .. [#] [Tracker-discuss] [issue624] bpo emails contain useless non-github > > pull_request number - users want a link to actual github PR > > > > > https://mail.python.org/pipermail/tracker-discuss/2018-June/004547.html > > > > .. [#] The commit reported in msg342882 closes the issue (see the > history below) > > > > https://bugs.python.org/issue36951#msg342882 > > > > .. [#] The cpython-emailer-webhook project > > > > https://github.com/berkerpeksag/cpython-emailer-webhook > > > > .. [#] A recent incident caused by GitHub > > > > https://github.com/python/bedevere/pull/163 > > > > .. [#] Jython issue tracker > > > > https://bugs.jython.org/ > > > > .. [#] Roundup issue tracker > > > > https://issues.roundup-tracker.org/ > > > > .. [#] GitHub clone of Roundup > > > > https://github.com/roundup-tracker/roundup > > > > .. [#] Travis-CI for Roundup > > > > https://travis-ci.org/roundup-tracker/roundup) and codecov > > > > .. [#] Codecov for Roundup > > > > https://codecov.io/gh/roundup-tracker/roundup/commits > > > > .. [#] User listing -- Python tracker > > > > https://bugs.python.org/user?@sort=username > > > > .. [#] Generating Special Links in a Comment -- Python Devguide > > > > > https://devguide.python.org/triaging/#generating-special-links-in-a-comment > > > > .. [#] The New-bugs-announce mailing list > > > > https://mail.python.org/mailman/listinfo/new-bugs-announce > > > > .. [#] The Python-bugs-list mailing list > > > > https://mail.python.org/mailman/listinfo/python-bugs-list > > > > .. [#] An example of [Python-Dev] Summary of Python tracker Issues > > > > https://mail.python.org/pipermail/python-dev/2019-May/157483.html > > > > .. [#] Issues stats -- Python tracker > > > > https://bugs.python.org/issue?@template=stats > > > > .. [#] s/n ratio -- Python -- Zulip > > > > > https://python.zulipchat.com/#narrow/stream/130206-pep581/topic/s.2Fn.20ratio > > > > .. [#] For example this and other related PRs: > > > > https://github.com/python/cpython/pull/9099 > > > > > > Copyright > > ========= > > > > This document has been placed in the public domain. > > > > .. > > Local Variables: > > mode: indented-text > > indent-tabs-mode: nil > > sentence-end-double-space: t > > fill-column: 70 > > coding: utf-8 > > End: > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Fri May 31 14:38:35 2019 From: barry at python.org (Barry Warsaw) Date: Fri, 31 May 2019 11:38:35 -0700 Subject: [Python-Dev] PEP 595: Improving bugs.python.org In-Reply-To: <20190531102224.0bf2229f@fsol> References: <20190531102224.0bf2229f@fsol> Message-ID: <43AB66F6-37CA-475F-BE78-E3D1FA726F43@python.org> On May 31, 2019, at 01:22, Antoine Pitrou wrote: > I second this. > > There are currently ~7000 bugs open on bugs.python.org. The Web UI > makes a good job of actually being able to navigate through these bugs, > search through them, etc. > > Did the Steering Council conduct a usability study of Github Issues > with those ~7000 bugs open? If not, then I think the acceptance of > migrating to Github is a rushed job. Please reconsider. Thanks for your feedback Antoine. This is a tricky issue, with many factors and tradeoffs to consider. I really appreciate Ezio and Berker working on PEP 595, so we can put all these issues on the table. I think one of the most important tradeoffs is balancing the needs of existing developers (those who actively triage bugs today), and future contributors. But this and other UX issues are difficult to compare on our actual data right now. I fully expect that just as with the switch to git, we?ll do lots of sample imports and prototyping to ensure that GitHub issues will actually work for us (given our unique requirements), and to help achieve the proper balance. It does us no good to switch if we just anger all the existing devs. IMHO, if the switch to GH doesn?t improve our workflow, then it definitely warrants a reevaluation. I think things will be better, but let?s prove it. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From njs at pobox.com Fri May 31 14:58:22 2019 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 31 May 2019 11:58:22 -0700 Subject: [Python-Dev] PEP 595: Improving bugs.python.org In-Reply-To: <43AB66F6-37CA-475F-BE78-E3D1FA726F43@python.org> References: <20190531102224.0bf2229f@fsol> <43AB66F6-37CA-475F-BE78-E3D1FA726F43@python.org> Message-ID: On Fri, May 31, 2019 at 11:39 AM Barry Warsaw wrote: > > On May 31, 2019, at 01:22, Antoine Pitrou wrote: > > > I second this. > > > > There are currently ~7000 bugs open on bugs.python.org. The Web UI > > makes a good job of actually being able to navigate through these bugs, > > search through them, etc. > > > > Did the Steering Council conduct a usability study of Github Issues > > with those ~7000 bugs open? If not, then I think the acceptance of > > migrating to Github is a rushed job. Please reconsider. > > Thanks for your feedback Antoine. > > This is a tricky issue, with many factors and tradeoffs to consider. I really appreciate Ezio and Berker working on PEP 595, so we can put all these issues on the table. > > I think one of the most important tradeoffs is balancing the needs of existing developers (those who actively triage bugs today), and future contributors. But this and other UX issues are difficult to compare on our actual data right now. I fully expect that just as with the switch to git, we?ll do lots of sample imports and prototyping to ensure that GitHub issues will actually work for us (given our unique requirements), and to help achieve the proper balance. It does us no good to switch if we just anger all the existing devs. > > IMHO, if the switch to GH doesn?t improve our workflow, then it definitely warrants a reevaluation. I think things will be better, but let?s prove it. Perhaps we should put an explicit step on the transition plan, after the prototyping, that's "gather feedback from prototypes, re-evaluate, make final go/no-go decision"? I assume we'll want to do that anyway, and having it formally written down might reassure people. It might also encourage more people to actually try out the prototypes if we make it very clear that they're going to be asked for feedback. -n -- Nathaniel J. Smith -- https://vorpus.org From willingc at gmail.com Fri May 31 15:39:41 2019 From: willingc at gmail.com (Carol Willing) Date: Fri, 31 May 2019 12:39:41 -0700 Subject: [Python-Dev] PEP 595: Improving bugs.python.org In-Reply-To: References: <20190531102224.0bf2229f@fsol> <43AB66F6-37CA-475F-BE78-E3D1FA726F43@python.org> Message-ID: Nathaniel Smith wrote: > > On Fri, May 31, 2019 at 11:39 AM Barry Warsaw wrote: >> >> >> On May 31, 2019, at 01:22, Antoine Pitrou wrote: >> >>> >>> I second this. >>> >>> There are currently ~7000 bugs open on bugs.python.org.? The Web UI >>> makes a good job of actually being able to navigate through these bugs, >>> search through them, etc. >>> >>> Did the Steering Council conduct a usability study of Github Issues >>> with those ~7000 bugs open?? If not, then I think the acceptance of >>> migrating to Github is a rushed job.? Please reconsider. >> >> > > >> >> Thanks for your feedback Antoine. >> >> This is a tricky issue, with many factors and tradeoffs to consider.? >> I really appreciate Ezio and Berker working on PEP 595, so we can put >> all these issues on the table. >> >> I think one of the most important tradeoffs is balancing the needs of >> existing developers (those who actively triage bugs today), and >> future contributors.? But this and other UX issues are difficult to >> compare on our actual data right now.? I fully expect that just as >> with the switch to git, we?ll do lots of sample imports and >> prototyping to ensure that GitHub issues will actually work for us >> (given our unique requirements), and to help achieve the proper >> balance.? It does us no good to switch if we just anger all the >> existing devs. >> >> IMHO, if the switch to GH doesn?t improve our workflow, then it >> definitely warrants a reevaluation.? I think things will be better, >> but let?s prove it. > > > Perhaps we should put an explicit step on the transition plan, after > the prototyping, that's "gather feedback from prototypes, re-evaluate, > make final go/no-go decision"? I assume we'll want to do that anyway, > and having it formally written down might reassure people. It might > also encourage more people to actually try out the prototypes if we > make it very clear that they're going to be asked for feedback. > > -n > As Barry mentioned, there are many considerations. - Will GitHub provide a similar experience as bpo? Can the features valued by existing developers be served in a suitable way by GitHub issues? - What opportunity costs exist for remaining on bpo? Planning, reporting, integration with services, and accessibility are some. - Chicken and egg question: Are there 7000 open issues because the existing workflow with bpo inhibits acting on open issues with patches? Antoine, as for large projects on GitHub with many issues, TypeScript is a reasonable proxy with close to 4000 open issues yet only 126 open PRs. There are some nice planning things that have been done in the repo https://github.com/microsoft/TypeScript/issues as well as good documentation around their process and workflow. Thanks to Ezio and Berker for raising points in PEP 595. - Carol -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri May 31 16:11:52 2019 From: brett at python.org (Brett Cannon) Date: Fri, 31 May 2019 13:11:52 -0700 Subject: [Python-Dev] Have a big machine and spare time? Here's a possible Python bug. In-Reply-To: References: Message-ID: On Fri, May 31, 2019 at 11:12 AM Tim Peters wrote: > [Tim] > >> I'm keen to get feedback on this before merging the PR, because this > >> case is so very much larger than anything I've ever tried that I'm > >> wary that there may be more than one "surprise" lurking here. ... > > [Inada Naoki ] > > I started r5a.4xlarge EC2 instance and started arena.py. > > I will post the result in next 12 hours. > > Thank you! Wrapping this up, Inada attached the results to the bug > report. For the OP's original big case, the time to reclaim the tree > storage dropped from nearly 5 hours to about 70 seconds. The time to > build the tree didn't materially change (it was a bit faster after the > patch, but offhand didn't look significantly faster to me). > > Since I called this a "design flaw" rather than "a bug", I'm not > inclined to backport it. If someone else thinks it should be, that's > up to them. I'm _assuming_ Github won't decide to backport it on its > own - but maybe I'm wrong (Github workflow is still pretty magical to > me). > Backports only happen if you add the appropriate labels to the PR itself (and if that wasn't clear from the devguide then please open an issue so we can fix that oversight). -------------- next part -------------- An HTML attachment was scrubbed... URL: From hodgestar+pythondev at gmail.com Fri May 31 17:28:42 2019 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Fri, 31 May 2019 23:28:42 +0200 Subject: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures In-Reply-To: <67fbae0d-0155-a21d-cd4b-6e16628debb6@gmail.com> References: <67fbae0d-0155-a21d-cd4b-6e16628debb6@gmail.com> Message-ID: As the maintainer of Genshi, one the libraries affected by the CodeType and similar changes, I thought I could add a users perspective to the discussion: Genshi is a templating engine that parses Python code from inside its templates. It supports Python 2.6+, 3.2+, pypy2 and pypy3. It parses Python using compile(...) and then walks the AST to, for example, translate variable lookups into template context lookups. Pretty much every major release of Python has broken Genshi's Python parsing in some way. While on the one hand this is a bit annoying, I also don't want Python to stop evolving where it makes sense. My requests to core developers are largely pragmatic: * Try not to change things unless there's a good reason to (i.e. it makes Python better). * Don't try declare that these things shouldn't be used (there is not much I can do about that now). * Do warn people that these things evolve with the language. * If changes do happen, try make them visible and give a clear description of what has changed. Also many thanks to the core developers who've submitted patches to update Genshi in the past -- that was awesome of you. The new CodeType.replace will remove some potential sources of breakages in the future, so thank you very much for adding that. Schiavo Simon -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Fri May 31 12:37:39 2019 From: steve.dower at python.org (Steve Dower) Date: Fri, 31 May 2019 09:37:39 -0700 Subject: [Python-Dev] PEP 590: vectorcall without tp_call In-Reply-To: References: <5CEE7A35.3000106@UGent.be> <4554973e-385a-fefa-18ff-ef1fe13d509c@gmail.com> <5CEE8AFB.7010808@UGent.be> Message-ID: <69c7f252-1a23-56fc-6b78-007dadc2fca9@python.org> On 29May2019 1311, Petr Viktorin wrote: > On 5/29/19 3:36 PM, Jeroen Demeyer wrote: >> On 2019-05-29 15:29, Petr Viktorin wrote: >>> That sounds like a good idea for PyType_FromSpec. >> >> I don't think we're planning to support vectorcall in PyType_FromSpec >> for now. That's maybe for 3.9 when vectorcall is no longer provisional. >> >>> For static types I either wouldn't bother at all, or only check in debug >>> builds and fail with Py_FatalError. >> >> So basically an assert(...)? > > Yes. That's the usual way to let C extension authors know they did > something wrong. It's the usual way to let core developers know they did something wrong :) I don't want to reignite the whole debug/release extension build debate again, but I'll point out that it's very common to develop extension modules against the release binaries on Windows, which means you won't see asserts coming from core (because they've been optimised out). So if this is something we're going to detect at runtime regardless of build, using Py_FatalError/abort or raising a SystemError is preferable. Otherwise we'll be forcing users to debug a segfault. Cheers, Steve From tjreedy at udel.edu Fri May 31 18:29:10 2019 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 31 May 2019 18:29:10 -0400 Subject: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures In-Reply-To: References: <67fbae0d-0155-a21d-cd4b-6e16628debb6@gmail.com> Message-ID: On 5/31/2019 5:28 PM, Simon Cross wrote: > As the maintainer of Genshi, one the libraries affected by the CodeType > and similar changes, I thought I could add a users perspective to the > discussion: > > Genshi is a templating engine that parses Python code from inside its > templates. It supports Python 2.6+, 3.2+, pypy2 and pypy3. It parses > Python using compile(...) and then walks the AST to, for example, > translate variable lookups into template context lookups. > > Pretty much every major release of Python has broken Genshi's Python > parsing in some way. While on the one hand this is a bit annoying, I > also don't want Python to stop evolving where it makes sense. > > My requests to core developers are largely pragmatic: > > * Try not to change things unless there's a good reason to (i.e. it > makes Python better). We try not to. There is already too much work. > * Don't try declare that these things shouldn't be used (there is not > much I can do about that now). That was removed from the PR before it was merged. > * Do warn people that these things evolve with the language. That was merged. > * If changes do happen, try make them visible and give a clear > description of what has changed. Another chunk was added to What's New. > Also many thanks to the core developers who've submitted patches to > update Genshi in the past -- that was awesome of you. > > The new CodeType.replace will remove some potential sources of breakages > in the future, so thank you very much for adding that. -- Terry Jan Reedy From hodgestar+pythondev at gmail.com Fri May 31 18:35:52 2019 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Sat, 1 Jun 2019 00:35:52 +0200 Subject: [Python-Dev] Expected stability of PyCode_New() and types.CodeType() signatures In-Reply-To: References: <67fbae0d-0155-a21d-cd4b-6e16628debb6@gmail.com> Message-ID: On Sat, Jun 1, 2019 at 12:29 AM Terry Reedy wrote: > We try not to. There is already too much work. > ... > That was removed from the PR before it was merged. > ... > That was merged. > ... > Another chunk was added to What's New. > Woot to all of these! :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From gregory.szorc at gmail.com Fri May 31 23:43:36 2019 From: gregory.szorc at gmail.com (Gregory Szorc) Date: Fri, 31 May 2019 20:43:36 -0700 Subject: [Python-Dev] RFC: PEP 587 "Python Initialization Configuration": 3rd version In-Reply-To: References: <7ddebc01-a067-4b52-48b7-e90347ed7cc7@gmail.com> <3a18be48-c63f-c70b-2f84-3a657a482df5@gmail.com> Message-ID: <5dfd10c9-815d-0ead-963b-66aa2af0aa96@gmail.com> On 5/20/2019 4:09 AM, Victor Stinner wrote: > Hi Gregory, > > IMHO your remarks are not directly related to the PEP 587 and can be > addressed in parallel. > >> It sounds like PyOxidizer and Hermetic Python are on the same page and >> we're working towards a more official solution. But I want to make sure >> by explicitly stating what PyOxidizer is doing. >> >> Essentially, to facilitate in-memory import, we need to register a >> custom sys.meta_path importer *before* any file-based imports are >> attempted. (...) >> I /think/ providing a 2-phase >> initialization that stops between _Py_InitializeCore() and >> _Py_InitializeMainInterpreter() would get the job done for PyOxidizer >> today. (...) > > Extract of PEP 587: "This extracts a subset of the API design from the > PEP 432 development and refactoring work that is now considered > sufficiently stable to make public (allowing 3rd party embedding > applications access to the same configuration APIs that the native > CPython CLI is now using)." > > We know that my PEP 587 is incomplete, but the work will continue in > Python 3.9 to support your use case. > > The PEP 587 introduces an experimental separation between "core" and > "main" initialization phases. PyConfig._init_main=0 stops at the > "core" phase, then you are free to run C and Python, > _Py_InitializeMain() finishes the Python initialization ("main" > phase). > > >>> In the "Isolate Python" section, I suggest to set the "isolated" >>> parameter to 1 which imply setting user_site_directory to 0. So >>> sys.path isn't modified afterwards. What you pass to PyConfig is what >>> you get in sys.path in this case. >> >> Regarding site.py, I agree it is problematic for embedding scenarios. >> Some features of site.py can be useful. Others aren't. It would be >> useful to have more granular control over which bits of site.run are >> run. My naive suggestion would be to add individual flags to control >> which functions site.py:main() runs. That way embedders can cherry-pick >> site.py features without having to manually import the module and call >> functions within. That feels much more robust for long-term maintainability. > > I agree that more work can be done on the site module. IMHO core > features which are needed by everybody should be done before calling > site. Maybe using a frozen "presite" module or whatever. I would be > interested to make possible to use Python for most cases without the > site module. > > >> Regarding Python calling exit(), this is problematic for embedding >> scenarios. > > I am working on that. I fixed dozens of functions. For example, > Py_RunMain() should not longer exit if there is an uncaught SystemExit > when calling PyErr_Print(). SystemExit is now handled separately > before calling PyErr_Print(). The work is not done, but it should be > way better than Python 3.6 and 3.7 state. > > >> This thread called attention to exit() during interpreter >> initialization. But it is also a problem elsewhere. For example, >> PyErr_PrintEx() will call Py_Exit() if the exception is a SystemExit. >> There's definitely room to improve the exception handling mechanism to >> give embedders better control when SystemExit is raised. As it stands, >> we need to check for SystemExit manually and reimplement >> _Py_HandleSystemExit() to emulate its behavior for e.g. exception value >> handling (fun fact: you can pass non-None, non-integer values to >> sys.exit/SystemExit). > > I don't know well these functions, maybe new functions are needed. It > can be done without/outside the PEP 587. > > >> Having a more efficient member lookup for BuiltinImporter and >> FrozenImporter might shave off a millisecond or two from startup. This >> would require some kind of derived data structure. (...) > > I don't think that the structures/API to define frozen/builtin modules > has to change. We can convert these lists into an hash table during > the initialization of the importlib module. > > I'm not saying that the current API is perfect, just that IMHO it can > be solved without the API. > > >> Unfortunately, as long as there is a global data structure that can be mutated any time >> (the API contract doesn't prohibit modifying these global arrays after >> initialization), you would need to check for "cache invalidation" on >> every lookup, undermining performance benefits. > > Do you really expect an application modifying these lists dynamically? At this time, not really. But the mark of good API design IMO is that its flexibility empowers new and novel ideas and ways of doing things. Bad APIs (including the use of global variables) inhibit flexibility and constrain creativity and advanced usage. For this particular item, I could see some potential uses in processes hosting multiple, independent interpreters. Maybe you want to give each interpreter its own set of modules. That's not something you see today because of the GIL and all the other global state in CPython. But with the GIL's days apparently being numbered, who knows what the future holds. I would highly encourage the official API surface to do away with globals completely and for the internals to only use globals in ways that impose minimal restrictions/caveats on usage/behavior. This PEP along with others are huge steps in the right direction.