From terri at toybox.ca Wed Feb 1 03:33:58 2017 From: terri at toybox.ca (Terri Oda) Date: Wed, 1 Feb 2017 00:33:58 -0800 Subject: [Python-Dev] Core Python Projects for GSoC 2017? Message-ID: <9a1f47aa-0153-2fe6-41de-c4827b8ccabc@toybox.ca> Hey all, it's Google Summer of Code time again! For those not familiar with it, Google Summer of Code is a mentoring program where Google provides stipends to pay students to work with selected open source organizations. Python has participated for many years now, and we're hoping to be selected to participate again. One of the things we have to do to apply is have a bunch of interesting ideas for students to work on and a bunch of mentors who are excited about working with students. The PSF page is http://python-gsoc.org/ That has contact information and answers to questions like "what does it take to be a mentor?" and please feel free to ask if there's more you want to know! A good project idea is one that is useful to the community and can be completed by a student with the help of mentors during the 3-month duration of the program. We want at least two mentors available per idea (so that it's not a big deal if someone's at a conference or sick) and at least one of the mentors (usually the primary one) should be able to review code and work with the community to get code merged upstream when it's ready. So... is anyone here interested in mentoring this year, and does anyone have any great ideas of things that we need and can support in Core Python? Brainstorming appreciated! (Although only ideas with actual mentoring support will make the final cut.) I've got a stub of a Core Python ideas page here: https://wiki.python.org/moin/SummerOfCode/2017/python-core (If you can't edit the wiki, I can add people's usernames to the editor lists, or I can add ideas that are emailed to me.) And our ideas page template that has a checklist of things students usually want to know is here: https://wiki.python.org/moin/SummerOfCode/OrgIdeasPageTemplate If you're interested in mentoring, email gsoc-admins at python.org so I can make sure you're on the list of people I ping with reminders. We need a good set of ideas by Feb 7 for our application, and then if we're accepted by Google, we can add a few other ideas up until Feb 28 or so. Terri From cory at lukasa.co.uk Wed Feb 1 05:02:25 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 1 Feb 2017 10:02:25 +0000 Subject: [Python-Dev] SSL certificates recommendations for downstream python packagers In-Reply-To: <781d78df-f5e5-6fba-136d-a2967bbbdf52@python.org> References: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk> <4cece125-61c5-ec85-7137-f425badb1bc6@python.org> <781d78df-f5e5-6fba-136d-a2967bbbdf52@python.org> Message-ID: <8FF8240C-EA4D-4F01-868F-28798E7235DB@lukasa.co.uk> > On 31 Jan 2017, at 18:26, Steve Dower wrote: > > In short, I want to allow Python code to set OpenSSL's certificate validation callback. Basically, given a raw certificate, return True/False based on whether it should be trusted. I then have separate code (yet to be published) implementing the callback on Windows by calling into the WinVerifyTrust API with the HTTPS provider, which (allegedly) behaves identically to browsers using the same API (again, allegedly - I have absolutely no evidence to support this other than some manual testing). For context here Steve, this is not quite what Chrome does (and I cannot stress enough that the Chrome approach is the best one I?ve seen, the folks working on it really do know what they?re doing). The reason here is a bit tricky, but essentially the validation callback is called incrementally for each step up the chain. This is not normally what a platform validation API actually wants: generally they want the entire cert chain the remote peer sent at once. Chrome, instead, essentially disables the OpenSSL cert validation entirely: they still require the certificate be presented, but override the verification callback to always say ?yeah that?s cool, no big deal?. They then take the complete cert chain provided by the remote peer and pass that to the platform validation code in one shot after the handshake is complete, but before they send/receive any data on the connection. This is still safe: so long as you don?t actually expose any data before you have validated the certificates you aren?t at risk. I have actually prototyped this approach for Requests/urllib3 in the past. I wrote a small Rust extension to call into the platform-native code, and then wrapped it in a CFFI library that exposed a single callable to validate a cert chain for a specific hostname (library is here: https://github.com/python-hyper/certitude ). This could then be called from urllib3 code that used PyOpenSSL using this patch here: https://github.com/shazow/urllib3/pull/802/files PLEASE DON?T ACTUALLY USE THIS CODE. I have not validated that certitude does entirely the right things with the platform APIs. This is just an example of a stripped-down version of what Chrome does, as a potential example of how to get something working for your Python use-case. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed Feb 1 09:20:49 2017 From: steve.dower at python.org (Steve Dower) Date: Wed, 1 Feb 2017 06:20:49 -0800 Subject: [Python-Dev] SSL certificates recommendations for downstreampython packagers In-Reply-To: <8FF8240C-EA4D-4F01-868F-28798E7235DB@lukasa.co.uk> References: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk> <4cece125-61c5-ec85-7137-f425badb1bc6@python.org> <781d78df-f5e5-6fba-136d-a2967bbbdf52@python.org> <8FF8240C-EA4D-4F01-868F-28798E7235DB@lukasa.co.uk> Message-ID: Sorry, I misspoke when I said "certificate validation callback", I meant the same callback Cory uses below (name escapes me now, but it's unfortunately similar to what I said). There are two callbacks in OpenSSL, one that allows you to verify each certificate in the chain individually, and one that requires you to validate the entire chain. I do indeed take the entire chain in one go and pass it to the OS API. Christian also didn't like that I was bypassing *all* of OpenSSL's certificate handling here, but maybe there's a way to make it reliable if Chrome has done it? Top-posted from my Windows Phone -----Original Message----- From: "Cory Benfield" Sent: ?2/?1/?2017 2:03 To: "Steve Dower" Cc: "Christian Heimes" ; "David Cournapeau" ; "python-dev" Subject: Re: [Python-Dev] SSL certificates recommendations for downstreampython packagers On 31 Jan 2017, at 18:26, Steve Dower wrote: In short, I want to allow Python code to set OpenSSL's certificate validation callback. Basically, given a raw certificate, return True/False based on whether it should be trusted. I then have separate code (yet to be published) implementing the callback on Windows by calling into the WinVerifyTrust API with the HTTPS provider, which (allegedly) behaves identically to browsers using the same API (again, allegedly - I have absolutely no evidence to support this other than some manual testing). For context here Steve, this is not quite what Chrome does (and I cannot stress enough that the Chrome approach is the best one I?ve seen, the folks working on it really do know what they?re doing). The reason here is a bit tricky, but essentially the validation callback is called incrementally for each step up the chain. This is not normally what a platform validation API actually wants: generally they want the entire cert chain the remote peer sent at once. Chrome, instead, essentially disables the OpenSSL cert validation entirely: they still require the certificate be presented, but override the verification callback to always say ?yeah that?s cool, no big deal?. They then take the complete cert chain provided by the remote peer and pass that to the platform validation code in one shot after the handshake is complete, but before they send/receive any data on the connection. This is still safe: so long as you don?t actually expose any data before you have validated the certificates you aren?t at risk. I have actually prototyped this approach for Requests/urllib3 in the past. I wrote a small Rust extension to call into the platform-native code, and then wrapped it in a CFFI library that exposed a single callable to validate a cert chain for a specific hostname (library is here: https://github.com/python-hyper/certitude). This could then be called from urllib3 code that used PyOpenSSL using this patch here: https://github.com/shazow/urllib3/pull/802/files PLEASE DON?T ACTUALLY USE THIS CODE. I have not validated that certitude does entirely the right things with the platform APIs. This is just an example of a stripped-down version of what Chrome does, as a potential example of how to get something working for your Python use-case. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Wed Feb 1 10:32:45 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Wed, 1 Feb 2017 15:32:45 +0000 Subject: [Python-Dev] SSL certificates recommendations for downstreampython packagers In-Reply-To: References: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk> <4cece125-61c5-ec85-7137-f425badb1bc6@python.org> <781d78df-f5e5-6fba-136d-a2967bbbdf52@python.org> <8FF8240C-EA4D-4F01-868F-28798E7235DB@lukasa.co.uk> Message-ID: > On 1 Feb 2017, at 14:20, Steve Dower wrote: > > Sorry, I misspoke when I said "certificate validation callback", I meant the same callback Cory uses below (name escapes me now, but it's unfortunately similar to what I said). There are two callbacks in OpenSSL, one that allows you to verify each certificate in the chain individually, and one that requires you to validate the entire chain. > > I do indeed take the entire chain in one go and pass it to the OS API. Christian also didn't like that I was bypassing *all* of OpenSSL's certificate handling here, but maybe there's a way to make it reliable if Chrome has done it? So, my understanding is that bypassing OpenSSL?s cert handling is basically fine. The risks are only in cases where OpenSSL?s cert handling would be a supplement to what the OS provides, which is not really very common and I don?t think is a major risk for Python. So in general, it is not unreasonable to ask your OS ?are these certificates valid for this connection based on your trust DB? and circumventing OpenSSL entirely there. Please do bear in mind you need to ask your OS the right question. For Windows this stuff is actually kinda hard because the API is somewhat opaque, but you have to worry about setting correct certificate usages, building up chain policies, and then doing appropriate error handling (AFAIK the crypto API can ?fail validation? for some reasons that have nothing to do with validation itself, so worth bearing that in mind). The TL;DR is: I understand Christian?s concern, but I don?t think it?s important if you?re very, very careful. Cory From levkivskyi at gmail.com Wed Feb 1 11:16:44 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Wed, 1 Feb 2017 17:16:44 +0100 Subject: [Python-Dev] Investigating Python memory footprint of one real Web application In-Reply-To: References: Message-ID: Inada-san, I have made a PR for typing module upstream https://github.com/python/typing/pull/383 It should reduce the memory consumption significantly (and also increase isinstance() speed). Could you please try it with your real code base and test memory consumption (and maybe speed) as compared to master? -- Ivan On 23 January 2017 at 12:25, INADA Naoki wrote: > On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi > wrote: > > On 20 January 2017 at 11:49, INADA Naoki wrote: > >> > >> * typing may increase memory footprint, through functions > >> __attributes__ and abc. > >> * Can we add option to remove or lazy evaluate __attributes__ ? > > > > > > This idea already appeared few times. I proposed to introduce a flag > (e.g. > > -OOO) to ignore function and variable annotations in compile.c > > It was decide to postpone this, but maybe we can get back to this idea. > > > > In 3.6, typing is already (quite heavily) optimized for both speed and > > space. > > I remember doing an experiment comparing a memory footprint with and > without > > annotations, the difference was few percent. > > Do you have such comparison (with and without annotations) for your app? > > It would be nice to have a realistic number to estimate what would the > > additional optimization flag save. > > > > -- > > Ivan > > > > > > Hi, Ivan. > > I investigated why our app has so many WeakSet today. > > We have dozen or hundreds of annotations like Iterable[User] or List[User]. > (User is one example of application's domain object. There are > hundreds of classes). > > On the other hand, SQLAlchemy calls isinstance(obj, > collections.Iterable) many times, > in [sqlalchemy.util._collections.to_list](https://github.com/ > zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_ > collections.py#L795-L804) > method. > > So there are (# of iterable subclasses) weaksets for negative cache, > and each weakset > contains (# of column types) entries. That's why WeakSet ate much RAM. > > It may be slowdown application startup too, because thousands of > __subclasscheck_ is called. > > I gave advice to use 'List[User]' instead of List[User] to the team of > the project, > if the team think RAM usage or boot speed is important. > > FWIW, stacktrace is like this: > > File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line > 84 > self.data.add(ref(item, self._remove)) > File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233 > cls._abc_negative_cache.add(subclass) > File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226 > if issubclass(subclass, scls): > File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226 > if issubclass(subclass, scls): > File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191 > return cls.__subclasscheck__(subclass) > File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py", > line 803 > or not isinstance(x, collections.Iterable): > File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line > 1680 > columns = util.to_list(prop) > File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line > 1575 > prop = self._property_from_column(key, prop) > File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line > 1371 > setparent=True) > File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line > 675 > self._configure_properties() > > Regards, > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eeppeliteloop at gmail.com Wed Feb 1 03:23:02 2017 From: eeppeliteloop at gmail.com (Philippe Proulx) Date: Wed, 1 Feb 2017 03:23:02 -0500 Subject: [Python-Dev] Why doesn't module finalization delete names as expected? Message-ID: Hello. I'm not sure if I'm posting to the right list. If it's not the case, please tell me which one to post to. Using Python 3.5.2. I'm developing a C module with the help of SWIG. My library manages objects with reference counting, much like Python, except that it's deterministic: there's no GC. I create two Python objects like this: bakery = Bakery() bread = bakery.create_bread() Behind the scenes, the situation looks like this: +--------------------+ | UserBread obj (Py) | +----------^---+-----+ | : | : | : +------------------+ +---------+---V---+ | Bakery obj (lib) <----------------+ Bread obj (lib) | +--------^---+-----+ +--------^--------+ | : | | : | +--------+---V----+ +--------+-------+ | Bakery obj (Py) | | Bread obj (Py) | +---------^-------+ +------^---------+ | | | | + + bakery bread A pipe link means one "strong" reference and a colon link means one borrowed/weak reference. I have some ownership inversion magic for the Bakery lib. and Python objects to always coexist. So here it's pretty clear what can happen. I don't know which reference gets deleted first, but let's assume it's `bakery`. Then the situation looks like this: +--------------------+ | UserBread obj (Py) | +----------^---+-----+ | : | : | : +------------------+ +---------+---V---+ | Bakery obj (lib) <----------------+ Bread obj (lib) | +--------^---+-----+ +--------^--------+ : | | : | | +--------+---V----+ +--------+-------+ | Bakery obj (Py) | | Bread obj (Py) | +-----------------+ +------^---------+ | | + bread The Bakery Python object's __del__() drops the reference to its library object, but first its reference count is incremented during this call (so it's not really destroyed) and it's marked as only owned by the library object from now on. When `bread` gets deleted: 1. The Bread Python object's __del__() method gets called: the reference to its library object is dropped. 2. The Bread library object's destroy function drops its reference to the Bakery library object. 3. The Bakery library object's destroy function drops its reference to the Bakery Python object. 4. The Bakery Python object's __del__() method does nothing this time, since the object is marked as only owned by its library object (inverted ownership). 5. The Bread library object's destroy function then drops its reference to the UserBread Python object. In the end, everything is correctly destroyed and released. This also works if `bread` is deleted before `bakery`. My problem is that this works as expected when used like this: def func(): bakery = Bakery() bread = bakery.create_bread() if __name__ == '__main__': func() but NOTHING is destroyed when used like this: bakery = Bakery() bread = bakery.create_bread() That is, directly during the module's initialization. It works, however, if I delete `bread` manually: bakery = Bakery() bread = bakery.create_bread() del bread It also works with `bakery` only: bakery = Bakery() My question is: what could explain this? My guess is that my logic is correct since it works fine in the function call situation. It feels like `bread` is never deleted in the module initialization situation, but I don't know why: the only reference to the Bread Python object is this `bread` name in the module... what could prevent this object's __del__() method to be called? It works when I call `del bread` manually: I would expect that the module finalization does the exact same thing? Am I missing anything? Thanks, Phil -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Feb 1 11:38:49 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 1 Feb 2017 17:38:49 +0100 Subject: [Python-Dev] Why doesn't module finalization delete names as expected? References: Message-ID: <20170201173849.5142f9b4@fsol> On Wed, 1 Feb 2017 03:23:02 -0500 Philippe Proulx wrote: > > It feels like `bread` is never deleted in the module initialization > situation, but I don't know why: the only reference to the Bread Python > object is this `bread` name in the module... what could prevent this > object's __del__() method to be called? It works when I call `del bread` > manually: I would expect that the module finalization does the exact > same thing? When do you expect module finalization to happen? Your module is recorded in sys.modules so, unless you explicitly remove it from there, module finalization will only happen at interpreter shutdown. (and this question is more appropriate for the python-list, anyway) Regards Antoine. From victor.stinner at gmail.com Wed Feb 1 12:32:16 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 1 Feb 2017 18:32:16 +0100 Subject: [Python-Dev] Heads up: possible double-comments on bpo for commits In-Reply-To: References: Message-ID: Hi, I noticed a strange issue with Roundup Robot on the issue #29318. I closed http://bugs.python.org/issue29318 as rejected: resolution: -> rejected status: open -> closed Roundup Robot made a first change: stage: -> resolved But then it made a second change: resolution: rejected -> fixed To finish with an empty change: => I got an email to notify me "Changes by Roundup Robot ", but with no useful email body. --- 8< --- Changes by Roundup Robot : _______________________________________ Python tracker --- 8< --- Victor From lukasz at langa.pl Wed Feb 1 14:42:56 2017 From: lukasz at langa.pl (Lukasz Langa) Date: Wed, 1 Feb 2017 11:42:56 -0800 Subject: [Python-Dev] re performance In-Reply-To: <371EBC7881C7844EAAF5556BFF21BCCC42EFF68E@ORSMSX105.amr.corp.intel.com> References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de> <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl> <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com> <20170128120705.3696f70c@subdivisions.wooz.org> <20170130092637.4665eaea@presto.wooz.org> <371EBC7881C7844EAAF5556BFF21BCCC42EFF68E@ORSMSX105.amr.corp.intel.com> Message-ID: <0CE3F76A-46D3-43F2-B3AB-8C8C18FF4FEB@langa.pl> > On Jan 31, 2017, at 11:40 AM, Wang, Peter Xihong wrote: > > Regarding to the performance difference between "re" and "regex" and packaging related options, we did a performance comparison using Python 3.6.0 to run some micro-benchmarks in the Python Benchmark Suite (https://github.com/python/performance ): > > Results in ms, and the lower the better (running on Ubuntu 15.10) > re regex (via pip install regex, and a replacement of "import re" with "import regex as re") > bm_regex_compile.py 229 298 > bm_regex_dna.py 171 267 > bm_regex_effbot.py 2.77 3.04 > bm_regex_v8.py 24.8 14.1 > This data shows "re" is better than "regex" in term of performance in 3 out of 4 above micro-benchmarks. This is very informative, thank you! This clearly shows we should rather pursue the PyPI route (with a documentation endorsement and possible bundling for 3.7) than full-blown replacement. However, this benchmark is incomplete in the sense that it only checks the compatibility mode of `regex`, whereas it's the new mode that lends the biggest performance gains. So, providing checks for the other engine would show us the full picture. We'd need to add checks that prove the regular expressions in said benchmarks end up with equivalent matches, to be sure we're testing the same thing. - ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Wed Feb 1 15:29:21 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 1 Feb 2017 22:29:21 +0200 Subject: [Python-Dev] re performance In-Reply-To: <371EBC7881C7844EAAF5556BFF21BCCC42EFF68E@ORSMSX105.amr.corp.intel.com> References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de> <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl> <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com> <20170128120705.3696f70c@subdivisions.wooz.org> <20170130092637.4665eaea@presto.wooz.org> <371EBC7881C7844EAAF5556BFF21BCCC42EFF68E@ORSMSX105.amr.corp.intel.com> Message-ID: On 31.01.17 21:40, Wang, Peter Xihong wrote: > Regarding to the performance difference between "re" and "regex" and packaging related options, we did a performance comparison using Python 3.6.0 to run some micro-benchmarks in the Python Benchmark Suite (https://github.com/python/performance): > > Results in ms, and the lower the better (running on Ubuntu 15.10) > re regex (via pip install regex, and a replacement of "import re" with "import regex as re") > bm_regex_compile.py 229 298 > bm_regex_dna.py 171 267 > bm_regex_effbot.py 2.77 3.04 > bm_regex_v8.py 24.8 14.1 > This data shows "re" is better than "regex" in term of performance in 3 out of 4 above micro-benchmarks. bm_regex_v8 is the one that is purposed to reflect real-world use of regular expressions. See also different comparison at https://mail.python.org/pipermail/speed/2016-March/000311.html. In some tests regex surpasses re, in other tests re surpasses regex. re2 is much faster than other engines in all tests except the one in which it is much slower (and this engine is the least featured). From victor.stinner at gmail.com Wed Feb 1 15:58:07 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 1 Feb 2017 21:58:07 +0100 Subject: [Python-Dev] re performance In-Reply-To: <0CE3F76A-46D3-43F2-B3AB-8C8C18FF4FEB@langa.pl> References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de> <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl> <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com> <20170128120705.3696f70c@subdivisions.wooz.org> <20170130092637.4665eaea@presto.wooz.org> <371EBC7881C7844EAAF5556BFF21BCCC42EFF68E@ORSMSX105.amr.corp.intel.com> <0CE3F76A-46D3-43F2-B3AB-8C8C18FF4FEB@langa.pl> Message-ID: 2017-02-01 20:42 GMT+01:00 Lukasz Langa : > However, this benchmark is incomplete in the sense that it only checks the > compatibility mode of `regex`, whereas it's the new mode that lends the > biggest performance gains. So, providing checks for the other engine would > show us the full picture. Would you mind to write a pull request for performance to add a command line option to test "re" (stdlib) or "regex" (PyPI, in the new mode)? Or maybe even regex and regex_compatibility_mode :-) Source: https://github.com/python/performance/blob/master/performance/benchmarks/bm_regex_v8.py#L1789 Example of benchmark with cmdline options: https://github.com/python/performance/blob/master/performance/benchmarks/bm_raytrace.py#L385 Victor From peter.xihong.wang at intel.com Wed Feb 1 17:59:14 2017 From: peter.xihong.wang at intel.com (Wang, Peter Xihong) Date: Wed, 1 Feb 2017 22:59:14 +0000 Subject: [Python-Dev] re performance In-Reply-To: References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de> <11439969-DBE6-4031-9EC6-3B2452340942@langa.pl> <65cf9e6e-7962-d198-6aa7-4302818a8819@mrabarnett.plus.com> <20170128120705.3696f70c@subdivisions.wooz.org> <20170130092637.4665eaea@presto.wooz.org> <371EBC7881C7844EAAF5556BFF21BCCC42EFF68E@ORSMSX105.amr.corp.intel.com> <0CE3F76A-46D3-43F2-B3AB-8C8C18FF4FEB@langa.pl> Message-ID: <371EBC7881C7844EAAF5556BFF21BCCC42F00872@ORSMSX105.amr.corp.intel.com> +1 We'd like to get more details on how to try this "new mode", and do a full/comprehensive comparison between the "re" vs "regex". Peter ? -----Original Message----- From: Victor Stinner [mailto:victor.stinner at gmail.com] Sent: Wednesday, February 01, 2017 12:58 PM To: Lukasz Langa Cc: Wang, Peter Xihong ; python-dev at python.org Subject: Re: [Python-Dev] re performance 2017-02-01 20:42 GMT+01:00 Lukasz Langa : > However, this benchmark is incomplete in the sense that it only checks > the compatibility mode of `regex`, whereas it's the new mode that > lends the biggest performance gains. So, providing checks for the > other engine would show us the full picture. Would you mind to write a pull request for performance to add a command line option to test "re" (stdlib) or "regex" (PyPI, in the new mode)? Or maybe even regex and regex_compatibility_mode :-) Source: https://github.com/python/performance/blob/master/performance/benchmarks/bm_regex_v8.py#L1789 Example of benchmark with cmdline options: https://github.com/python/performance/blob/master/performance/benchmarks/bm_raytrace.py#L385 Victor From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Feb 1 22:38:23 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 2 Feb 2017 12:38:23 +0900 Subject: [Python-Dev] SSL certificates recommendations for downstreampython packagers In-Reply-To: References: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk> <4cece125-61c5-ec85-7137-f425badb1bc6@python.org> <781d78df-f5e5-6fba-136d-a2967bbbdf52@python.org> <8FF8240C-EA4D-4F01-868F-28798E7235DB@lukasa.co.uk> Message-ID: <22674.43439.871545.345238@turnbull.sk.tsukuba.ac.jp> Cory Benfield writes: > The TL;DR is: I understand Christian?s concern, but I don?t think > it?s important if you?re very, very careful. But AIUI, the "you" above is the end-user or admin of end-user's system, no? We know that they aren't very careful (or perhaps more accurate, this is too fsckin' complicated for anybody but an infosec expert to do very well). I[1] still agree with you that it's *unlikely* that end-users/admins will need to worry about it. But we need to be really careful about what we say here, or at least where the responsible parties will be looking. Thanks to all who are contributing so much time and skull sweat on this. This is insanely hard, but important. Footnotes: [1] Infosec wannabe, I've thought carefully but don't claim real expertise. From leewangzhong+python at gmail.com Wed Feb 1 23:37:16 2017 From: leewangzhong+python at gmail.com (Franklin? Lee) Date: Wed, 1 Feb 2017 23:37:16 -0500 Subject: [Python-Dev] re performance In-Reply-To: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de> References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de> Message-ID: On Thu, Jan 26, 2017 at 4:13 PM, Sven R. Kunze wrote: > Hi folks, > > I recently refreshed regular expressions theoretical basics *indulging in > reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html > > However, reaching the chart in the lower third of the article, I saw Python > 2.4 measured against a naive Thompson matching implementation. And I was > surprised about how bad it performed compared to an unoptimized version of > an older than dirt algorithm. > From my perspective, I can say, that regular expressions might worth > optimizing especially for web applications (url matching usually uses > regexes) but also for other applications where I've seen many tight loops > using regexes as well. So, I am probing interest on this topic here. What I (think I) know: - Both re and regex use the same C backend, which is not based on NFA. - The re2 library, which the writer of that article made, allows capture groups (but only up to a limit) and bounded repetitions (up to a limit). - Perl has started to optimize some regex patterns. What I think: - The example is uncharacteristic of what people write, like URL matching. - But enabling naive code to perform well is usually a good thing. The fewer details newbies need to know to write code, the better. - It's possible for Python to optimize a large category of patterns, even if not all. - It's also possible to optimize large parts of patterns with backrefs. E.g. If there's a backref, the group that the backref refers to can still be made into an NFA. - To do the above, you'd need a way to generate all possible matches. - Optimization can be costly. The full NFA construction could be generated only upon request, or maybe the code automatically tries to optimize after 100 uses (like a JIT). This should only be considered if re2's construction really is costly. If people want NFAs, I think the "easiest" way is to use re2. Jakub Wilk mentioned it before, but here it is again. https://github.com/google/re2 re2 features: https://github.com/google/re2/wiki/Syntax Named references aren't supported, but I think that can be worked around with some renumbering. It's just a matter of translating the pattern. It also doesn't like lookarounds, which AFAIK are perfectly doable with NFAs. Looks like lookaheads and lookbehinds are hard to do without a lot of extra space (https://github.com/google/re2/issues/5). Facebook has a Python wrapper for re2. https://github.com/facebook/pyre2/ In a post linked to from this thread, Serhiy mentioned another Python wrapper for re2. This wrapper is designed to be like re, and should probably be the basis of any efforts. It's not been updated for 11 months, though. https://pypi.python.org/pypi/re2/ https://github.com/axiak/pyre2/ From songofacandy at gmail.com Thu Feb 2 05:07:25 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 2 Feb 2017 19:07:25 +0900 Subject: [Python-Dev] Investigating Python memory footprint of one real Web application In-Reply-To: References: Message-ID: Thanks, Ivan. I confirmed 3000 negative cache entries were removed by your patch! https://gist.github.com/methane/3c34f11fb677365a7e92afe73aca24e7 On Thu, Feb 2, 2017 at 1:16 AM, Ivan Levkivskyi wrote: > Inada-san, > > I have made a PR for typing module upstream > https://github.com/python/typing/pull/383 > It should reduce the memory consumption significantly (and also increase > isinstance() speed). > Could you please try it with your real code base and test memory consumption > (and maybe speed) as compared to master? > > -- > Ivan > > > On 23 January 2017 at 12:25, INADA Naoki wrote: >> >> On Fri, Jan 20, 2017 at 8:52 PM, Ivan Levkivskyi >> wrote: >> > On 20 January 2017 at 11:49, INADA Naoki wrote: >> >> >> >> * typing may increase memory footprint, through functions >> >> __attributes__ and abc. >> >> * Can we add option to remove or lazy evaluate __attributes__ ? >> > >> > >> > This idea already appeared few times. I proposed to introduce a flag >> > (e.g. >> > -OOO) to ignore function and variable annotations in compile.c >> > It was decide to postpone this, but maybe we can get back to this idea. >> > >> > In 3.6, typing is already (quite heavily) optimized for both speed and >> > space. >> > I remember doing an experiment comparing a memory footprint with and >> > without >> > annotations, the difference was few percent. >> > Do you have such comparison (with and without annotations) for your app? >> > It would be nice to have a realistic number to estimate what would the >> > additional optimization flag save. >> > >> > -- >> > Ivan >> > >> > >> >> Hi, Ivan. >> >> I investigated why our app has so many WeakSet today. >> >> We have dozen or hundreds of annotations like Iterable[User] or >> List[User]. >> (User is one example of application's domain object. There are >> hundreds of classes). >> >> On the other hand, SQLAlchemy calls isinstance(obj, >> collections.Iterable) many times, >> in >> [sqlalchemy.util._collections.to_list](https://github.com/zzzeek/sqlalchemy/blob/master/lib/sqlalchemy/util/_collections.py#L795-L804) >> method. >> >> So there are (# of iterable subclasses) weaksets for negative cache, >> and each weakset >> contains (# of column types) entries. That's why WeakSet ate much RAM. >> >> It may be slowdown application startup too, because thousands of >> __subclasscheck_ is called. >> >> I gave advice to use 'List[User]' instead of List[User] to the team of >> the project, >> if the team think RAM usage or boot speed is important. >> >> FWIW, stacktrace is like this: >> >> File "/Users/inada-n/local/py37dbg/lib/python3.7/_weakrefset.py", line >> 84 >> self.data.add(ref(item, self._remove)) >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 233 >> cls._abc_negative_cache.add(subclass) >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226 >> if issubclass(subclass, scls): >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 226 >> if issubclass(subclass, scls): >> File "/Users/inada-n/local/py37dbg/lib/python3.7/abc.py", line 191 >> return cls.__subclasscheck__(subclass) >> File "venv/lib/python3.7/site-packages/sqlalchemy/util/_collections.py", >> line 803 >> or not isinstance(x, collections.Iterable): >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1680 >> columns = util.to_list(prop) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1575 >> prop = self._property_from_column(key, prop) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 1371 >> setparent=True) >> File "venv/lib/python3.7/site-packages/sqlalchemy/orm/mapper.py", line >> 675 >> self._configure_properties() >> >> Regards, > > From victor.stinner at gmail.com Thu Feb 2 05:08:14 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 2 Feb 2017 11:08:14 +0100 Subject: [Python-Dev] Heads up: possible double-comments on bpo for commits In-Reply-To: References: Message-ID: Oh, I noticed another strange thing: The Roundup Robot closed the issue http://bugs.python.org/issue29368 but I don't see any explicit "Close issue #xxx" or "Close #xxx" in the commit message of the two commits. Is it deliberate to close an issue after any commit? Victor 2017-01-31 19:18 GMT+01:00 Brett Cannon : > I've activated the webhook for receiving comments on issues when a commit > lands mentioning an issue, so if you see a commit from our hg integration > and another from GitHub, understand that's why (mention issues as "bpo NNNN" > in commit messages if you want to see it in action). If it becomes too much > of a hassle to see the duplicates before we migrate I can turn off the > notifications, but obviously more testing the better. :) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From cory at lukasa.co.uk Thu Feb 2 05:16:19 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Thu, 2 Feb 2017 10:16:19 +0000 Subject: [Python-Dev] SSL certificates recommendations for downstreampython packagers In-Reply-To: <22674.43439.871545.345238@turnbull.sk.tsukuba.ac.jp> References: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk> <4cece125-61c5-ec85-7137-f425badb1bc6@python.org> <781d78df-f5e5-6fba-136d-a2967bbbdf52@python.org> <8FF8240C-EA4D-4F01-868F-28798E7235DB@lukasa.co.uk> <22674.43439.871545.345238@turnbull.sk.tsukuba.ac.jp> Message-ID: <80524675-7B66-4742-B367-EEA10F37D1CA@lukasa.co.uk> > On 2 Feb 2017, at 03:38, Stephen J. Turnbull wrote: > > Cory Benfield writes: > >> The TL;DR is: I understand Christian?s concern, but I don?t think >> it?s important if you?re very, very careful. > > But AIUI, the "you" above is the end-user or admin of end-user's > system, no? We know that they aren't very careful (or perhaps more > accurate, this is too fsckin' complicated for anybody but an infosec > expert to do very well). I think "you" is the coder of the interface. From a security perspective I think we have to discount the possibility of administrator error from our threat model. A threat model that includes ?defend the system against intrusions that the administrator incorrectly allows? is an insanely difficult one to respond to, given that it basically requires psychic powers to determine what the administrator *meant* instead of what they configured. Now, where we allow configuration we have a duty to ensure that it?s as easy as possible to configure correctly, but when using the system trust store most of the configuration is actually provided by the OS tools, rather than by the above-mentioned ?you?, so that?s not in our control. The risk, and the need to be very very careful, comes from ensuring that the semantics of the OS configuration are preserved through to the behaviour of the program. This is definitely a place with razor-blades all around, which is why I have tended to defer to the Chrome security team on this issue. In particular, the BoringSSL developers are razor-sharp people who have their heads screwed on when it comes to practical security decisions, and I?ve found that emulating them is usually a safe bet in the face of ambiguity. However, it?s unquestionable that the *safest* route to go down in terms of preserving the expectations of users is to use the platform-native TLS implementation wholesale, rather than do a hybrid model like Chrome does where OpenSSL does the protocol bits and the system does the X509 bits. That way Python ends up behaving basically like Edge or Safari on the relevant platforms, or perhaps more importantly behaving like .NET on Windows and like CoreFoundation on macOS, which is a much better place to be in terms of user and administrator expectations. As a side benefit, that approach helps take Python a bit closer to feeling ?platform-native? on many platforms, which can only be a good thing for those of us who want to see more Python on the desktop (or indeed on the mobile device). > I[1] still agree with you that it's *unlikely* that end-users/admins > will need to worry about it. But we need to be really careful about > what we say here, or at least where the responsible parties will be > looking. I agree. In an ideal world I?d say to Steve that he should shelve his current work and wait for the TLS ABC PEP that is incoming (hopefully I?ll send a first-draft to python-dev today). However, I?m nothing if not pragmatic, and having Steve continue his current work in parallel to the TLS ABC PEP is probably a good idea so that we can avoid having all our eggs in one basket. Perhaps we can get the TLS ABC stuff in place in time for Steve to just swap over to using SChannel altogether, but if that doesn?t work out and Steve can get a halfway-house out the door earlier then that?s fine by me. Cory From victor.stinner at gmail.com Thu Feb 2 12:01:58 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 2 Feb 2017 18:01:58 +0100 Subject: [Python-Dev] Heads up: possible double-comments on bpo for commits In-Reply-To: References: Message-ID: 2017-02-01 18:32 GMT+01:00 Victor Stinner : > To finish with an empty change: Information on IRC: Known issue: http://psf.upfronthosting.co.za/roundup/meta/issue611 "Maciej is working on it" Good & thanks! Victor From python at mrabarnett.plus.com Thu Feb 2 13:39:11 2017 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 2 Feb 2017 18:39:11 +0000 Subject: [Python-Dev] re performance In-Reply-To: References: <9133389a-9b24-562b-89d3-02dde355af5f@mail.de> Message-ID: On 2017-02-02 04:37, Franklin? Lee wrote: > On Thu, Jan 26, 2017 at 4:13 PM, Sven R. Kunze wrote: >> Hi folks, >> >> I recently refreshed regular expressions theoretical basics *indulging in >> reminiscences* So, I read https://swtch.com/~rsc/regexp/regexp1.html >> >> However, reaching the chart in the lower third of the article, I saw Python >> 2.4 measured against a naive Thompson matching implementation. And I was >> surprised about how bad it performed compared to an unoptimized version of >> an older than dirt algorithm. > >> From my perspective, I can say, that regular expressions might worth >> optimizing especially for web applications (url matching usually uses >> regexes) but also for other applications where I've seen many tight loops >> using regexes as well. So, I am probing interest on this topic here. > > What I (think I) know: > - Both re and regex use the same C backend, which is not based on NFA. > - The re2 library, which the writer of that article made, allows > capture groups (but only up to a limit) and bounded repetitions (up to > a limit). > - Perl has started to optimize some regex patterns. > [snip] re and regex use different C backends. Both are based on NFA. re2 is based on DFA, with a fallback to re. From soltysh at gmail.com Thu Feb 2 16:24:37 2017 From: soltysh at gmail.com (Maciej Szulik) Date: Thu, 2 Feb 2017 22:24:37 +0100 Subject: [Python-Dev] Heads up: possible double-comments on bpo for commits In-Reply-To: References: Message-ID: On Thu, Feb 2, 2017 at 6:01 PM, Victor Stinner wrote: > 2017-02-01 18:32 GMT+01:00 Victor Stinner : > > To finish with an empty change: > > Information on IRC: > > Known issue: http://psf.upfronthosting.co.za/roundup/meta/issue611 > > "Maciej is working on it" > > Good & thanks! > > I just fixed all the problems mentioned by Victor in this thread, this includes: 1. accidentally closing issues 2. multiple entries in history 3. empty date for gh commit messages (path by Berker - thank you) Ezio confirmed pushing all the changes to bugs.python.org already. I'm very sorry for all the inconvenience caused by previous mistakes. If your issues was closed make sure to verify it was closed properly and re-open if needed. Maciej -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Thu Feb 2 16:44:41 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 2 Feb 2017 22:44:41 +0100 Subject: [Python-Dev] Heads up: possible double-comments on bpo for commits In-Reply-To: References: Message-ID: 2017-02-02 22:24 GMT+01:00 Maciej Szulik : > Ezio confirmed pushing all the changes to bugs.python.org already. > I'm very sorry for all the inconvenience caused by previous mistakes. If > your > issues was closed make sure to verify it was closed properly and re-open if > needed. Don't worry. Brett warned us and it was quickly fixed. Thanks! Victor From ajm at flonidan.dk Thu Feb 2 05:17:57 2017 From: ajm at flonidan.dk (Anders Munch) Date: Thu, 2 Feb 2017 10:17:57 +0000 Subject: [Python-Dev] Generator objects and list comprehensions? Message-ID: Craig Rodrigues : > Make this return a list on Python 3, like in Python 2: ?[(yield?1)?for?x?in?range(10)]? Give Python 2 a little more credit. Python 2.7.10 (default, May 23 2015, 09:40:32) [MSC v.1500 32 bit (Intel)] on win32 Type "help", "copyright", "credits" or "license" for more information. >>> [(yield 1) for x in range(10)] File "", line 1 SyntaxError: 'yield' outside function regards, Anders From lukasz at langa.pl Fri Feb 3 01:21:10 2017 From: lukasz at langa.pl (Lukasz Langa) Date: Thu, 2 Feb 2017 22:21:10 -0800 Subject: [Python-Dev] Generator objects and list comprehensions? In-Reply-To: References: Message-ID: <2CCCA174-6405-4F89-A58D-382B485F4C15@langa.pl> > On Feb 2, 2017, at 2:17 AM, Anders Munch wrote: > > Give Python 2 a little more credit. We are, it told you what your issue was: yield outside a function. Consider: >>> def f(): ... l = [(yield 1) for x in range(10)] ... print(l) >>> gen = f() >>> for i in range(11): ... gen.send(i or None) ... 1 1 1 1 1 1 1 1 1 1 [1, 2, 3, 4, 5, 6, 7, 8, 9, 10] But this is a very convoluted "feature" and likely people don't expect *this* to be what's happening. - ? From status at bugs.python.org Fri Feb 3 12:09:12 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 3 Feb 2017 18:09:12 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170203170912.0AA6756907@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-01-27 - 2017-02-03) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5769 ( +1) closed 35423 (+55) total 41192 (+56) Open issues with patches: 2479 Issues opened (35) ================== #29300: Modify the _struct module to use FASTCALL and Argument Clinic http://bugs.python.org/issue29300 reopened by serhiy.storchaka #29382: IDLE search and replace should use 'hit' tag instead of 'hilit http://bugs.python.org/issue29382 opened by d s2 #29386: select.epoll.poll may behave differently if timeout = -1 vs ti http://bugs.python.org/issue29386 opened by Gabriel McManus #29387: Tabs vs spaces FAQ out of date http://bugs.python.org/issue29387 opened by martin.panter #29392: msvcrt.locking crashes python http://bugs.python.org/issue29392 opened by RazerM #29394: Cannot tunnel TLS connection through TLS connection http://bugs.python.org/issue29394 opened by Maximilian Blochberger #29395: Edit with IDLE 3.6 don??t work if Name of File "code.py" http://bugs.python.org/issue29395 opened by wueste87 #29396: Re-opening /dev/tty breaks readline http://bugs.python.org/issue29396 opened by silvioricardoc #29397: linux/random.h present but cannot be compiled http://bugs.python.org/issue29397 opened by zhaw_munt #29399: python3.dll export forwarders not resolved in all situations d http://bugs.python.org/issue29399 opened by Elli Pirelli #29400: Instruction level tracing via sys.settrace http://bugs.python.org/issue29400 opened by George.King #29402: Problem with Checkbutton and duplicate last name components http://bugs.python.org/issue29402 opened by terry.reedy #29403: mock's autospec's behavior on method-bound builtin functions i http://bugs.python.org/issue29403 opened by Aaron Gallagher #29405: improve csv.Sniffer().sniff() behavior http://bugs.python.org/issue29405 opened by mepstein #29406: asyncio SSL contexts leak sockets after calling close with cer http://bugs.python.org/issue29406 opened by thehesiod #29408: Add timestamp-based dependency system to build_clib http://bugs.python.org/issue29408 opened by daniel.nunes #29409: Implement PEP 529 for io.FileIO http://bugs.python.org/issue29409 opened by eryksun #29410: Moving to SipHash-1-3 http://bugs.python.org/issue29410 opened by inada.naoki #29411: Option --executable for entry_points http://bugs.python.org/issue29411 opened by Michal Cyprian #29412: IndexError thrown on email.message.Message.get http://bugs.python.org/issue29412 opened by uckelman #29413: ssl.SSLSocket.unwrap() is not flexible enough http://bugs.python.org/issue29413 opened by Greg Stark #29414: Change 'the for statement is such an iterator' in Tutorial http://bugs.python.org/issue29414 opened by Jim Fasarakis-Hilliard #29416: Path.mkdir can get into a recursive error loop http://bugs.python.org/issue29416 opened by Dan Buchoff #29417: Sort entries in foo.dist-info/RECORD http://bugs.python.org/issue29417 opened by dgreiman #29418: inspect.isroutine does not return True for some bound builtin http://bugs.python.org/issue29418 opened by Wheerd #29419: Argument Clinic: inline PyArg_UnpackTuple and PyArg_ParseStack http://bugs.python.org/issue29419 opened by haypo #29422: add socket independent timeout to httplib/http.client read http://bugs.python.org/issue29422 opened by Patrick Michaud #29424: Multiple vulnerabilities in BaseHTTPRequestHandler enable HTTP http://bugs.python.org/issue29424 opened by meitar #29425: File-altering aspects of pathlib should return new pathlib obj http://bugs.python.org/issue29425 opened by Walter Szeliga #29427: Option to skip padding for base64 urlsafe encoding/decoding http://bugs.python.org/issue29427 opened by Thorney #29428: Doctest documentation unclear about multi-line fixtures http://bugs.python.org/issue29428 opened by JDLH #29430: zipfile: invalid link http://bugs.python.org/issue29430 opened by lucasmus #29432: wait_for(gather(...)) logs weird error message http://bugs.python.org/issue29432 opened by Martin.Teichmann #29435: Allow to pass fileobj to is_tarfile http://bugs.python.org/issue29435 opened by twiggers #29436: Compilation failure against Android NDK r14 beta 2 http://bugs.python.org/issue29436 opened by Chi Hsuan Yen Most recent 15 issues with no replies (15) ========================================== #29435: Allow to pass fileobj to is_tarfile http://bugs.python.org/issue29435 #29432: wait_for(gather(...)) logs weird error message http://bugs.python.org/issue29432 #29430: zipfile: invalid link http://bugs.python.org/issue29430 #29427: Option to skip padding for base64 urlsafe encoding/decoding http://bugs.python.org/issue29427 #29425: File-altering aspects of pathlib should return new pathlib obj http://bugs.python.org/issue29425 #29422: add socket independent timeout to httplib/http.client read http://bugs.python.org/issue29422 #29418: inspect.isroutine does not return True for some bound builtin http://bugs.python.org/issue29418 #29417: Sort entries in foo.dist-info/RECORD http://bugs.python.org/issue29417 #29413: ssl.SSLSocket.unwrap() is not flexible enough http://bugs.python.org/issue29413 #29411: Option --executable for entry_points http://bugs.python.org/issue29411 #29409: Implement PEP 529 for io.FileIO http://bugs.python.org/issue29409 #29408: Add timestamp-based dependency system to build_clib http://bugs.python.org/issue29408 #29396: Re-opening /dev/tty breaks readline http://bugs.python.org/issue29396 #29395: Edit with IDLE 3.6 don??t work if Name of File "code.py" http://bugs.python.org/issue29395 #29370: "./configure --enable-optimizations && make install" does not http://bugs.python.org/issue29370 Most recent 15 issues waiting for review (15) ============================================= #29436: Compilation failure against Android NDK r14 beta 2 http://bugs.python.org/issue29436 #29424: Multiple vulnerabilities in BaseHTTPRequestHandler enable HTTP http://bugs.python.org/issue29424 #29414: Change 'the for statement is such an iterator' in Tutorial http://bugs.python.org/issue29414 #29410: Moving to SipHash-1-3 http://bugs.python.org/issue29410 #29409: Implement PEP 529 for io.FileIO http://bugs.python.org/issue29409 #29408: Add timestamp-based dependency system to build_clib http://bugs.python.org/issue29408 #29406: asyncio SSL contexts leak sockets after calling close with cer http://bugs.python.org/issue29406 #29403: mock's autospec's behavior on method-bound builtin functions i http://bugs.python.org/issue29403 #29400: Instruction level tracing via sys.settrace http://bugs.python.org/issue29400 #29387: Tabs vs spaces FAQ out of date http://bugs.python.org/issue29387 #29376: threading._DummyThread.__repr__ raises AssertionError http://bugs.python.org/issue29376 #29362: regrtest: don't fail immediately if a child does crash http://bugs.python.org/issue29362 #29353: Incorrect handling of HTTP response with "Content-Type: messag http://bugs.python.org/issue29353 #29352: provide the authorative source for s[i:j] negative slice indic http://bugs.python.org/issue29352 #29351: absolute imports for logging http://bugs.python.org/issue29351 Top 10 most discussed issues (10) ================================= #29300: Modify the _struct module to use FASTCALL and Argument Clinic http://bugs.python.org/issue29300 29 msgs #29410: Moving to SipHash-1-3 http://bugs.python.org/issue29410 19 msgs #11549: Build-out an AST optimizer, moving some functionality out of t http://bugs.python.org/issue11549 18 msgs #29400: Instruction level tracing via sys.settrace http://bugs.python.org/issue29400 11 msgs #18842: Add float.is_finite is_nan is_infinite to match Decimal method http://bugs.python.org/issue18842 10 msgs #27200: make doctest in CPython has failures http://bugs.python.org/issue27200 7 msgs #27647: Update Windows 2.7 build to Tcl/Tk 8.5.19 http://bugs.python.org/issue27647 7 msgs #29344: sock_recv not detected a coroutine http://bugs.python.org/issue29344 7 msgs #15373: copy.copy() does not properly copy os.environment http://bugs.python.org/issue15373 5 msgs #29399: python3.dll export forwarders not resolved in all situations d http://bugs.python.org/issue29399 5 msgs Issues closed (54) ================== #14376: sys.exit documents argument as "integer" but actually requires http://bugs.python.org/issue14376 closed by mark.dickinson #18590: Found text not always highlighted by Replace dialog on Windows http://bugs.python.org/issue18590 closed by terry.reedy #23298: Add ArgumentParser.add_mutually_dependence_group http://bugs.python.org/issue23298 closed by paul.j3 #23551: IDLE to provide menu link to PIP gui. http://bugs.python.org/issue23551 closed by terry.reedy #24736: argparse add_mutually_exclusive_group do not print help http://bugs.python.org/issue24736 closed by paul.j3 #24875: pyvenv doesn??t install PIP inside a new venv with --system-si http://bugs.python.org/issue24875 closed by python-dev #27051: Create PIP gui http://bugs.python.org/issue27051 closed by ncoghlan #28618: Decorate hot functions using __attribute__((hot)) to optimize http://bugs.python.org/issue28618 closed by haypo #29140: time_hash() reads the wrong bytes to get microseconds http://bugs.python.org/issue29140 closed by haypo #29157: random.c: Prefer getrandom() over getentropy() to support glib http://bugs.python.org/issue29157 closed by haypo #29169: update zlib to 1.2.11 http://bugs.python.org/issue29169 closed by doko #29182: Remove the warning in urllib docs that it doesn't do certifica http://bugs.python.org/issue29182 closed by orsenthil #29213: python -m venv activate.bat has weird mix of line endings http://bugs.python.org/issue29213 closed by python-dev #29218: distutils: Remove unused install_misc class http://bugs.python.org/issue29218 closed by berker.peksag #29227: Reduce C stack consumption in function calls http://bugs.python.org/issue29227 closed by haypo #29234: Disable inlining of _PyStack_AsTuple() to reduce the stack con http://bugs.python.org/issue29234 closed by haypo #29260: Use designated initializers to define PyTypeObject types http://bugs.python.org/issue29260 closed by haypo #29263: Implement LOAD_METHOD/CALL_METHOD for C functions http://bugs.python.org/issue29263 closed by inada.naoki #29264: sparc/ffi.c:440 error: 'asm' undeclared http://bugs.python.org/issue29264 closed by xdegaye #29283: duplicate README in site-packages http://bugs.python.org/issue29283 closed by berker.peksag #29286: Use METH_FASTCALL in str methods http://bugs.python.org/issue29286 closed by haypo #29289: Convert OrderedDict methods to Argument Clinic http://bugs.python.org/issue29289 closed by haypo #29310: Document typing.NamedTuple default argument syntax http://bugs.python.org/issue29310 closed by rhettinger #29311: Argument Clinic: convert dict methods http://bugs.python.org/issue29311 closed by haypo #29318: Optimize _PyFunction_FastCallDict() for **kwargs http://bugs.python.org/issue29318 closed by haypo #29329: Incorrect documentation for custom `hex()` support on Python 2 http://bugs.python.org/issue29329 closed by Mariatta #29349: Update old Python 2 code in Docs/tools/extensions/patchlevel.p http://bugs.python.org/issue29349 closed by martin.panter #29354: Python 2.7.12: pydoc.help(lambda (a,): lambda x: a) raises Ind http://bugs.python.org/issue29354 closed by serhiy.storchaka #29364: msilib Fetch raises MSIError rather than return None http://bugs.python.org/issue29364 closed by Jason Matthew #29367: python-gdb: display wrapper_call() http://bugs.python.org/issue29367 closed by python-dev #29368: Optimize unpickling list-like objects: use extend() instead of http://bugs.python.org/issue29368 closed by python-dev #29377: Add the 'wrapper_descriptor' type to the types module http://bugs.python.org/issue29377 closed by gvanrossum #29381: Tutorial documentation contains undefined reference to #! http://bugs.python.org/issue29381 closed by python-dev #29383: Reduce temporary unicode object while adding descriptors http://bugs.python.org/issue29383 closed by inada.naoki #29384: Unused beos build scripts http://bugs.python.org/issue29384 closed by martin.panter #29385: Sockets Crashes or Memory Corruption http://bugs.python.org/issue29385 closed by inada.naoki #29388: regex mismatch in the simple cases http://bugs.python.org/issue29388 closed by steven.daprano #29389: math.isclose signature contains incorrect start parameter http://bugs.python.org/issue29389 closed by ammar2 #29390: Python Tutorial should introduce iterators and generators http://bugs.python.org/issue29390 closed by rhettinger #29391: Windows Defender finds trojan Spursint in Numpy for Py36 Win64 http://bugs.python.org/issue29391 closed by eryksun #29393: Enabiling IPv6 by default http://bugs.python.org/issue29393 closed by ned.deily #29398: memory corruption in xxlimited http://bugs.python.org/issue29398 closed by python-dev #29401: Python3.6 docs should not have a "what's new in Python 3.7" pa http://bugs.python.org/issue29401 closed by ned.deily #29404: "TypeError: 'int' does not have the buffer interface" on memor http://bugs.python.org/issue29404 closed by brett.cannon #29407: Remove redundant ensure_future() calls in factorial example http://bugs.python.org/issue29407 closed by berker.peksag #29415: Exposing handle._callback and handle._args in asyncio http://bugs.python.org/issue29415 closed by yselivanov #29420: Python 3.6 change in dict iteration when inserting keys http://bugs.python.org/issue29420 closed by r.david.murray #29421: Make int.to_bytes() and int.from_bytes() slightly faster http://bugs.python.org/issue29421 closed by serhiy.storchaka #29423: using concurrent.futures.ProcessPoolExecutor in class giving ' http://bugs.python.org/issue29423 closed by serhiy.storchaka #29426: Using pywin32 on Python3 on linux http://bugs.python.org/issue29426 closed by eryksun #29429: Requests package is incompatible with py2exe http://bugs.python.org/issue29429 closed by inada.naoki #29431: Add a PYTHONREVERSEDICTKEYORDER environment variable http://bugs.python.org/issue29431 closed by rhettinger #29433: any, all and sum should accept variadic args http://bugs.python.org/issue29433 closed by serhiy.storchaka #29434: tuple.__repr__ may segv when it contains NULL http://bugs.python.org/issue29434 closed by inada.naoki From mdaglow at daglowconsulting.com Fri Feb 3 20:09:49 2017 From: mdaglow at daglowconsulting.com (Marta Daglow) Date: Fri, 3 Feb 2017 17:09:49 -0800 Subject: [Python-Dev] Top python roles Message-ID: <6086424b08d92461c2e91e968a249eef@mail.gmail.com> Hi Guys, How are you guys? I?ve just gotten off the phone with a top engineering leader from a wonderful company in SF and they are looking for someone with python expertise. They are looking to hire many people in the range of $100 ? 160K. Warm Regards, Marta -------------- next part -------------- An HTML attachment was scrubbed... URL: From brian at python.org Fri Feb 3 20:20:10 2017 From: brian at python.org (Brian Curtin) Date: Sat, 04 Feb 2017 01:20:10 +0000 Subject: [Python-Dev] Top python roles In-Reply-To: <6086424b08d92461c2e91e968a249eef@mail.gmail.com> References: <6086424b08d92461c2e91e968a249eef@mail.gmail.com> Message-ID: There are a lot more than "guys" on this list, and it's for the development of the Python language, not for recruiting. Please take this elsewhere. On Fri, Feb 3, 2017 at 20:10 Marta Daglow wrote: > Hi Guys, > > > > How are you guys? > > > > I?ve just gotten off the phone with a top engineering leader from a > wonderful company in SF and they are looking for someone with python > expertise. > > > > They are looking to hire many people in the range of $100 ? 160K. > > > > Warm Regards, > > > > Marta > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brian%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Sat Feb 4 04:59:34 2017 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sat, 4 Feb 2017 10:59:34 +0100 Subject: [Python-Dev] Review request: optimizing typing ABC caches Message-ID: There is a PR implementing typing ABC cache optimization: https://github.com/python/typing/pull/383 The main idea is straightforward: subscripted generic ABCs like Iterable[int], Iterable[str], etc. should not have separate ABC caches (positive and negative), since they all are equivalent to plain Iterable at runtime. It is proposed that they will share their caches with a parent ABC from abc module for abstract collections, or with original (unsubscripted) class generic for concrete classes. Inada-san confirmed that this optimization reduces the memory footprint. I will be grateful for a code review. Best regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From marco.buttu at gmail.com Sun Feb 5 01:59:04 2017 From: marco.buttu at gmail.com (Marco Buttu) Date: Sun, 05 Feb 2017 07:59:04 +0100 Subject: [Python-Dev] User + sys time bigger than real time, in case of no real parallelism Message-ID: <5896CD38.5020302@oa-cagliari.inaf.it> I am really sorry for the OT :-( I asked elsewhere but without any answer :-( I can not figure out why in this short example the user+sys time is bigger than real time. The example executes the task() functions twice, with each execution in a separate thread. The task() just increment 10**6 times a global int: $ cat foo.py from threading import Thread, Lock result = 0 lock = Lock() def task(): global result for i in range(10**6): lock.acquire() result += 1 lock.release() if __name__ == '__main__': t1, t2 = Thread(target=task), Thread(target=task) t1.start() t2.start() t1.join() t2.join() print('result:', result) When I execute it (Python 3.6), I get a sys+user time bigger than the real time: $ time python foo.py result: 2000000 real 0m7.088s user 0m6.597s sys 0m5.043s That is usually what I can expect in case of tasks executed in parallel on different CPUs. But my example should not be the case, due to the GIL. What am I missing? Thank you very much, and sorry again for the OT :( -- Marco Buttu INAF-Osservatorio Astronomico di Cagliari Via della Scienza n. 5, 09047 Selargius (CA) Phone: 070 711 80 217 Email: mbuttu at oa-cagliari.inaf.it From stefanrin at gmail.com Sun Feb 5 03:58:51 2017 From: stefanrin at gmail.com (Stefan Ring) Date: Sun, 5 Feb 2017 09:58:51 +0100 Subject: [Python-Dev] User + sys time bigger than real time, in case of no real parallelism In-Reply-To: <5896CD38.5020302@oa-cagliari.inaf.it> References: <5896CD38.5020302@oa-cagliari.inaf.it> Message-ID: > That is usually what I can expect in case of tasks executed in parallel on > different CPUs. But my example should not be the case, due to the GIL. What > am I missing? Thank you very much, and sorry again for the OT :( With such finely intermingled thread activity, there might be a fair bit of spinlock spinning involved. Additionally, I suspect that the kernel does not track CPU time at microsecond precision and will tend to round the used times up. Obviously, this is not a reasonable way to use threads. The example is only effective at producing lots of overhead. From hpj at urpla.net Sun Feb 5 08:31:19 2017 From: hpj at urpla.net (Hans-Peter Jansen) Date: Sun, 05 Feb 2017 14:31:19 +0100 Subject: [Python-Dev] User + sys time bigger than real time, in case of no real parallelism In-Reply-To: <5896CD38.5020302@oa-cagliari.inaf.it> References: <5896CD38.5020302@oa-cagliari.inaf.it> Message-ID: <5389121.hDbacd5G2U@xrated> On Sonntag, 5. Februar 2017 07:59:04 Marco Buttu wrote: > I am really sorry for the OT :-( I asked elsewhere but without any > answer :-( > I can not figure out why in this short example the user+sys time is > bigger than real time. The example executes the task() functions twice, > with each execution in a separate thread. The task() just increment > 10**6 times a global int: > > $ cat foo.py > from threading import Thread, Lock > > result = 0 > lock = Lock() > > def task(): > global result > for i in range(10**6): > lock.acquire() > result += 1 > lock.release() > > if __name__ == '__main__': > t1, t2 = Thread(target=task), Thread(target=task) > t1.start() > t2.start() > t1.join() > t2.join() > print('result:', result) > > When I execute it (Python 3.6), I get a sys+user time bigger than the > real time: > > $ time python foo.py > result: 2000000 > > real 0m7.088s > user 0m6.597s > sys 0m5.043s > > That is usually what I can expect in case of tasks executed in parallel > on different CPUs. But my example should not be the case, due to the > GIL. What am I missing? Thank you very much, and sorry again for the OT :( /usr/bin/time(!) gives some insight: $ /usr/bin/time python3 foo.py result: 2000000 1.18user 1.30system 0:01.52elapsed 163%CPU (0avgtext+0avgdata 8228maxresident)k 8inputs+0outputs (0major+1151minor)pagefaults 0swaps Obviously, the spinning happens outside the GIL... Cheers, Pete From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Feb 5 20:34:35 2017 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 6 Feb 2017 10:34:35 +0900 Subject: [Python-Dev] SSL certificates recommendations for downstreampython packagers In-Reply-To: <80524675-7B66-4742-B367-EEA10F37D1CA@lukasa.co.uk> References: <8296B2E6-5444-43B0-8C71-6398DDDCB74A@lukasa.co.uk> <4cece125-61c5-ec85-7137-f425badb1bc6@python.org> <781d78df-f5e5-6fba-136d-a2967bbbdf52@python.org> <8FF8240C-EA4D-4F01-868F-28798E7235DB@lukasa.co.uk> <22674.43439.871545.345238@turnbull.sk.tsukuba.ac.jp> <80524675-7B66-4742-B367-EEA10F37D1CA@lukasa.co.uk> Message-ID: <22679.53931.254274.593383@turnbull.sk.tsukuba.ac.jp> Cory Benfield writes: > From a security perspective I think we have to discount the > possibility of administrator error from our threat model. I disagree in a certain sense, and in that sense you don't discount it -- see below. > A threat model that includes ?defend the system against intrusions > that the administrator incorrectly allows? I agree that child-proof locks don't work. The point of having a category called "administrator error" in the threat model is not to instantiate it, but merely to recognize it: > where we allow configuration we have a duty to ensure that it?s as > easy as possible to configure correctly, and in particular defaults should (1) "deny everything" (well, nearly), and (2) be robust ("forbid what is not explicitly permitted") to configuration changes that allow accesses wherever Python can reasonably achieve that. > but when using the system trust store most of the configuration is > actually provided by the OS tools, rather than by the > above-mentioned ?you?, so that?s not in our control. OK, up to the problem that OS tools may not be accessible or may be considered unreliable. I trust you guys to do something sane there, and I agree it's covered by the "we can't correct admin mistakes in complex environments" clause that you invoked above. Python cannot take responsibility for guessing what might happen in any given configuration in such environments. > However, it?s unquestionable that the *safest* route to go down in > terms of preserving the expectations of users is to use the > platform-native TLS implementation wholesale, rather than do a > hybrid model like Chrome does where OpenSSL does the protocol bits > and the system does the X509 bits. That way Python ends up behaving > basically like Edge or Safari on the relevant platforms, or perhaps > more importantly behaving like .NET on Windows and like > CoreFoundation on macOS, which is a much better place to be in > terms of user and administrator expectations. OK, I can't help you with the details, but I can at least say I feel safer when you say that's where you're going. :-) From brett at python.org Tue Feb 7 13:03:24 2017 From: brett at python.org (Brett Cannon) Date: Tue, 07 Feb 2017 18:03:24 +0000 Subject: [Python-Dev] GitHub migration scheduled for Friday Message-ID: To let the non-core devs know, the GitHub migration will be happening this Friday. For those of you who use the current GitHub mirror to create patches, do be aware that the hashes will most likely be changing so don't expect your checkout to work past Thursday (you can always generate a patch and apply it to a fresh checkout). Otherwise https://cpython-devguide.readthedocs.io/en/github/ is what the devguide will become on Friday if you want to read now instead of waiting for the official switch-over (although for non-core devs the migration basically means you can use GitHub to submit changes instead of uploading patches). -------------- next part -------------- An HTML attachment was scrubbed... URL: From jelle.zijlstra at gmail.com Tue Feb 7 13:52:19 2017 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Tue, 7 Feb 2017 10:52:19 -0800 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: 2017-02-07 10:03 GMT-08:00 Brett Cannon : > To let the non-core devs know, the GitHub migration will be happening this > Friday. For those of you who use the current GitHub mirror to create > patches, do be aware that the hashes will most likely be changing so don't > expect your checkout to work past Thursday (you can always generate a patch > and apply it to a fresh checkout). Otherwise https://cpython- > devguide.readthedocs.io/en/github/ is what the devguide will become on > Friday if you want to read now instead of waiting for the official > switch-over (although for non-core devs the migration basically means you > can use GitHub to submit changes instead of uploading patches). > > This is great, I'm looking forward to being able to submit pull requests to CPython. Thanks Brett and others for all your hard work on this! > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > jelle.zijlstra%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.f.hilliard at gmail.com Tue Feb 7 14:07:19 2017 From: d.f.hilliard at gmail.com (Jim F.Hilliard) Date: Tue, 7 Feb 2017 21:07:19 +0200 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: That's great, congratulations! I believe this change will make it way easier for people to get involved! A small question, since people can now submit new issues via pulls instead of going to bugs.python.org, what will be the purpose of the latter? As I skimmed through cpython-devguide.readthedocs.io I've see the issue tracker being referenced in certain areas (File a bug section, Reviewing) so I'm not sure I understand how they overlap. -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephane at wirtel.be Tue Feb 7 14:21:37 2017 From: stephane at wirtel.be (Stephane Wirtel) Date: Tue, 7 Feb 2017 20:21:37 +0100 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: <20170207192137.e4jbuwzcw4cy76sh@sg1> Nice, good news. On 02/07, Brett Cannon wrote: >To let the non-core devs know, the GitHub migration will be happening this >Friday. For those of you who use the current GitHub mirror to create >patches, do be aware that the hashes will most likely be changing so don't >expect your checkout to work past Thursday (you can always generate a patch >and apply it to a fresh checkout). Otherwise >https://cpython-devguide.readthedocs.io/en/github/ is what the devguide >will become on Friday if you want to read now instead of waiting for the >official switch-over (although for non-core devs the migration basically >means you can use GitHub to submit changes instead of uploading patches). >_______________________________________________ >Python-Dev mailing list >Python-Dev at python.org >https://mail.python.org/mailman/listinfo/python-dev >Unsubscribe: https://mail.python.org/mailman/options/python-dev/stephane%40wirtel.be -- St?phane Wirtel - http://wirtel.be - @matrixise From brett at python.org Tue Feb 7 16:49:58 2017 From: brett at python.org (Brett Cannon) Date: Tue, 07 Feb 2017 21:49:58 +0000 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: On Tue, 7 Feb 2017 at 11:17 Jim F.Hilliard wrote: > That's great, congratulations! I believe this change will make it way > easier for people to get involved! > > A small question, since people can now submit new issues via pulls instead > of going to bugs.python.org, what will be the purpose of the latter? > The issue tracker is not moving. GitHub will be used for code hosting and pull request management. Tracking bugs and such will stay at bugs.python.org. There's now integration between GitHub and bpo so that if you reference an issue in a pull request a connection will be made on bpo. -Brett > > As I skimmed through cpython-devguide.readthedocs.io I've see the issue > tracker being referenced in certain areas (File a bug section, Reviewing) > so I'm not sure I understand how they overlap. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-dev at mgmiller.net Tue Feb 7 17:15:10 2017 From: python-dev at mgmiller.net (Mike Miller) Date: Tue, 7 Feb 2017 14:15:10 -0800 Subject: [Python-Dev] Py 3.6 on Ubuntu Zesty Message-ID: <62fa0468-add8-b2aa-6781-b687dd7c4eb9@mgmiller.net> Hi, Does anyone know why Python 3.6 is not the default Python 3 under the upcoming Ubuntu Zesty, or what may be holding it back? Is there anyone that could give it a nudge? It's in the repos but not as python3: http://packages.ubuntu.com/zesty/python3 http://packages.ubuntu.com/zesty/python3.6 -Mike From barry at python.org Tue Feb 7 17:38:43 2017 From: barry at python.org (Barry Warsaw) Date: Tue, 7 Feb 2017 17:38:43 -0500 Subject: [Python-Dev] Py 3.6 on Ubuntu Zesty In-Reply-To: <62fa0468-add8-b2aa-6781-b687dd7c4eb9@mgmiller.net> References: <62fa0468-add8-b2aa-6781-b687dd7c4eb9@mgmiller.net> Message-ID: <20170207173843.596fd297@subdivisions.wooz.org> On Feb 07, 2017, at 02:15 PM, Mike Miller wrote: >Does anyone know why Python 3.6 is not the default Python 3 under the >upcoming Ubuntu Zesty, or what may be holding it back? I guess that would be me. :) >Is there anyone that could give it a nudge? It's in the repos but not as >python3: > >http://packages.ubuntu.com/zesty/python3 >http://packages.ubuntu.com/zesty/python3.6 I posted about this on the ubuntu-devel mailing list: https://lists.ubuntu.com/archives/ubuntu-devel/2017-January/039648.html It's in the Zesty (and actually Yakkety) archives so that we can do the test rebuilds described in the above message. But it's not yet a "supported" version, which means that while most pure-Python third party modules should be importable, anything with an extension module will not be. On Debian and derivatives, we share what is essentially site-packages among all *installed* versions of Python 3, which doesn't necessarily mean that those packages will work, but they should be available on the import path with no extra work. Packages may still need to be functionally ported to Python 3.6 of course. However, because extension modules need to be recompiled by the Ubuntu build daemons, they only get done for "supported" versions. That term has a specific meaning in the Debian Python packaging sense, but I don't think the details are important (ask if you're interested). "Supported" is a step before "default", which is where we change the /usr/bin/python3 symlink. This step does not require an archive rebuild. Still for now you can `apt install python3.6` and run it via /usr/bin/python3.6 with the above caveats of course. It's functional enough to build venvs (you might want the python3.6-venv package), and tox will work, so it's everything folks should need to do porting work. Here's another page collecting some information: https://wiki.ubuntu.com/Python With a 9-12% build failure (again, forgetting about whether the packages actually work), it's enough that given where we are in the cycle, supporting Python 3.6 is not feasible. Debian Stretch just entered freeze so any work we do in Ubuntu to make packages Python 3.6 compatible will eventually have to be synced back to Debian, since they can't be uploaded there directly until Stretch is released. Zesty itself enters feature freeze on Feb 16, so that's another looming deadline on the amount of work we can accomplish this cycle. My plan is to enable Python 3.6 as a supported version early in the Zesty+1 cycle, hopefully before the first archive rebuild. I don't know whether Stretch will be released by then, but we'll deal with as we have before if that's the case. In the meantime, this should give upstreams a couple of more months to continue to port, and it will give us a full cycle to work on porting packages in Ubuntu. I plan for 18.04 LTS to ship only Python 3.6 (similarly for Debian Stretch+1). Dropping Python 3.5 does require an archive rebuild, but it's much less risky than adding a new version. The best way to help right now is to work with upstreams to port to Python 3.6 if necessary. You can see our test rebuild results here: https://launchpad.net/~pythoneers/+archive/ubuntu/python-rebuilds/+packages Anything with a red X means the package failed to build. That usually flexes at least the upstream's own unittests, but there may be deeper issues and other tests that fail, which may or may not hold up individual packages, and anything that depends on them. The Fedora/RH ecosystem probably has their own list, which I'd expect to mostly overlap with ours, but I don't have those links handy. Cheers, -Barry From python at nedharvey.com Tue Feb 7 17:19:31 2017 From: python at nedharvey.com (Edward Ned Harvey (python)) Date: Tue, 7 Feb 2017 22:19:31 +0000 Subject: [Python-Dev] Mac OSX SSL certs Message-ID: I would like to suggest that the OSX installer automatically run "Install Certificates.command", or display a prompt to users saying "Run Now" during installation. Having the readme is helpful - but only after you google for 20 minutes, because of an exception you encountered. Of course nobody reads the readme during install. "I've installed python a thousand times before, I know what I'm doing." There are so many things that require SSL, and it's reasonably assumed to be functional by default. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Tue Feb 7 20:23:14 2017 From: nad at python.org (Ned Deily) Date: Tue, 7 Feb 2017 20:23:14 -0500 Subject: [Python-Dev] Mac OSX SSL certs In-Reply-To: References: Message-ID: <8788FAEB-10E2-4EB0-8046-76E65555A699@python.org> On Feb 7, 2017, at 17:19, Edward Ned Harvey (python) wrote: > I would like to suggest that the OSX installer automatically run "Install Certificates.command", or display a prompt to users saying "Run Now" during installation. Thanks for your suggestion. Please open an issue for this on our bug tracker, http://bugs.python.org. -- Ned Deily nad at python.org -- [] From victor.stinner at gmail.com Wed Feb 8 02:35:04 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 8 Feb 2017 08:35:04 +0100 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: (Oops, I wrote privately to Brett, so he replied me in private. So here is a copy of our emails.) Brett Cannon via gmail.com On Tue, 7 Feb 2017 at 13:42 Victor Stinner wrote: > > If I push a patch file written by someone else, should I try to use the author full name + email? I'm sure the patch author would appreciate it, but I don't think we need to require it as we have gone this long without it. > Currently, it requires some tricks to get these informations (the email is partially hidden in the big tracker user list). Or are we moving slowly to GitHub pull requests only? I hope we're moving quickly and not slowly. :) > Maybe the simplest option is to ask these informations to the author :-) I suspect if the patch author is still active you could ask them to make their patch into a PR and that will solve this problem. -Brett From victor.stinner at gmail.com Wed Feb 8 02:38:00 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 8 Feb 2017 08:38:00 +0100 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: 2017-02-08 8:35 GMT+01:00 Victor Stinner : > I'm sure the patch author would appreciate it, but I don't think we > need to require it as we have gone this long without it. I know that different people have different expectation on GitHub. I would like to take the opportunity of migrating to Git to use the "author" and "committer" fields. If the author is set to the real author, the one who proposed the change on the bug tracker or someone else, we will be able to compute statistics on most active contributors to more easily detect them and promote them to core developers. What do you think? I agree that pull requests avoid the issues of filling manually the author field, but I expect that no everyone will use PR, especially because we will still use our current bug tracker which accepts to attach patch files. Ok to propose them to create a PR instead. Victor From senthil at uthcode.com Wed Feb 8 02:52:03 2017 From: senthil at uthcode.com (Senthil Kumaran) Date: Tue, 7 Feb 2017 23:52:03 -0800 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: On Tue, Feb 7, 2017 at 11:38 PM, Victor Stinner wrote: > I know that different people have different expectation on GitHub. I > would like to take the opportunity of migrating to Git to use the > "author" and "committer" fields. If the author is set to the real > author, the one who proposed the change on the bug tracker or someone > else, we will be able to compute statistics on most active > contributors to more easily detect them and promote them to core > developers. > > What do you think? I am +1 to this idea. The intention behind this idea is also good one. * When the patches come from Github PRs, the contribution authors are automatically tracked. The comitter would be merging the changes from the authors. * When contribution comes via Patches/ or for many existing patches, setting the author is a good idea. I have one question. If we disallow direct push to branches and the core-dev has to create a PR to merge the changes in, I guess it is still possible to have author information in the commit. From mail at timgolden.me.uk Wed Feb 8 03:45:01 2017 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 8 Feb 2017 08:45:01 +0000 Subject: [Python-Dev] Py 3.6 on Ubuntu Zesty In-Reply-To: <20170207173843.596fd297@subdivisions.wooz.org> References: <62fa0468-add8-b2aa-6781-b687dd7c4eb9@mgmiller.net> <20170207173843.596fd297@subdivisions.wooz.org> Message-ID: On 07/02/2017 22:38, Barry Warsaw wrote: > On Feb 07, 2017, at 02:15 PM, Mike Miller wrote: > >> Does anyone know why Python 3.6 is not the default Python 3 under the >> upcoming Ubuntu Zesty, or what may be holding it back? > > I guess that would be me. :) > >> Is there anyone that could give it a nudge? It's in the repos but not as >> python3: >> >> http://packages.ubuntu.com/zesty/python3 >> http://packages.ubuntu.com/zesty/python3.6 > > I posted about this on the ubuntu-devel mailing list: [... snip comprehensive explanation about how versions get promoted within Debian / Ubuntu ...] Well I'm only a casual Linux user, but this was interesting just because of the better understanding it gives of the processes which distros have to go through once new versions are released upstream. Thanks TJG From victor.stinner at gmail.com Wed Feb 8 05:24:55 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 8 Feb 2017 11:24:55 +0100 Subject: [Python-Dev] OpenIndiana and Solaris support In-Reply-To: References: Message-ID: Hi, Is there anything new about Solaris or OpenIndiana since September? Right now, it seems like the cea-indiana-x86 buildbot slave is offline since longer than 54 days. Oracle decided to stop Solaris 12 development: https://arstechnica.com/information-technology/2017/01/oracle-sort-of-confirms-demise-of-solaris-12-effort/ I'm not opposed to help if someone provides patches to fix Solaris issues, but it seems like no one wants to do this job. So I suggest to drop official Solaris support, but I don't propose to remove the C code specific to Solaris. In practice, I suggest to remove Solaris and OpenIndiana buildbots since they are broken for months and are more annoying than useful. Victor 2016-09-27 0:54 GMT+02:00 Brett Cannon : > > > On Mon, 26 Sep 2016 at 15:38 Guido van Rossum wrote: >> >> Thanks for the reality check Trent! I think if enough people with core >> committer bits want to keep supporting Solaris / Illumos / OpenIndiana >> / other variants that's fine, but I don't think that just having some >> VMs to test on is enough -- we also need people who can fix problems >> if those buildbots start failing, and that requires pretty specialized >> knowledge. Plus of course we won't know if fixing it for OpenIndiana >> will also fix it for Solaris 11 Express or for other Illumos forks. >> (For Linux it's easier to assess these things because so many people >> in open source use Linux and its many forks.) > > > The official requirement to support a platform is a stable buildbot and a > core dev to keep the support up: > https://www.python.org/dev/peps/pep-0011/#supporting-platforms. Victor has > asked that the OpenIndiana buildbot be removed from the stable pool as it > consistently throws MemoryError which means its support is not improving. If > Trent is willing to maintain a buildbot in a Joyent VM that at least takes > care of that part, but it still requires Jesus to volunteer to keep the > support up if it's going to be supported for free. Otherwise Joyent could > consider contracting with one of the various core devs who happen to be > consultants to help maintain the support. > > At minimum, though, a new buildbot could go into the unstable pool so > illumos devs can keep an eye on when things break to try and get > platform-independent changes upstreamed that happen to help illumos (e.g. no > #ifdef changes specific to illumos, but if something just needed to be made > more robust and it happens to help illumos that's typically fine). > >> >> >> On Mon, Sep 26, 2016 at 3:32 PM, Trent Mick wrote: >> > I work for Joyent (joyent.com) now, which employs a number of devs that >> > work >> > on illumos (illumos.org). We also provide cloud infrastructure. Would it >> > help if we offered one or more instances (VMs) on which to run buildbot >> > slaves (and on which volunteers for bug fixing could hack)? I know a >> > lot of >> > people in the illumos community would be quite sad to have it dropped as >> > a >> > core Python plat. >> > >> > Guido, >> > Yes you are correct that Oracle owns the Solaris brand. >> > >> > tl;dr history if you care: >> > - sunos -> Solaris >> > - Sun open sources Solaris, called OpenSolaris (2005) >> > - Oracle acquires Sun and closes Solaris (Aug 2010). Shortly after, the >> > community forks OpenSolaris and calls it illumos (Sep 2010) >> > - OpenIndiana is a distro of illumos (somewhat similar to how Ubuntu is >> > a >> > distro of Linux). Other distros are SmartOS (the one Joyent works on), >> > and >> > OmniOS. >> > - Oracle continues work on Solaris, releasing "Solaris 11 Express". >> > >> > I've no real numbers of usage of illumos vs Solaris 11 vs others. >> > >> > Cheers, >> > Trent >> > >> > p.s. I hear that Jesus is also in contact with some of the illumos-devs >> > on >> > IRC (and perhaps email). I hope we can help there. >> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com > From encukou at gmail.com Wed Feb 8 07:10:14 2017 From: encukou at gmail.com (Petr Viktorin) Date: Wed, 8 Feb 2017 13:10:14 +0100 Subject: [Python-Dev] Py 3.6 on Ubuntu Zesty In-Reply-To: <20170207173843.596fd297@subdivisions.wooz.org> References: <62fa0468-add8-b2aa-6781-b687dd7c4eb9@mgmiller.net> <20170207173843.596fd297@subdivisions.wooz.org> Message-ID: <843f993a-ec7f-17d7-0456-508466ed97fe@gmail.com> On 02/07/2017 11:38 PM, Barry Warsaw wrote: > On Feb 07, 2017, at 02:15 PM, Mike Miller wrote: > >> Does anyone know why Python 3.6 is not the default Python 3 under the >> upcoming Ubuntu Zesty, or what may be holding it back? > > I guess that would be me. :) > Thank for the in-depth explanation! > > The Fedora/RH ecosystem probably has their own list, which I'd expect to > mostly overlap with ours, but I don't have those links handy. Hello, Python 3.6 is the python3 in Fedora Rawhide, on track to become Fedora 26 in June. These packages fail to build (which, for properly packaged software, includes passing upstream tests where feasible): insight (failure unrelated to Python) python-django-admin-honeypot (not ported for django 1.10) python-os-brick (unrelated packaging problems) python-oslo-messaging (blocked by python-oslo-middleware) python-oslo-middleware (not ported for webob 1.7+) python-oslo-versionedobjects (blocked by python3-oslo-messaging) python-recommonmark (not ported to latest commonmark) python-repoze-who-plugins-sa (test failures) rb_libtorrent (probably unrelated to Python) shogun (probably unrelated to Python) That's 10 of 3349 python3-compatible packages. The list used to be much longer and contain much more important packages, but got to the current state thanks to individual maintainers and people like Adam Williamson, Zbigniew J?drzejewski-Szmek, and Igor Gnatenko (and the python-maint team, but we're paid for it). The list was also longer than usual because there was no mass rebuild of packages for Fedora 25, so some build failures introduced in the last year are only popping up now. Released versions of Fedora will be getting getting a "python36" package soon; for now it's installable using: dnf install --enablerepo=updates-testing python36 This is one of Fedora's "other Pythons" (interpreters other than the "current" 2.x and 3.x, i.e. python26, python34, python33). It's enough to make virtualenv/venv and tox work, but no system packages are built for these. From jcea at jcea.es Wed Feb 8 09:14:36 2017 From: jcea at jcea.es (Jesus Cea) Date: Wed, 8 Feb 2017 15:14:36 +0100 Subject: [Python-Dev] OpenIndiana and Solaris support In-Reply-To: References: Message-ID: On 08/02/17 11:24, Victor Stinner wrote: > So I suggest to drop official Solaris support, but I don't propose to > remove the C code specific to Solaris. In practice, I suggest to > remove Solaris and OpenIndiana buildbots since they are broken for > months and are more annoying than useful. Give me a week to move this forward. Last hope. -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From jcea at jcea.es Wed Feb 8 10:18:57 2017 From: jcea at jcea.es (Jesus Cea) Date: Wed, 8 Feb 2017 16:18:57 +0100 Subject: [Python-Dev] OpenIndiana and Solaris support In-Reply-To: References: Message-ID: <5528259d-fe6e-9a9a-bc11-f0a539b72e82@jcea.es> On 08/02/17 11:24, Victor Stinner wrote: > So I suggest to drop official Solaris support, but I don't propose to > remove the C code specific to Solaris. In practice, I suggest to > remove Solaris and OpenIndiana buildbots since they are broken for > months and are more annoying than useful. The main issue is that something wrong is going on with buildbot. From time to time, it eats gigabytes of RAM (last time, 64GB) and kills the machine. maybe once per week. This is not very welcomed by my host and his reaction is to shutdown the solaris "zone" because we don't "behave" and this is impacting production system. I am trying to convince him to launch buildbot process tree with an "ulimit" to protect the machine. Lets see. Sorry. Thanks for your patience. -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From jcea at jcea.es Wed Feb 8 11:06:28 2017 From: jcea at jcea.es (Jesus Cea) Date: Wed, 8 Feb 2017 17:06:28 +0100 Subject: [Python-Dev] OpenIndiana and Solaris support In-Reply-To: <5528259d-fe6e-9a9a-bc11-f0a539b72e82@jcea.es> References: <5528259d-fe6e-9a9a-bc11-f0a539b72e82@jcea.es> Message-ID: On 08/02/17 16:18, Jesus Cea wrote: > I am trying to convince him to launch buildbot process tree with an > "ulimit" to protect the machine. Lets see. > > Sorry. Thanks for your patience. I am launching now the buildbot with a limit of 1GB *PER PROCESS*. At least, when the memory skyrockets it will die without impacting the rest of the machine. Unless it does a fork-bomb... Backlog is huge. That will be useful to stress the buildbot and, hopefuly, make it fails faster. -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From brett at python.org Wed Feb 8 12:15:51 2017 From: brett at python.org (Brett Cannon) Date: Wed, 08 Feb 2017 17:15:51 +0000 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: On Tue, 7 Feb 2017 at 23:38 Victor Stinner wrote: > 2017-02-08 8:35 GMT+01:00 Victor Stinner : > > I'm sure the patch author would appreciate it, but I don't think we > > need to require it as we have gone this long without it. > > I know that different people have different expectation on GitHub. I > would like to take the opportunity of migrating to Git to use the > "author" and "committer" fields. If the author is set to the real > author, the one who proposed the change on the bug tracker or someone > else, we will be able to compute statistics on most active > contributors to more easily detect them and promote them to core > developers. > > What do you think? > Personally I think this would be great for everyone to do. Do you want to propose a PR for the devguide to document what to include (I assume committer will automatically be filled with the core dev's info so that should only require authorship to be specified). > > I agree that pull requests avoid the issues of filling manually the > author field, but I expect that not everyone will use PR, especially > because we will still use our current bug tracker which accepts to > attach patch files. Ok to propose them to create a PR instead. > My expectation is there will be a transition period of dealing with patches on bpo which have submitters who don't want to bother making an equivalent PR on GitHub. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Feb 8 15:32:24 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 8 Feb 2017 15:32:24 -0500 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: On 2/8/2017 2:38 AM, Victor Stinner wrote: > I know that different people have different expectation on GitHub. I > would like to take the opportunity of migrating to Git to use the > "author" and "committer" fields. If the author is set to the real > author, the one who proposed the change on the bug tracker or someone > else, we will be able to compute statistics on most active > contributors to more easily detect them and promote them to core > developers. > > What do you think? Many patches have multiple authors. Does the 'author' field allow that? Often, the committer adds missing chunks or rewrites the work with various degrees of editing. Sometimes a read a patch for the idea and then start fresh with the actual code. An acknowledgement in the news entry can be written different ways to reflect the situation. -- Terry Jan Reedy From brett at python.org Wed Feb 8 15:45:57 2017 From: brett at python.org (Brett Cannon) Date: Wed, 08 Feb 2017 20:45:57 +0000 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: On Wed, 8 Feb 2017 at 12:33 Terry Reedy wrote: > On 2/8/2017 2:38 AM, Victor Stinner wrote: > > > I know that different people have different expectation on GitHub. I > > would like to take the opportunity of migrating to Git to use the > > "author" and "committer" fields. If the author is set to the real > > author, the one who proposed the change on the bug tracker or someone > > else, we will be able to compute statistics on most active > > contributors to more easily detect them and promote them to core > > developers. > > > > What do you think? > > Many patches have multiple authors. Does the 'author' field allow that? > No. > Often, the committer adds missing chunks or rewrites the work with > various degrees of editing. Sometimes a read a patch for the idea and > then start fresh with the actual code. An acknowledgement in the news > entry can be written different ways to reflect the situation. > Yep, I assume the Misc/NEWS entry, commit log, and Misc/ACKS all provide opportunities to overcome the shortcoming of a single author attribution. -Brett > > -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Feb 8 16:31:43 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 8 Feb 2017 22:31:43 +0100 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: 2017-02-08 21:32 GMT+01:00 Terry Reedy : > Many patches have multiple authors. Does the 'author' field allow that? > Often, the committer adds missing chunks or rewrites the work with various > degrees of editing. Sometimes a read a patch for the idea and then start > fresh with the actual code. An acknowledgement in the news entry can be > written different ways to reflect the situation. In my experience, the common case is two "authors": the initial patch autor, and the core dev who makes minor changes like filling Misc/NEWS and adjust some parts of the code. If changes are minor, there is no need to elaborate. But if changes are non straightforward, they can be listed in the commit messages. If a patch has more author, you have to pick the name, decide which author helped the most, and add the other authors in the commit message. Using GitHub, it will become easier to use patch series, and so smaller changes. Each change may have a different author :-D Victor From brett at python.org Wed Feb 8 17:42:44 2017 From: brett at python.org (Brett Cannon) Date: Wed, 08 Feb 2017 22:42:44 +0000 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: On Wed, 8 Feb 2017 at 13:33 Victor Stinner wrote: > 2017-02-08 21:32 GMT+01:00 Terry Reedy : > > Many patches have multiple authors. Does the 'author' field allow that? > > Often, the committer adds missing chunks or rewrites the work with > various > > degrees of editing. Sometimes a read a patch for the idea and then start > > fresh with the actual code. An acknowledgement in the news entry can be > > written different ways to reflect the situation. > > In my experience, the common case is two "authors": the initial patch > autor, and the core dev who makes minor changes like filling Misc/NEWS > and adjust some parts of the code. If changes are minor, there is no > need to elaborate. But if changes are non straightforward, they can be > listed in the commit messages. > > If a patch has more author, you have to pick the name, decide which > author helped the most, and add the other authors in the commit > message. > > Using GitHub, it will become easier to use patch series, and so > smaller changes. Each change may have a different author :-D > Don't forget we are doing squash merges, so if a PR has multiple authors then the author information won't carry forward in git (I don't know if GitHub has custom logic to work around this). -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Wed Feb 8 18:03:36 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 9 Feb 2017 00:03:36 +0100 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: 2017-02-08 23:42 GMT+01:00 Brett Cannon : > Don't forget we are doing squash merges, Ah, I didn't know. Why not using merges? Victor From srkunze at mail.de Wed Feb 8 18:18:48 2017 From: srkunze at mail.de (Sven R. Kunze) Date: Thu, 9 Feb 2017 00:18:48 +0100 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: <231600f9-7aa9-549b-37c9-5de453a9f2e9@mail.de> On 09.02.2017 00:03, Victor Stinner wrote: > 2017-02-08 23:42 GMT+01:00 Brett Cannon : >> Don't forget we are doing squash merges, > Ah, I didn't know. Why not using merges? > Same question here. I see no benefit just overhead, mistakes and longer processes. Sven From brett at python.org Wed Feb 8 18:53:08 2017 From: brett at python.org (Brett Cannon) Date: Wed, 08 Feb 2017 23:53:08 +0000 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: On Wed, 8 Feb 2017 at 15:04 Victor Stinner wrote: > 2017-02-08 23:42 GMT+01:00 Brett Cannon : > > Don't forget we are doing squash merges, > > Ah, I didn't know. Why not using merges? > Because other core devs wanted a linear history. This preference was very strong to the point people were willing to forgo the Merge button in GitHub's web UI to enforce it until GitHub added the squash merge support for the Merge button. This was decided over a year ago and documented in PEP 512 as the decision made since I believe the beginning of that PEP. Now I know Victor was asking out of curiosity, but I'm going to ask nicely now and then ignore later anyone who starts second-guessing my decisions at this point as someone did as a follow-up to Victor's question. This process has been discussed for over 2 years and PEP 512 has existed for over one year. There has also been an open mailing list where I have held discussions on various topics and people have been free to ask and participate on. Now is not the time to start second-guessing things that have already been decided and discussed to great length before we even have any experience with the chosen workflow. The final stretch of this whole process has been going smoothly, so I'm trying to ask nicely that everyone give me the benefit of the doubt and assume everything has been thought out and there is a reason behind everything before you choose to second-guess my decisions at the 11th hour. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcea at jcea.es Wed Feb 8 22:17:21 2017 From: jcea at jcea.es (Jesus Cea) Date: Thu, 9 Feb 2017 04:17:21 +0100 Subject: [Python-Dev] OpenIndiana and Solaris support In-Reply-To: References: <5528259d-fe6e-9a9a-bc11-f0a539b72e82@jcea.es> Message-ID: <9d6cbf84-30da-9077-b518-5bd52b84138e@jcea.es> On 08/02/17 17:06, Jesus Cea wrote: > On 08/02/17 16:18, Jesus Cea wrote: >> I am trying to convince him to launch buildbot process tree with an >> "ulimit" to protect the machine. Lets see. >> >> Sorry. Thanks for your patience. > > I am launching now the buildbot with a limit of 1GB *PER PROCESS*. At > least, when the memory skyrockets it will die without impacting the rest > of the machine. Unless it does a fork-bomb... > > Backlog is huge. That will be useful to stress the buildbot and, > hopefuly, make it fails faster. Checking the mailing list archives, I am seen Openindiana buildbot memory issues since 2011, at least. . It was never triaged. I am getting mercurial errors because some incompatibility with current release . Since we are moving to github in a couple of days, I will wait until that time to keep pursuing this. -- Jes?s Cea Avi?n _/_/ _/_/_/ _/_/_/ jcea at jcea.es - http://www.jcea.es/ _/_/ _/_/ _/_/ _/_/ _/_/ Twitter: @jcea _/_/ _/_/ _/_/_/_/_/ jabber / xmpp:jcea at jabber.org _/_/ _/_/ _/_/ _/_/ _/_/ "Things are not so easy" _/_/ _/_/ _/_/ _/_/ _/_/ _/_/ "My name is Dump, Core Dump" _/_/_/ _/_/_/ _/_/ _/_/ "El amor es poner tu felicidad en la felicidad de otro" - Leibniz -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 473 bytes Desc: OpenPGP digital signature URL: From soltysh at gmail.com Thu Feb 9 05:18:01 2017 From: soltysh at gmail.com (Maciej Szulik) Date: Thu, 9 Feb 2017 11:18:01 +0100 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: On Wed, Feb 8, 2017 at 8:52 AM, Senthil Kumaran wrote: > On Tue, Feb 7, 2017 at 11:38 PM, Victor Stinner > wrote: > > I know that different people have different expectation on GitHub. I > > would like to take the opportunity of migrating to Git to use the > > "author" and "committer" fields. If the author is set to the real > > author, the one who proposed the change on the bug tracker or someone > > else, we will be able to compute statistics on most active > > contributors to more easily detect them and promote them to core > > developers. > > > > What do you think? > > I am +1 to this idea. The intention behind this idea is also good one. > > * When the patches come from Github PRs, the contribution authors are > automatically tracked. The comitter would be merging the changes from > the authors. > > * When contribution comes via Patches/ or for many existing patches, > setting the author is a good idea. > > If we enable auto-PR creation, which was done by Anish as part of his GSoC we might be able to set the original author based on the github username associated with his user on bpo. But I'm guessing we need more work in this area ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ihor at kalnytskyi.com Thu Feb 9 03:34:14 2017 From: ihor at kalnytskyi.com (Ihor Kalnytskyi) Date: Thu, 9 Feb 2017 10:34:14 +0200 Subject: [Python-Dev] GitHub migration scheduled for Friday In-Reply-To: References: Message-ID: Brett Cannon wrote: > Because other core devs wanted a linear history. This preference was very > strong to the point people were willing to forgo the Merge button in > GitHub's web UI to enforce it until GitHub added the squash merge support > for the Merge button. Actually, there's a third option - using *rebase and merge* [1] button. It does not produce a merge commit but preserves commits from the pull request. [1] https://help.github.com/articles/about-pull-request-merges/#rebase-and-merge-your-pull-request-commits On Thu, Feb 9, 2017 at 1:53 AM, Brett Cannon wrote: > > > On Wed, 8 Feb 2017 at 15:04 Victor Stinner wrote: >> >> 2017-02-08 23:42 GMT+01:00 Brett Cannon : >> > Don't forget we are doing squash merges, >> >> Ah, I didn't know. Why not using merges? > > > Because other core devs wanted a linear history. This preference was very > strong to the point people were willing to forgo the Merge button in > GitHub's web UI to enforce it until GitHub added the squash merge support > for the Merge button. This was decided over a year ago and documented in PEP > 512 as the decision made since I believe the beginning of that PEP. > > Now I know Victor was asking out of curiosity, but I'm going to ask nicely > now and then ignore later anyone who starts second-guessing my decisions at > this point as someone did as a follow-up to Victor's question. This process > has been discussed for over 2 years and PEP 512 has existed for over one > year. There has also been an open mailing list where I have held discussions > on various topics and people have been free to ask and participate on. Now > is not the time to start second-guessing things that have already been > decided and discussed to great length before we even have any experience > with the chosen workflow. > > The final stretch of this whole process has been going smoothly, so I'm > trying to ask nicely that everyone give me the benefit of the doubt and > assume everything has been thought out and there is a reason behind > everything before you choose to second-guess my decisions at the 11th hour. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ihor%40kalnytskyi.com > From freddyrietdijk at fridh.nl Thu Feb 9 04:27:29 2017 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Thu, 9 Feb 2017 10:27:29 +0100 Subject: [Python-Dev] Deterministic builds of the interpreter Message-ID: Hi, I'm attempting to make the builds of the Python interpreters for Nixpkgs [1] deterministic. In the case of Python 2.7 we have a patch [2] that fixes the timestamp used in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We also remove `wininst*.exe`. This works fine, although there are 4 small issues left [3]. Do you have any idea what is going on in these files that could make them indeterministic? For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and modified `py_compile` to use `0` instead of `source_stats['mtime']`. The builds are not yet deterministic [4]. Any suggestions what could be fixed here? Kind regards, Freddy [1] https://github.com/NixOS/nixpkgs [2] https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/development/interpreters/python/cpython/2.7/deterministic-build.patch [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment-278474082 [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e6b -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Thu Feb 9 12:04:45 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 10 Feb 2017 02:04:45 +0900 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: As reading [4], mtime is not 0. data = bytearray(MAGIC_NUMBER) data.extend(_w_long(mtime)) data.extend(_w_long(source_size)) data.extend(marshal.dumps(code)) First 4 bytes are magic. Next 4 bytes are mtime. ? ? ? ? -00000000: 160d 0d0a 6b2e 9c58 6c21 0000 e300 0000 ....k..Xl!...... ? ? ? ? +00000000: 160d 0d0a e631 9c58 6c21 0000 e300 0000 .....1.Xl!...... mtime is 6b2e9c58 vs e6319c53 (little endian) maybe, you failed to use customized py_compile when building? On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk wrote: > Hi, > > I'm attempting to make the builds of the Python interpreters for Nixpkgs [1] > deterministic. > > In the case of Python 2.7 we have a patch [2] that fixes the timestamp used > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We also > remove `wininst*.exe`. This works fine, although there are 4 small issues > left [3]. Do you have any idea what is going on in these files that could > make them indeterministic? > > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and modified > `py_compile` to use `0` instead of `source_stats['mtime']`. The builds are > not yet deterministic [4]. Any suggestions what could be fixed here? > > Kind regards, > > Freddy > > > [1] https://github.com/NixOS/nixpkgs > [2] > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/development/interpreters/python/cpython/2.7/deterministic-build.patch > [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment-278474082 > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e6b > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com > From freddyrietdijk at fridh.nl Thu Feb 9 12:45:12 2017 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Thu, 9 Feb 2017 18:45:12 +0100 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: Correct, that was an older version from before I patched `_bootstrap_external.py`. A more recent diff can be found at https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa These all seem to be sets. On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki wrote: > As reading [4], mtime is not 0. > > data = bytearray(MAGIC_NUMBER) > data.extend(_w_long(mtime)) > data.extend(_w_long(source_size)) > data.extend(marshal.dumps(code)) > > First 4 bytes are magic. > Next 4 bytes are mtime. > > ? ? ? ? -00000000: 160d 0d0a 6b2e 9c58 6c21 0000 e300 0000 ....k..Xl!...... > ? ? ? ? +00000000: 160d 0d0a e631 9c58 6c21 0000 e300 0000 .....1.Xl!...... > > mtime is 6b2e9c58 vs e6319c53 (little endian) > > maybe, you failed to use customized py_compile when building? > > > On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk > wrote: > > Hi, > > > > I'm attempting to make the builds of the Python interpreters for Nixpkgs > [1] > > deterministic. > > > > In the case of Python 2.7 we have a patch [2] that fixes the timestamp > used > > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We also > > remove `wininst*.exe`. This works fine, although there are 4 small issues > > left [3]. Do you have any idea what is going on in these files that could > > make them indeterministic? > > > > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and modified > > `py_compile` to use `0` instead of `source_stats['mtime']`. The builds > are > > not yet deterministic [4]. Any suggestions what could be fixed here? > > > > Kind regards, > > > > Freddy > > > > > > [1] https://github.com/NixOS/nixpkgs > > [2] > > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/ > development/interpreters/python/cpython/2.7/deterministic-build.patch > > [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment-278474082 > > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e6b > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/ > songofacandy%40gmail.com > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Thu Feb 9 12:51:42 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 10 Feb 2017 02:51:42 +0900 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk wrote: > Correct, that was an older version from before I patched > `_bootstrap_external.py`. A more recent diff can be found at > > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa > > These all seem to be sets. Maybe, PYTHONHASHSEED help you. https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED > > On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki wrote: >> >> As reading [4], mtime is not 0. >> >> data = bytearray(MAGIC_NUMBER) >> data.extend(_w_long(mtime)) >> data.extend(_w_long(source_size)) >> data.extend(marshal.dumps(code)) >> >> First 4 bytes are magic. >> Next 4 bytes are mtime. >> >> ? ? ? ? -00000000: 160d 0d0a 6b2e 9c58 6c21 0000 e300 0000 >> ....k..Xl!...... >> ? ? ? ? +00000000: 160d 0d0a e631 9c58 6c21 0000 e300 0000 >> .....1.Xl!...... >> >> mtime is 6b2e9c58 vs e6319c53 (little endian) >> >> maybe, you failed to use customized py_compile when building? >> >> >> On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk >> wrote: >> > Hi, >> > >> > I'm attempting to make the builds of the Python interpreters for Nixpkgs >> > [1] >> > deterministic. >> > >> > In the case of Python 2.7 we have a patch [2] that fixes the timestamp >> > used >> > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We also >> > remove `wininst*.exe`. This works fine, although there are 4 small >> > issues >> > left [3]. Do you have any idea what is going on in these files that >> > could >> > make them indeterministic? >> > >> > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and >> > modified >> > `py_compile` to use `0` instead of `source_stats['mtime']`. The builds >> > are >> > not yet deterministic [4]. Any suggestions what could be fixed here? >> > >> > Kind regards, >> > >> > Freddy >> > >> > >> > [1] https://github.com/NixOS/nixpkgs >> > [2] >> > >> > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/development/interpreters/python/cpython/2.7/deterministic-build.patch >> > [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment-278474082 >> > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e6b >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> > >> > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com >> > > > From hugo.prodhomme at gmx.fr Thu Feb 9 13:18:34 2017 From: hugo.prodhomme at gmx.fr (Hugo PROD HOMME) Date: Thu, 9 Feb 2017 19:18:34 +0100 Subject: [Python-Dev] "format" builtin function, docstring improvement Message-ID: An HTML attachment was scrubbed... URL: -------------- next part -------------- An embedded and charset-unspecified text was scrubbed... Name: format_docstring.txt URL: From alexander.belopolsky at gmail.com Thu Feb 9 16:21:43 2017 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 9 Feb 2017 16:21:43 -0500 Subject: [Python-Dev] "format" builtin function, docstring improvement In-Reply-To: References: Message-ID: On Thu, Feb 9, 2017 at 1:18 PM, Hugo PROD HOMME wrote: > > I have made a template of the docstring to begin with, that I send as an attached file, but I am not familiar with docstring format. Please open an issue at bugs.python.org. We did something similar in issue #9650, [1] so I think this will be well received. Python docstrings are in plain text, but you need to consult PEP 436 (The Argument Clinic DSL) [2] for details. The format() docstring is specified in Python/bltinmodule.c. [1]: http://bugs.python.org/issue9650 [2]: https://www.python.org/dev/peps/pep-0436/#function-docstring -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Fri Feb 10 06:03:03 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 10 Feb 2017 20:03:03 +0900 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: On Fri, Feb 10, 2017 at 7:58 PM, Freddy Rietdijk wrote: > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > still seem indeterministic. > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` > where $out is the path where I installed Python. > > Do you have an idea why in [3], this is Python 2.7, the timestamps are still > incorrect? I think they're all required for `compileall` and somehow it > doesn't seem capable of taking into account DETERMINISTIC_BUILD. Explicitly > removing those pyc and pyo files and recompiling them to bytecode still > results in timestamp issues for these 4 files. Sorry, I have no motivation about Python 2 anymore. From freddyrietdijk at fridh.nl Fri Feb 10 05:58:34 2017 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Fri, 10 Feb 2017 11:58:34 +0100 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items still seem indeterministic. To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` where $out is the path where I installed Python. Do you have an idea why in [3], this is Python 2.7, the timestamps are still incorrect? I think they're all required for `compileall` and somehow it doesn't seem capable of taking into account DETERMINISTIC_BUILD. Explicitly removing those pyc and pyo files and recompiling them to bytecode still results in timestamp issues for these 4 files. On Thu, Feb 9, 2017 at 6:51 PM, INADA Naoki wrote: > On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk > wrote: > > Correct, that was an older version from before I patched > > `_bootstrap_external.py`. A more recent diff can be found at > > > > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa > > > > These all seem to be sets. > > Maybe, PYTHONHASHSEED help you. > https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED > > > > > On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki > wrote: > >> > >> As reading [4], mtime is not 0. > >> > >> data = bytearray(MAGIC_NUMBER) > >> data.extend(_w_long(mtime)) > >> data.extend(_w_long(source_size)) > >> data.extend(marshal.dumps(code)) > >> > >> First 4 bytes are magic. > >> Next 4 bytes are mtime. > >> > >> ? ? ? ? -00000000: 160d 0d0a 6b2e 9c58 6c21 0000 e300 0000 > >> ....k..Xl!...... > >> ? ? ? ? +00000000: 160d 0d0a e631 9c58 6c21 0000 e300 0000 > >> .....1.Xl!...... > >> > >> mtime is 6b2e9c58 vs e6319c53 (little endian) > >> > >> maybe, you failed to use customized py_compile when building? > >> > >> > >> On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk < > freddyrietdijk at fridh.nl> > >> wrote: > >> > Hi, > >> > > >> > I'm attempting to make the builds of the Python interpreters for > Nixpkgs > >> > [1] > >> > deterministic. > >> > > >> > In the case of Python 2.7 we have a patch [2] that fixes the timestamp > >> > used > >> > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We > also > >> > remove `wininst*.exe`. This works fine, although there are 4 small > >> > issues > >> > left [3]. Do you have any idea what is going on in these files that > >> > could > >> > make them indeterministic? > >> > > >> > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and > >> > modified > >> > `py_compile` to use `0` instead of `source_stats['mtime']`. The builds > >> > are > >> > not yet deterministic [4]. Any suggestions what could be fixed here? > >> > > >> > Kind regards, > >> > > >> > Freddy > >> > > >> > > >> > [1] https://github.com/NixOS/nixpkgs > >> > [2] > >> > > >> > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/ > development/interpreters/python/cpython/2.7/deterministic-build.patch > >> > [3] https://github.com/NixOS/nixpkgs/issues/22570# > issuecomment-278474082 > >> > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d110f0e > 6b > >> > > >> > _______________________________________________ > >> > Python-Dev mailing list > >> > Python-Dev at python.org > >> > https://mail.python.org/mailman/listinfo/python-dev > >> > Unsubscribe: > >> > > >> > https://mail.python.org/mailman/options/python-dev/ > songofacandy%40gmail.com > >> > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From freddyrietdijk at fridh.nl Fri Feb 10 05:59:25 2017 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Fri, 10 Feb 2017 11:59:25 +0100 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: That should have been `PYTHONHASHSEED=0 $out/bin/python -m compileall -f $out`. On Fri, Feb 10, 2017 at 11:58 AM, Freddy Rietdijk wrote: > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > still seem indeterministic. > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f $out` > where $out is the path where I installed Python. > > Do you have an idea why in [3], this is Python 2.7, the timestamps are > still incorrect? I think they're all required for `compileall` and somehow > it doesn't seem capable of taking into account DETERMINISTIC_BUILD. > Explicitly removing those pyc and pyo files and recompiling them to > bytecode still results in timestamp issues for these 4 files. > > On Thu, Feb 9, 2017 at 6:51 PM, INADA Naoki > wrote: > >> On Fri, Feb 10, 2017 at 2:45 AM, Freddy Rietdijk >> wrote: >> > Correct, that was an older version from before I patched >> > `_bootstrap_external.py`. A more recent diff can be found at >> > >> > https://gist.github.com/anonymous/d40f24fd6b636ba40d345ff3f12a0aaa >> > >> > These all seem to be sets. >> >> Maybe, PYTHONHASHSEED help you. >> https://docs.python.org/3/using/cmdline.html#envvar-PYTHONHASHSEED >> >> > >> > On Thu, Feb 9, 2017 at 6:04 PM, INADA Naoki >> wrote: >> >> >> >> As reading [4], mtime is not 0. >> >> >> >> data = bytearray(MAGIC_NUMBER) >> >> data.extend(_w_long(mtime)) >> >> data.extend(_w_long(source_size)) >> >> data.extend(marshal.dumps(code)) >> >> >> >> First 4 bytes are magic. >> >> Next 4 bytes are mtime. >> >> >> >> ? ? ? ? -00000000: 160d 0d0a 6b2e 9c58 6c21 0000 e300 0000 >> >> ....k..Xl!...... >> >> ? ? ? ? +00000000: 160d 0d0a e631 9c58 6c21 0000 e300 0000 >> >> .....1.Xl!...... >> >> >> >> mtime is 6b2e9c58 vs e6319c53 (little endian) >> >> >> >> maybe, you failed to use customized py_compile when building? >> >> >> >> >> >> On Thu, Feb 9, 2017 at 6:27 PM, Freddy Rietdijk < >> freddyrietdijk at fridh.nl> >> >> wrote: >> >> > Hi, >> >> > >> >> > I'm attempting to make the builds of the Python interpreters for >> Nixpkgs >> >> > [1] >> >> > deterministic. >> >> > >> >> > In the case of Python 2.7 we have a patch [2] that fixes the >> timestamp >> >> > used >> >> > in .pyc files in case the env var `DETERMINISTIC_BUILD` is set. We >> also >> >> > remove `wininst*.exe`. This works fine, although there are 4 small >> >> > issues >> >> > left [3]. Do you have any idea what is going on in these files that >> >> > could >> >> > make them indeterministic? >> >> > >> >> > For Python 3.x I disabled ensurepip, removed `wininst*.exe`, and >> >> > modified >> >> > `py_compile` to use `0` instead of `source_stats['mtime']`. The >> builds >> >> > are >> >> > not yet deterministic [4]. Any suggestions what could be fixed here? >> >> > >> >> > Kind regards, >> >> > >> >> > Freddy >> >> > >> >> > >> >> > [1] https://github.com/NixOS/nixpkgs >> >> > [2] >> >> > >> >> > https://github.com/NixOS/nixpkgs/blob/1da6775/pkgs/developme >> nt/interpreters/python/cpython/2.7/deterministic-build.patch >> >> > [3] https://github.com/NixOS/nixpkgs/issues/22570#issuecomment- >> 278474082 >> >> > [4] https://gist.github.com/anonymous/7cc147af6511dee2dc5a5b8d11 >> 0f0e6b >> >> > >> >> > _______________________________________________ >> >> > Python-Dev mailing list >> >> > Python-Dev at python.org >> >> > https://mail.python.org/mailman/listinfo/python-dev >> >> > Unsubscribe: >> >> > >> >> > https://mail.python.org/mailman/options/python-dev/songofaca >> ndy%40gmail.com >> >> > >> > >> > >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Feb 10 12:09:00 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 10 Feb 2017 18:09:00 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170210170900.5FD38562D7@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-02-03 - 2017-02-10) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5799 (+30) closed 35483 (+60) total 41282 (+90) Open issues with patches: 2489 Issues opened (53) ================== #29430: zipfile: invalid link http://bugs.python.org/issue29430 reopened by berker.peksag #29438: use after free in key sharing dict http://bugs.python.org/issue29438 opened by audric #29440: _dbm requires -lgdbm if gdbm is built as static libraries http://bugs.python.org/issue29440 opened by Chi Hsuan Yen #29442: Use argparse and drop dirty optparse hacks in setup.py http://bugs.python.org/issue29442 opened by Chi Hsuan Yen #29446: Improve tkinter 'import *' situation http://bugs.python.org/issue29446 opened by terry.reedy #29447: Add os.PathLike support to the tempfile module http://bugs.python.org/issue29447 opened by brett.cannon #29448: Implement os.Pathlike for importlib.machinery http://bugs.python.org/issue29448 opened by brett.cannon #29450: xattr functions aren't in os.supports_fd, os.supports_follow_s http://bugs.python.org/issue29450 opened by Omar Sandoval #29451: Use _PyArg_Parser for _PyArg_ParseStack(): support positional http://bugs.python.org/issue29451 opened by haypo #29453: Remove reference to undefined dictionary ordering in Tutorial http://bugs.python.org/issue29453 opened by Jim Fasarakis-Hilliard #29454: Shutting down consumer on a remote queue http://bugs.python.org/issue29454 opened by Semi #29455: Mention coverage.py in trace module documentation http://bugs.python.org/issue29455 opened by brett.cannon #29456: bug in unicodedata.normalize: u1176 http://bugs.python.org/issue29456 opened by pusnow #29459: `__contains__` and `get` methods for match objects? http://bugs.python.org/issue29459 opened by irdb #29461: Experiment usage of likely/unlikely in CPython core http://bugs.python.org/issue29461 opened by haypo #29463: Add `docstring` attribute to AST nodes http://bugs.python.org/issue29463 opened by inada.naoki #29464: Specialize FASTCALL for functions with positional-only paramet http://bugs.python.org/issue29464 opened by serhiy.storchaka #29465: Add _PyObject_FastCall() to reduce stack consumption http://bugs.python.org/issue29465 opened by haypo #29466: pickle does not serialize Exception __cause__ field http://bugs.python.org/issue29466 opened by diekhans #29468: zipfile should catch ValueError as well as OSError to detect b http://bugs.python.org/issue29468 opened by llllllllll #29469: AST-level Constant folding http://bugs.python.org/issue29469 opened by inada.naoki #29470: [python] Error in `/usr/bin/python': free(): invalid size: 0x0 http://bugs.python.org/issue29470 opened by David Ford (FirefighterBlu3) #29471: AST: add an attribute to FunctionDef to distinguish functions http://bugs.python.org/issue29471 opened by haypo #29472: subprocess.communicate with timeout 0 and already terminated p http://bugs.python.org/issue29472 opened by Griffon26 #29473: subprocces check_output http://bugs.python.org/issue29473 opened by ?????????? ???????? #29474: Grammatical errors in weakref.WeakValueDictionary docs http://bugs.python.org/issue29474 opened by Mariatta #29475: option to not follow symlinks when globbing http://bugs.python.org/issue29475 opened by o11c #29476: Simplify set_add_entry() http://bugs.python.org/issue29476 opened by rhettinger #29477: Lambda with complex arguments is ctx STORE http://bugs.python.org/issue29477 opened by malthe #29478: email.policy.Compat32(max_line_length=None) not as documented http://bugs.python.org/issue29478 opened by martin.panter #29479: httplib: could not skip "ACCEPT-ENCODING" header http://bugs.python.org/issue29479 opened by song1st #29480: Mac OSX Installer SSL Roots http://bugs.python.org/issue29480 opened by rahvee #29481: 3.6.0 doc describes 3.6.1 feature - typing.Deque http://bugs.python.org/issue29481 opened by Guy Arad #29502: Should PyObject_Call() call the profiler on C functions, use C http://bugs.python.org/issue29502 opened by haypo #29503: Make embedded-Python detectable http://bugs.python.org/issue29503 opened by Andi Bergmeier #29504: blake2: compile error with -march=bdver2 http://bugs.python.org/issue29504 opened by floppymaster #29505: Submit the re, json, & csv modules to oss-fuzz testing http://bugs.python.org/issue29505 opened by gregory.p.smith #29506: Incorrect documentation for the copy module http://bugs.python.org/issue29506 opened by pgacv2 #29507: Use FASTCALL in call_method() to avoid temporary tuple http://bugs.python.org/issue29507 opened by haypo #29509: redundant interning in PyObject_GetAttrString http://bugs.python.org/issue29509 opened by inada.naoki #29511: Add 'find' as build-in method for lists http://bugs.python.org/issue29511 opened by george-shuklin #29512: regrtest refleak: implement bisection feature http://bugs.python.org/issue29512 opened by haypo #29514: Add a test case that prevents magic number changes in minor re http://bugs.python.org/issue29514 opened by ncoghlan #29515: socket module missing IPPROTO_IPV6, IPPROTO_IPV4 on Windows http://bugs.python.org/issue29515 opened by mhils #29516: shutil.copytree(symlinks=True) fails when copying symlinks cro http://bugs.python.org/issue29516 opened by poljakowski #29517: "Can't pickle local object" when uses functools.partial with m http://bugs.python.org/issue29517 opened by DAVID ALEJANDRO Pineda #29518: 'for' loop not automatically breaking (index error on line of http://bugs.python.org/issue29518 opened by Justin McNiel #29520: Documentation uses deprecated "defindex.html" Sphinx template http://bugs.python.org/issue29520 opened by JDLH #29521: Minor warning messages when compiling documentation http://bugs.python.org/issue29521 opened by JDLH #29523: latest setuptools breaks virtualenvs due to unbundling depende http://bugs.python.org/issue29523 opened by cstratak #29524: Move functions to call objects into a new Objects/call.c file http://bugs.python.org/issue29524 opened by haypo #29525: Python 2.7.13 for Windows broken (from prompt) http://bugs.python.org/issue29525 opened by Frak N #29526: Documenting format() function http://bugs.python.org/issue29526 opened by hugo.prodhomme at gmx.fr Most recent 15 issues with no replies (15) ========================================== #29517: "Can't pickle local object" when uses functools.partial with m http://bugs.python.org/issue29517 #29516: shutil.copytree(symlinks=True) fails when copying symlinks cro http://bugs.python.org/issue29516 #29512: regrtest refleak: implement bisection feature http://bugs.python.org/issue29512 #29509: redundant interning in PyObject_GetAttrString http://bugs.python.org/issue29509 #29505: Submit the re, json, & csv modules to oss-fuzz testing http://bugs.python.org/issue29505 #29481: 3.6.0 doc describes 3.6.1 feature - typing.Deque http://bugs.python.org/issue29481 #29472: subprocess.communicate with timeout 0 and already terminated p http://bugs.python.org/issue29472 #29468: zipfile should catch ValueError as well as OSError to detect b http://bugs.python.org/issue29468 #29454: Shutting down consumer on a remote queue http://bugs.python.org/issue29454 #29450: xattr functions aren't in os.supports_fd, os.supports_follow_s http://bugs.python.org/issue29450 #29448: Implement os.Pathlike for importlib.machinery http://bugs.python.org/issue29448 #29447: Add os.PathLike support to the tempfile module http://bugs.python.org/issue29447 #29435: Allow to pass fileobj to is_tarfile http://bugs.python.org/issue29435 #29427: Option to skip padding for base64 urlsafe encoding/decoding http://bugs.python.org/issue29427 #29422: add socket independent timeout to httplib/http.client read http://bugs.python.org/issue29422 Most recent 15 issues waiting for review (15) ============================================= #29524: Move functions to call objects into a new Objects/call.c file http://bugs.python.org/issue29524 #29523: latest setuptools breaks virtualenvs due to unbundling depende http://bugs.python.org/issue29523 #29509: redundant interning in PyObject_GetAttrString http://bugs.python.org/issue29509 #29507: Use FASTCALL in call_method() to avoid temporary tuple http://bugs.python.org/issue29507 #29476: Simplify set_add_entry() http://bugs.python.org/issue29476 #29475: option to not follow symlinks when globbing http://bugs.python.org/issue29475 #29474: Grammatical errors in weakref.WeakValueDictionary docs http://bugs.python.org/issue29474 #29469: AST-level Constant folding http://bugs.python.org/issue29469 #29465: Add _PyObject_FastCall() to reduce stack consumption http://bugs.python.org/issue29465 #29464: Specialize FASTCALL for functions with positional-only paramet http://bugs.python.org/issue29464 #29463: Add `docstring` attribute to AST nodes http://bugs.python.org/issue29463 #29461: Experiment usage of likely/unlikely in CPython core http://bugs.python.org/issue29461 #29456: bug in unicodedata.normalize: u1176 http://bugs.python.org/issue29456 #29455: Mention coverage.py in trace module documentation http://bugs.python.org/issue29455 #29453: Remove reference to undefined dictionary ordering in Tutorial http://bugs.python.org/issue29453 Top 10 most discussed issues (10) ================================= #29507: Use FASTCALL in call_method() to avoid temporary tuple http://bugs.python.org/issue29507 28 msgs #29438: use after free in key sharing dict http://bugs.python.org/issue29438 18 msgs #29465: Add _PyObject_FastCall() to reduce stack consumption http://bugs.python.org/issue29465 15 msgs #29453: Remove reference to undefined dictionary ordering in Tutorial http://bugs.python.org/issue29453 13 msgs #29464: Specialize FASTCALL for functions with positional-only paramet http://bugs.python.org/issue29464 12 msgs #29463: Add `docstring` attribute to AST nodes http://bugs.python.org/issue29463 11 msgs #29503: Make embedded-Python detectable http://bugs.python.org/issue29503 11 msgs #29476: Simplify set_add_entry() http://bugs.python.org/issue29476 10 msgs #28686: py.exe ignored PATH when using python3 shebang http://bugs.python.org/issue28686 9 msgs #29306: Check usage of Py_EnterRecursiveCall() and Py_LeaveRecursiveCa http://bugs.python.org/issue29306 9 msgs Issues closed (57) ================== #13196: subprocess: undocumented if shell=True is necessary to find ex http://bugs.python.org/issue13196 closed by martin.panter #19240: iglob should try to use `readdir` http://bugs.python.org/issue19240 closed by serhiy.storchaka #20927: Different behaviour on Posix and Windows when using subprocess http://bugs.python.org/issue20927 closed by martin.panter #26355: Emit major version based canonical URLs for docs http://bugs.python.org/issue26355 closed by ncoghlan #27659: Prohibit implicit C function declarations http://bugs.python.org/issue27659 closed by Chi Hsuan Yen #28489: Fix comment in tokenizer.c http://bugs.python.org/issue28489 closed by berker.peksag #28870: Reduce stack consumption of PyObject_CallFunctionObjArgs() and http://bugs.python.org/issue28870 closed by haypo #29028: Use-After-Free in PyString_FromStringAndSize() of stringobject http://bugs.python.org/issue29028 closed by serhiy.storchaka #29100: datetime.fromtimestamp() doesn't check min/max year anymore: r http://bugs.python.org/issue29100 closed by haypo #29147: registry value to be cleared when python is uninstalled http://bugs.python.org/issue29147 closed by steve.dower #29198: AsyncGenerator is missing from typing http://bugs.python.org/issue29198 closed by berker.peksag #29314: asyncio.async deprecation warning is missing stacklevel=2 http://bugs.python.org/issue29314 closed by Mariatta #29326: Blank lines in ._pth file are not ignored http://bugs.python.org/issue29326 closed by ammar2 #29351: absolute imports for logging http://bugs.python.org/issue29351 closed by vinay.sajip #29362: regrtest: don't fail immediately if a child does crash http://bugs.python.org/issue29362 closed by haypo #29371: Typo in doctest documentation http://bugs.python.org/issue29371 closed by Mariatta #29372: RotatingFileHandler rotates empty logfile if it emits a large http://bugs.python.org/issue29372 closed by vinay.sajip #29395: Edit with IDLE 3.6 don??t work if Name of File "code.py" http://bugs.python.org/issue29395 closed by terry.reedy #29405: improve csv.Sniffer().sniff() behavior http://bugs.python.org/issue29405 closed by xiang.zhang #29437: installation not listed for all users http://bugs.python.org/issue29437 closed by steve.dower #29439: _decimal on Android requires linking with libm http://bugs.python.org/issue29439 closed by skrah #29441: Update examples to use async and await keywords in asyncio-tas http://bugs.python.org/issue29441 closed by berker.peksag #29443: Re-running Windows installer should have option to set PATH http://bugs.python.org/issue29443 closed by steve.dower #29444: Out-of-bounds buffer access in match_getslice_by_index http://bugs.python.org/issue29444 closed by serhiy.storchaka #29445: http.client: missing response headers when malformed header is http://bugs.python.org/issue29445 closed by r.david.murray #29449: clear() should return prior state in threading.Event http://bugs.python.org/issue29449 closed by jmoy #29452: Use FASTCALL for collections.deque methods: index, insert, rot http://bugs.python.org/issue29452 closed by haypo #29457: strftime('%x') does not use my locale http://bugs.python.org/issue29457 closed by eryksun #29458: random.seed version=1 behavior http://bugs.python.org/issue29458 closed by rhettinger #29460: Speed up _PyArg_NoKeywords() and like http://bugs.python.org/issue29460 closed by serhiy.storchaka #29462: RFC822-comments in email header fields can fool, e.g., get_fil http://bugs.python.org/issue29462 closed by ale2017 #29467: Allow return mismatch to be caught by function http://bugs.python.org/issue29467 closed by eric.smith #29482: AddressSanitizer: attempting double-free on 0x60b000007050 http://bugs.python.org/issue29482 closed by matrixise #29483: AddressSanitizer: heap-buffer-overflow on address 0x60200000e7 http://bugs.python.org/issue29483 closed by matrixise #29484: AddressSanitizer: heap-buffer-overflow on address 0x60200000e7 http://bugs.python.org/issue29484 closed by matrixise #29485: AddressSanitizer: SEGV on unknown address 0x7fab556df550 http://bugs.python.org/issue29485 closed by matrixise #29486: AddressSanitizer: SEGV on unknown address 0x7f16f88e3560 http://bugs.python.org/issue29486 closed by matrixise #29487: AddressSanitizer: heap-buffer-overflow on address 0x60200000e7 http://bugs.python.org/issue29487 closed by matrixise #29488: AddressSanitizer: SEGV on unknown address 0x0001a5525c1b http://bugs.python.org/issue29488 closed by matrixise #29489: AddressSanitizer: SEGV on unknown address 0x7f4a36c604d0 http://bugs.python.org/issue29489 closed by matrixise #29490: AddressSanitizer: heap-buffer-overflow on address 0x60200000e7 http://bugs.python.org/issue29490 closed by matrixise #29491: AddressSanitizer: heap-buffer-overflow on address 0x60200000e7 http://bugs.python.org/issue29491 closed by matrixise #29492: AddressSanitizer: SEGV on unknown address 0x0000a0013639 http://bugs.python.org/issue29492 closed by matrixise #29493: AddressSanitizer: SEGV on unknown address 0x000cffff800d http://bugs.python.org/issue29493 closed by matrixise #29494: AddressSanitizer: SEGV on unknown address 0x00009fff8001 http://bugs.python.org/issue29494 closed by matrixise #29495: AddressSanitizer: SEGV on unknown address 0x02007ea947c3 http://bugs.python.org/issue29495 closed by matrixise #29496: AddressSanitizer: SEGV on unknown address 0x01ffe96de071 http://bugs.python.org/issue29496 closed by matrixise #29497: AddressSanitizer: SEGV on unknown address 0x000000000008 http://bugs.python.org/issue29497 closed by matrixise #29498: AddressSanitizer: SEGV on unknown address 0x0005ffff800d http://bugs.python.org/issue29498 closed by matrixise #29499: AddressSanitizer: SEGV on unknown address 0x000ebfff800d http://bugs.python.org/issue29499 closed by matrixise #29500: AddressSanitizer: heap-buffer-overflow on address 0x61600004a9 http://bugs.python.org/issue29500 closed by matrixise #29501: AddressSanitizer: SEGV on unknown address 0x0000000028cb http://bugs.python.org/issue29501 closed by matrixise #29508: Cannot install python-3.2.5 on Mac 10.11.6 http://bugs.python.org/issue29508 closed by zach.ware #29510: gitignore settings files for Eclipse IDE http://bugs.python.org/issue29510 closed by berker.peksag #29513: os.scandir(str) reference leak (test_os refleak) http://bugs.python.org/issue29513 closed by serhiy.storchaka #29519: weakref spewing exceptions during finalization when combined w http://bugs.python.org/issue29519 closed by lukasz.langa #29522: PyLong_AsDouble Behaviour is Weird http://bugs.python.org/issue29522 closed by serhiy.storchaka From tjreedy at udel.edu Fri Feb 10 16:15:46 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 10 Feb 2017 16:15:46 -0500 Subject: [Python-Dev] bugs.python.org access problems Message-ID: Today I have been getting the following for about 1/3 of access attempts (with Firefox). ''' Secure Connection Failed An error occurred during a connection to bugs.python.org. A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem. [Try Again] ''' A couple of retries usually work, but I don't remember such repeated errors before. Anyone else having a similar problem? -- Terry Jan Reedy From brett at python.org Fri Feb 10 18:20:36 2017 From: brett at python.org (Brett Cannon) Date: Fri, 10 Feb 2017 23:20:36 +0000 Subject: [Python-Dev] CPython is now on GitHub Message-ID: [this is a duplicate of what I just sent python-committers; rendered version at https://paper.dropbox.com/doc/CPython-workflow-changes-mx1k8G6M0rg5JLy80F1r6 ] # CPython workflow changes After more than two years, our new GitHub workflow is ready to accept changes (you can look back to my first ?[vision statement]( https://mail.python.org/pipermail/python-dev/2014-December/137472.html)? on changing our workflow to see how things have changed since I started working on this)! I hope you are all excited to see this finished; I know my wife is very excited as she?s tired of listening to me talk about it for a third of our marriage. ;) # Thanks First and foremost, I want to thank everyone who helped with this. Thanks to Donald and Barry for writing the initial PEPs proposing [GitHub]( https://www.python.org/dev/peps/pep-0481/) and [GitLab]( https://www.python.org/dev/peps/pep-0507/) and Nick [pre-proposing RhodeCode](https://www.python.org/dev/peps/pep-0474/). Thanks to everyone on [core-workflow](https://mail.python.org/mailman/listinfo/core-workflow) for helping out with various discussion (and which will continue to host discussions on future improvements on our workflow). Thanks to Ezio, Maciej, and Anish Shah for helping with the changes required to bugs.python.org in order to keep the issue tracker around. Thanks to the infrastructure team for helping deal with the migration of the [peps]( https://github.com/python/peps) and [devguide]( https://github.com/python/devguide) repos (especially Donald and Ernest). Thanks to Senthil for doing the conversion of the repo itself. Thanks to Benjamin for helping with hg.python.org stuff. Thanks to Zach for helping with the buildbots (and the devguide). Thanks to Mariatta, Carol Willing, Berker, Oleg, and St?phane Wirtel for helping with the devguide. There are also plenty of other people who have provided feedback over the past 2 years on mailing lists and in-person. # What has changed The documentation in the [devguide](https://cpython-devguide.readthedocs.io) should be up-to-date, so don?t worry about keeping this around as a reference. Consider this more of an announcement letter to get people quickly up-to-speed and excited about the new workflow. ## The location of the code repository CPython?s code now lives at https://github.com/python/cpython . hg.python.org/cpython is and will stay read-only (no determination has been made as to how long the Mercurial repository will be kept running). It should also be mentioned that we are doing squash commits and not rebased commits or merge commits as some projects on GitHub do. This basically means we will continue to have a single contribution lead to a single commit, keeping our history linear and compact. To up the bus factor on the new repository, I have set up a team for [Python release managers]( https://github.com/orgs/python/teams/python-release-managers) and made them administrators on the repository. I don?t necessarily expect RMs to use this power, but it?s there in case any of them need to change a setting in order to get a release out (to be upfront about it, I?m also in the team as its creator, but I have administrative privileges for the [Python team]( https://github.com/python) on GitHub so it doesn?t change what I?m able to do). ## Specifying issue numbers Traditionally we have specified issues as ?Issue #NNNN?. The problem with this format is that [GitHub automatically links]( https://help.github.com/articles/autolinked-references-and-urls/#issues-and-pull-requests) text in this format to GitHub issues and pull requests. While our repository will initially have no issues or PRs to automatically link to, this might not be true long-term (GitHub does the automatic linking eagerly at push time, so there?s no worry for older commit messages; we actually almost mutated the history to match the new format but in the end I made the decision not to as I didn?t consult with python-committers prior to the migration to make sure such a change was acceptable to everyone). To avoid this issue we are going to start specifying issues as ?bpo-NNNN?. This clearly delineates that issue numbers directly relate to bugs.python.org since ?[namespaces are one honking great idea]( https://www.python.org/dev/peps/pep-0020/)?. This is also a format that GitHub supports ? ?GH-NNNN? ? as well as JIRA who [lets you specify the prefix]( https://confluence.atlassian.com/adminjiraserver072/changing-the-project-key-format-828787722.html), so there?s precedent for choosing it. This change applies both to `[Misc/NEWS]( https://cpython-devguide.readthedocs.io/committing.html#what-s-new-and-news-entries)` and in [commit messages]( https://cpython-devguide.readthedocs.io/committing.html#commit-messages). Mentioning an issue this way in a pull request title or comment will connect the PR to the corresponding issue on bugs.python.org. Mentioning an issue this way in a commit message will cause a comment to show up in the issue relating to the commit. ## Cherry-picking instead of merging When a patch has spanned more than one version/branch we have always done a forward merge. The common issue with this, though, is it leads to racing with other committers from when you make to your initial commit in the oldest version/branch to pushing to hg.python.org on the newest version/branch. There was also the problem of having to remember that Python 2.7 is a special branch which was never merged forward. To deal with these issues we will use [cherry-picking]( https://cpython-devguide.readthedocs.io/committing.html#backporting-changes-to-python-3-6-or-older-version) going forward. This allows changes to be pulled into other branches as independent commits. This prevents any commit races with other core developers as we have traditionally needed to deal with when doing forward merges that span e.g. three branches. It also allows using CI to easily verify a change works if each cherry-pick is done as a separate pull request. The Python 2.7 branch also stops being a special case when backporting. It also prevents potential issues stemming from contributors submitting pull requests against the master branch by default and not the oldest branch a change should be applied to. Finally, this also removes the discussion of whether a change should be backported or not from blocking the commit into master to begin with. Labels will be provided for people to use to help track any cherry-picking that needs to occur for a pull request (e.g. ?backport to 3.6?). I also left the ?bug? and ?enhancement? labels to help classify PRs (adding more labels is easy so we can do that as our experience and workflow organically converge towards common practices). ## Protected branches All feature branches have been marked as [protected]( https://help.github.com/articles/about-protected-branches/). This means that feature branches cannot be deleted nor can they be pushed to directly. The latter will be the biggest change as it means all changes must go through a pull request. This helps make sure that there are no accidental breakage of code (I know I have done this multiple times when in a rush and I didn?t take my time when preparing a commit). This also means that all core developers follow the same development workflow as any other contributor. This not only allows all core developers to be able to help any other contributors with our workflow, but it also helps make sure we are aware of any sticking points in the contributor process so that we can all work towards resolving them for everyone?s benefit. (If experience shows that this is too much overhead we can turn this off.) All feature branches that are in security-only mode are locked down so that only [Python release managers]( https://github.com/orgs/python/teams/python-release-managers) can approve pull requests to them. For all branches that have reached EOL, no one is able to push to them. I expect that RMs will also use this feature when they are ready to gate all commits to a branch on their approval (e.g. when a release reaches RC, maybe even beta if they choose to go that far). # What has improved ## Accepting PRs through GitHub?s web UI While using hg.python.org, all commits had to be done through Mercurial?s CLI. With the move to GitHub we gain the ability to accept pull requests through a web UI. While this will only accept the change into the branch it was submitted against (which can be changed in the web UI), for situations where a change does not need to be backported it will allow for easier acceptance of a change. (When a change does need to be backported this is when you need to cherry-pick and that requires using the git CLI). If a change does need to be cherry-picked into an older branch you can either wait to accept the PR when you have a clone to work with or accept the change into master now and then cherry-pick later when you have a clone available. ## Continuous integration Previously changes required running the test suite manually along with verifying various other things like the documentation building. Moving to GitHub allows us to leverage the Travis continuous integration service to test several things in parallel automatically for each pull request: 1. Debug build under gcc 2. Debug build under clang 3. Documentation is valid and has no stale links 4. Python.h C++ compatibility While this doesn?t solve all testing scenarios (e.g. this doesn?t test a macOS or Windows-related change due to the added hours it take for a PR to be ?green? when run on Travis for macOS or AppVeyor for Windows), it does help with the common case of a cross-platform change. (There is an [open issue](https://github.com/python/core-workflow/issues/14) to add some code so that these tests only run when appropriate files have changed so that e.g. fixing a spelling mistake doesn?t run the test suite.) It should be mentioned that status checks on issues are **not** required prior to committing a pull request. While this may be a good idea long-term, until we know that our test suite is stable enough to not have regular flaky tests this would be more trouble than it?s worth (GitHub does visibly show, though, when not all status checks have passed so you won?t easily ignore this situation either). ## Code coverage Traditionally the code coverage of our tests was only known when someone ran the test suite manually under something like [coverage.py]( https://coverage.readthedocs.io). Even when someone did generate a coverage report it was generally not shared with other developers, and so it wasn?t widely known if a pull request increased or lowered test coverage. With the move to GitHub we are able to use [Codecov](https://codecov.io/) to calculate code coverage for each pull request. This also implicitly tests a non-debug build as that?s used to make the coverage results run faster. It should be noted, though, that some tests are skipped due to them holding up the coverage run from completing. (There is an [open issue]( https://github.com/python/core-workflow/issues/18) to use coverage.py?s fullcoverage hack so that the coverage report can even be accurate for modules imported during interpreter startup.) ## CLA enforcement To tell if someone has signed the PSF contributor license agreement you have to look to see if they have an asterisk by their name on the issue tracker. Unfortunately this is a passive thing to need to check for and is easily forgotten. Thanks to GitHub?s webhook events and developer API we now have a bot which checks if the contributor(s) to a pull request have signed the CLA, adding an appropriate label to the PR to signal the CLA signing status (the bot is named [The Knights Who Say Ni]( https://github.com/python/the-knights-who-say-ni)). If the contributor(s) have not signed the CLA then a message is left on the PR explaining how to rectify the issue (it?s either they need to connect their GitHub account to their bugs.python.org account or they need to sign the CLA; there?s also is an easter egg that occasionally appears in the message). If a contributor does end up fixing the issue that leads to the bot thinking the contributor had not signed the CLA, you can remove the ?CLA not signed? label and the bot will recheck the PR and add the appropriate label (this also happens automatically if any code changes are made to the PR). If for some the reason the bot has a hiccup then no label will be applied (this is to act as a safeguard against false-negatives and to make it easy to spot when something has gone wrong due to the absence of either a ?CLA signed? or ?CLA not signed? label). To trigger the bot again you can simply apply the ?CLA not signed? label and then remove it. ## Contribution guidelines There is now a `[.github/CONTRIBUTING.rst]( https://github.com/python/cpython/blob/master/.github/CONTRIBUTING.rst)` file which gives general info on contributing. Primarily it has all the various build status badges and links, link to the devguide and an overview of how we differ from most GitHub projects, and a mention of the CoC. For core devs I suspect the badge list for all active branches will be useful to know when something is broken (I am only listing the test coverage for the master branch to prevent encouraging people spending time trying to increase test coverage for bugfix-only branches) . # The future This isn?t here for discussion per-se, but to let people know what I am thinking should change next. If you want to help or discuss anything in this section, please subscribe to core-workflow and participate there. Ideas are also tracked on the [core-workflow issue tracker]( https://github.com/python/core-workflow/issues) (where there are other ideas for improvements beyond the two listed below). First, we need to solve the Misc/NEWS problem. The [plan]( https://github.com/python/core-workflow/issues/6) is to move to individual files per NEWS entry and then have a script to roll them all up into the NEWS file at release time. This will do away with merge conflicts which has always been the biggest hassle when a change spanned versions. Second, we will probably develop some solution to [automatically generate cherry-picking PRs](https://github.com/python/core-workflow/issues/8). This plus the Misc/NEWS solution should be enough to allow most patches to be accepted through the web UI entirely. All the other changes are nice-to-have, but I think these two will lead to the greatest improvement to our workflow and get us far beyond the workflow we had on hg.python.org. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Feb 10 19:05:32 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 11 Feb 2017 01:05:32 +0100 Subject: [Python-Dev] bugs.python.org access problems References: Message-ID: <20170211010532.25adf235@fsol> On Fri, 10 Feb 2017 16:15:46 -0500 Terry Reedy wrote: > Today I have been getting the following for about 1/3 of access > attempts (with Firefox). > ''' > Secure Connection Failed > > An error occurred during a connection to bugs.python.org. A PKCS #11 > module returned CKR_DEVICE_ERROR, indicating that a problem has occurred > with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR > > The page you are trying to view cannot be shown because the > authenticity of the received data could not be verified. > Please contact the website owners to inform them of this problem. > > [Try Again] > ''' > A couple of retries usually work, but I don't remember such repeated > errors before. Anyone else having a similar problem? Yes, I do. Regards Antoine. From brett at python.org Sat Feb 11 14:01:46 2017 From: brett at python.org (Brett Cannon) Date: Sat, 11 Feb 2017 19:01:46 +0000 Subject: [Python-Dev] bugs.python.org access problems In-Reply-To: <20170211010532.25adf235@fsol> References: <20170211010532.25adf235@fsol> Message-ID: On Fri, 10 Feb 2017 at 16:06 Antoine Pitrou wrote: > On Fri, 10 Feb 2017 16:15:46 -0500 > Terry Reedy wrote: > > > Today I have been getting the following for about 1/3 of access > > attempts (with Firefox). > > ''' > > Secure Connection Failed > > > > An error occurred during a connection to bugs.python.org. A PKCS #11 > > module returned CKR_DEVICE_ERROR, indicating that a problem has occurred > > with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR > > > > The page you are trying to view cannot be shown because the > > authenticity of the received data could not be verified. > > Please contact the website owners to inform them of this problem. > > > > [Try Again] > > ''' > > A couple of retries usually work, but I don't remember such repeated > > errors before. Anyone else having a similar problem? > > Yes, I do. > I have had a similar issue on Chrome but it seems to reload itself and move on from it too fast to read the error page. GitHub has also been complaining on their side about it. I mentioned it to Ezio yesterday during the migration but he said he didn't see anything odd on the bugs.python.org side. I'll shoot an email to the infrastructure team to see if they know what is going on. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sat Feb 11 14:09:11 2017 From: brett at python.org (Brett Cannon) Date: Sat, 11 Feb 2017 19:09:11 +0000 Subject: [Python-Dev] bugs.python.org access problems In-Reply-To: References: <20170211010532.25adf235@fsol> Message-ID: I emailed the infrastructure team and Mark Mangoba replied back saying it's because the SSL cert is from StartCom which is in the process of being distrusted. They should have a new cert hopefully sometime today. On Sat, 11 Feb 2017 at 11:01 Brett Cannon wrote: > On Fri, 10 Feb 2017 at 16:06 Antoine Pitrou wrote: > > On Fri, 10 Feb 2017 16:15:46 -0500 > Terry Reedy wrote: > > > Today I have been getting the following for about 1/3 of access > > attempts (with Firefox). > > ''' > > Secure Connection Failed > > > > An error occurred during a connection to bugs.python.org. A PKCS #11 > > module returned CKR_DEVICE_ERROR, indicating that a problem has occurred > > with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR > > > > The page you are trying to view cannot be shown because the > > authenticity of the received data could not be verified. > > Please contact the website owners to inform them of this problem. > > > > [Try Again] > > ''' > > A couple of retries usually work, but I don't remember such repeated > > errors before. Anyone else having a similar problem? > > Yes, I do. > > > I have had a similar issue on Chrome but it seems to reload itself and > move on from it too fast to read the error page. GitHub has also been > complaining on their side about it. I mentioned it to Ezio yesterday during > the migration but he said he didn't see anything odd on the > bugs.python.org side. > > I'll shoot an email to the infrastructure team to see if they know what is > going on. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mmangoba at python.org Sat Feb 11 16:17:24 2017 From: mmangoba at python.org (Mark Mangoba) Date: Sat, 11 Feb 2017 13:17:24 -0800 Subject: [Python-Dev] bugs.python.org access problems In-Reply-To: References: <20170211010532.25adf235@fsol> Message-ID: Hi All, The new cert has been installed. Let me know if you have any issues. https://bugs.python.org/ Best regards, Mark On Sat, Feb 11, 2017 at 11:09 AM, Brett Cannon wrote: > I emailed the infrastructure team and Mark Mangoba replied back saying > it's because the SSL cert is from StartCom which is in the process of being > distrusted. They should have a new cert hopefully sometime today. > > On Sat, 11 Feb 2017 at 11:01 Brett Cannon wrote: > >> On Fri, 10 Feb 2017 at 16:06 Antoine Pitrou wrote: >> >> On Fri, 10 Feb 2017 16:15:46 -0500 >> Terry Reedy wrote: >> >> > Today I have been getting the following for about 1/3 of access >> > attempts (with Firefox). >> > ''' >> > Secure Connection Failed >> > >> > An error occurred during a connection to bugs.python.org. A PKCS #11 >> > module returned CKR_DEVICE_ERROR, indicating that a problem has occurred >> > with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR >> > >> > The page you are trying to view cannot be shown because the >> > authenticity of the received data could not be verified. >> > Please contact the website owners to inform them of this problem. >> > >> > [Try Again] >> > ''' >> > A couple of retries usually work, but I don't remember such repeated >> > errors before. Anyone else having a similar problem? >> >> Yes, I do. >> >> >> I have had a similar issue on Chrome but it seems to reload itself and >> move on from it too fast to read the error page. GitHub has also been >> complaining on their side about it. I mentioned it to Ezio yesterday during >> the migration but he said he didn't see anything odd on the >> bugs.python.org side. >> >> I'll shoot an email to the infrastructure team to see if they know what >> is going on. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Feb 12 12:44:38 2017 From: brett at python.org (Brett Cannon) Date: Sun, 12 Feb 2017 17:44:38 +0000 Subject: [Python-Dev] bugs.python.org access problems In-Reply-To: References: <20170211010532.25adf235@fsol> Message-ID: On Sat, 11 Feb 2017 at 13:18 Mark Mangoba wrote: > Hi All, > > The new cert has been installed. Let me know if you have any issues. > > https://bugs.python.org/ > Thanks, Mark! At least from the GitHub webhook side that seems to have solved the problem. And when I load the site up on Chrome I don't get the brief error page anymore. -Brett > > > Best regards, > Mark > > On Sat, Feb 11, 2017 at 11:09 AM, Brett Cannon wrote: > > I emailed the infrastructure team and Mark Mangoba replied back saying > it's because the SSL cert is from StartCom which is in the process of being > distrusted. They should have a new cert hopefully sometime today. > > On Sat, 11 Feb 2017 at 11:01 Brett Cannon wrote: > > On Fri, 10 Feb 2017 at 16:06 Antoine Pitrou wrote: > > On Fri, 10 Feb 2017 16:15:46 -0500 > Terry Reedy wrote: > > > Today I have been getting the following for about 1/3 of access > > attempts (with Firefox). > > ''' > > Secure Connection Failed > > > > An error occurred during a connection to bugs.python.org. A PKCS #11 > > module returned CKR_DEVICE_ERROR, indicating that a problem has occurred > > with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR > > > > The page you are trying to view cannot be shown because the > > authenticity of the received data could not be verified. > > Please contact the website owners to inform them of this problem. > > > > [Try Again] > > ''' > > A couple of retries usually work, but I don't remember such repeated > > errors before. Anyone else having a similar problem? > > Yes, I do. > > > I have had a similar issue on Chrome but it seems to reload itself and > move on from it too fast to read the error page. GitHub has also been > complaining on their side about it. I mentioned it to Ezio yesterday during > the migration but he said he didn't see anything odd on the > bugs.python.org side. > > I'll shoot an email to the infrastructure team to see if they know what is > going on. > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Mon Feb 13 09:26:48 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Mon, 13 Feb 2017 14:26:48 +0000 Subject: [Python-Dev] PEP 543: A Unified TLS API for Python Message-ID: <0AC4A5C1-B6D3-41FF-80E0-2B26266A5956@lukasa.co.uk> All, Python has never exposed the ability to use TLS backends other than OpenSSL in the standard library. As Christian and I discussed back at the Developer Summit at PyCon US 2016, the ssl module would more properly be called the openssl module, due to exposing many OpenSSL-specific concepts and behaviours in its API. This has meant that the Python ecosystem is overwhelmingly also an OpenSSL ecosystem, which is problematic on Windows and macOS (and Unices that for whatever reason aren?t interested in shipping an OpenSSL), as it has meant that Python needs to bring its own OpenSSL, and that it is troublesome to interact with the system trust database. The first step to resolving this would be to provide a new module that exposes TLS concepts in a much more generic manner. There are two possible approaches to this. The first is to have this module be a generic concrete implementation that can be compiled against multiple TLS backends (like curl). This would require that all the relevant bindings be built into the standard library from the get-go, which provides a substantial maintenance burden on a team of people that are already understaffed maintaining the ssl module alone. The second approach is to define a generic high-level TLS interface that provides a minimal usable interface that can be implemented by both first- and third-party modules. This would allow the standard library to continue to ship with only exactly what it needs (for example, OpenSSL, SecureTransport and SChannel), but would give those who want to use more esoteric TLS choices (NSS, GnuTLS, mbedTLS, and BoringSSL are some examples) an API that they can implement that will guarantee that complying modules can use the appropriate TLS backend. To that end, I?ve composed a draft PEP that would define this API: PEP 543. This draft can be found online here[1], and I have reproduced it below for those who want to reply inline. I should apologise in advance that this PEP is quite large: it is this large because it lays out a substantial proportion of the new code that would be added directly in the PEP. This is to help de-risk the work by showing as much of it up-front during the PEP stage, rather than having it emerge over the implementation process. Please note that this proposal is not intended to resolve the current problem pip has with TLSv1.2, and so is not subject to the tight time constraints associated with that problem. That problem will be solved by building a temporary shim into urllib3 (and therefore pip) that can use SecureTransport bindings on the Mac. This proposal is intended as a solution to the more-general problem of supporting different TLS backends in Python, and so is larger in scope and less urgent. I should also mention that there will be a tendency to want to make this API all things to all people from the get-go. I?m going to strongly resist attempts to extend this API too much, because each additional bit of API surface makes it harder for us to encourage module authors to conform to this API. I will encourage people to extend this API over time as needed, but to begin with I think it is most important that basic TLS clients and servers can be constructed with this API. More specialised features should be considered future enhancements, rather than being tacked on to this initial PEP. Please let me know what you think. I?d like significant feedback from the community before I ask the BDFL to pronounce on this PEP, as this is the kind of change that will affect a large amount of the Python ecosystem. It?s important to me that we minimise the risk to ourselves and the community by getting this as right as possible, and while we can help with that by limiting scope, it?s also true that we need as many eyes as possible. Please let me know what you think. Cory [1]: https://www.python.org/dev/peps/pep-0543/ ? PEP: 543 Title: A Unified TLS API for Python Version: $Revision$ Last-Modified: $Date$ Author: Cory Benfield , Christian Heimes Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 17-Oct-2016 Python-Version: 3.7 Post-History: 11-Jan-2017, 19-Jan-2017, 02-Feb-2017, 09-Feb-2017 Abstract ======== This PEP would define a standard TLS interface in the form of a collection of abstract base classes. This interface would allow Python implementations and third-party libraries to provide bindings to TLS libraries other than OpenSSL that can be used by tools that expect the interface provided by the Python standard library, with the goal of reducing the dependence of the Python ecosystem on OpenSSL. Rationale ========= In the 21st century it has become increasingly clear that robust and user-friendly TLS support is an extremely important part of the ecosystem of any popular programming language. For most of its lifetime, this role in the Python ecosystem has primarily been served by the `ssl module`_, which provides a Python API to the `OpenSSL library`_. Because the ``ssl`` module is distributed with the Python standard library, it has become the overwhelmingly most-popular method for handling TLS in Python. An extraordinary majority of Python libraries, both in the standard library and on the Python Package Index, rely on the ``ssl`` module for their TLS connectivity. Unfortunately, the preeminence of the ``ssl`` module has had a number of unforeseen side-effects that have had the effect of tying the entire Python ecosystem tightly to OpenSSL. This has forced Python users to use OpenSSL even in situations where it may provide a worse user experience than alternative TLS implementations, which imposes a cognitive burden and makes it hard to provide "platform-native" experiences. Problems -------- The fact that the ``ssl`` module is built into the standard library has meant that all standard-library Python networking libraries are entirely reliant on the OpenSSL that the Python implementation has been linked against. This leads to the following issues: * It is difficult to take advantage of new, higher-security TLS without recompiling Python to get a new OpenSSL. While there are third-party bindings to OpenSSL (e.g. `pyOpenSSL`_), these need to be shimmed into a format that the standard library understands, forcing projects that want to use them to maintain substantial compatibility layers. * For Windows distributions of Python, they need to be shipped with a copy of OpenSSL. This puts the CPython development team in the position of being OpenSSL redistributors, potentially needing to ship security updates to the Windows Python distributions when OpenSSL vulnerabilities are released. * For macOS distributions of Python, they need either to be shipped with a copy of OpenSSL or linked against the system OpenSSL library. Apple has formally deprecated linking against the system OpenSSL library, and even if they had not, that library version has been unsupported by upstream for nearly one year as of the time of writing. The CPython development team has started shipping newer OpenSSLs with the Python available from python.org, but this has the same problem as with Windows. * Many systems, including but not limited to Windows and macOS, do not make their system certificate stores available to OpenSSL. This forces users to either obtain their trust roots from elsewhere (e.g. `certifi`_) or to attempt to export their system trust stores in some form. Relying on `certifi`_ is less than ideal, as most system administrators do not expect to receive security-critical software updates from PyPI. Additionally, it is not easy to extend the `certifi`_ trust bundle to include custom roots, or to centrally manage trust using the `certifi`_ model. Even in situations where the system certificate stores are made available to OpenSSL in some form, the experience is still sub-standard, as OpenSSL will perform different validation checks than the platform-native TLS implementation. This can lead to users experiencing different behaviour on their browsers or other platform-native tools than they experience in Python, with little or no recourse to resolve the problem. * Users may wish to integrate with TLS libraries other than OpenSSL for many other reasons, such as OpenSSL missing features (e.g. TLS 1.3 support), or because OpenSSL is simply too large and unwieldy for the platform (e.g. for embedded Python). Those users are left with the requirement to use third-party networking libraries that can interact with their preferred TLS library or to shim their preferred library into the OpenSSL-specific ``ssl`` module API. Additionally, the ``ssl`` module as implemented today limits the ability of CPython itself to add support for alternative TLS backends, or remove OpenSSL support entirely, should either of these become necessary or useful. The ``ssl`` module exposes too many OpenSSL-specific function calls and features to easily map to an alternative TLS backend. Proposal ======== This PEP proposes to introduce a few new Abstract Base Classes in Python 3.7 to provide TLS functionality that is not so strongly tied to OpenSSL. It also proposes to update standard library modules to use only the interface exposed by these abstract base classes wherever possible. There are three goals here: 1. To provide a common API surface for both core and third-party developers to target their TLS implementations to. This allows TLS developers to provide interfaces that can be used by most Python code, and allows network developers to have an interface that they can target that will work with a wide range of TLS implementations. 2. To provide an API that has few or no OpenSSL-specific concepts leak through. The ``ssl`` module today has a number of warts caused by leaking OpenSSL concepts through to the API: the new ABCs would remove those specific concepts. 3. To provide a path for the core development team to make OpenSSL one of many possible TLS backends, rather than requiring that it be present on a system in order for Python to have TLS support. The proposed interface is laid out below. Interfaces ---------- There are several interfaces that require standardisation. Those interfaces are: 1. Configuring TLS, currently implemented by the `SSLContext`_ class in the ``ssl`` module. 2. Providing an in-memory buffer for doing in-memory encryption or decryption with no actual I/O (necessary for asynchronous I/O models), currently implemented by the `SSLObject`_ class in the ``ssl`` module. 3. Wrapping a socket object, currently implemented by the `SSLSocket`_ class in the ``ssl`` module. 4. Applying TLS configuration to the wrapping objects in (2) and (3). Currently this is also implemented by the `SSLContext`_ class in the ``ssl`` module. 5. Specifying TLS cipher suites. There is currently no code for doing this in the standard library: instead, the standard library uses OpenSSL cipher suite strings. 6. Specifying application-layer protocols that can be negotiated during the TLS handshake. 7. Specifying TLS versions. 8. Reporting errors to the caller, currently implemented by the `SSLError`_ class in the ``ssl`` module. 9. Specifying certificates to load, either as client or server certificates. 10. Specifying which trust database should be used to validate certificates presented by a remote peer. 11. Finding a way to get hold of these interfaces at run time. For the sake of simplicitly, this PEP proposes to take a unified approach to (2) and (3) (that is, buffers and sockets). The Python socket API is a sizeable one, and implementing a wrapped socket that has the same behaviour as a regular Python socket is a subtle and tricky thing to do. However, it is entirely possible to implement a *generic* wrapped socket in terms of wrapped buffers: that is, it is possible to write a wrapped socket (3) that will work for any implementation that provides (2). For this reason, this PEP proposes to provide an ABC for wrapped buffers (2) but a concrete class for wrapped sockets (3). This decision has the effect of making it impossible to bind a small number of TLS libraries to this ABC, because those TLS libraries *cannot* provide a wrapped buffer implementation. The most notable of these at this time appears to be Amazon's `s2n`_, which currently does not provide an I/O abstraction layer. However, even this library consider this a missing feature and are `working to add it`_. For this reason, it is safe to assume that a concrete implementation of (3) in terms of (2) will be a substantial effort-saving device and a great tool for correctness. Therefore, this PEP proposes doing just that. Obviously, (5) doesn't require an abstract base class: instead, it requires a richer API for configuring supported cipher suites that can be easily updated with supported cipher suites for different implementations. (9) is a thorny problem, because in an ideal world the private keys associated with these certificates would never end up in-memory in the Python process (that is, the TLS library would collaborate with a Hardware Security Module (HSM) to provide the private key in such a way that it cannot be extracted from process memory). Thus, we need to provide an extensible model of providing certificates that allows concrete implementations the ability to provide this higher level of security, while also allowing a lower bar for those implementations that cannot. This lower bar would be the same as the status quo: that is, the certificate may be loaded from an in-memory buffer or from a file on disk. (10) also represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. Some implementations have specific trust store formats that only they can use (such as the OpenSSL CA directory format that is created by ``c_rehash``), and others may not allow you to specify a trust store that does not include their default trust store. For this reason, we need to provide a model that assumes very little about the form that trust stores take. The "Trust Store" section below goes into more detail about how this is achieved. Finally, this API will split the responsibilities currently assumed by the `SSLContext`_ object: specifically, the responsibility for holding and managing configuration and the responsibility for using that configuration to build wrapper objects. This is necessarily primarily for supporting functionality like Server Name Indication (SNI). In OpenSSL (and thus in the ``ssl`` module), the server has the ability to modify the TLS configuration in response to the client telling the server what hostname it is trying to reach. This is mostly used to change certificate chain so as to present the correct TLS certificate chain for the given hostname. The specific mechanism by which this is done is by returning a new `SSLContext`_ object with the appropriate configuration. This is not a model that maps well to other TLS implementations. Instead, we need to make it possible to provide a return value from the SNI callback that can be used to indicate what configuration changes should be made. This means providing an object that can hold TLS configuration. This object needs to be applied to specific TLSWrappedBuffer, and TLSWrappedSocket objects. For this reason, we split the responsibility of `SSLContext`_ into two separate objects. The ``TLSConfiguration`` object is an object that acts as container for TLS configuration: the ``ClientContext`` and ``ServerContext`` objects are objects that are instantiated with a ``TLSConfiguration`` object. All three objects would be immutable. .. note:: The following API declarations uniformly use type hints to aid reading. Some of these type hints cannot actually be used in practice because they are circularly referential. Consider them more a guideline than a reflection of the final code in the module. Configuration ~~~~~~~~~~~~~ The ``TLSConfiguration`` concrete class defines an object that can hold and manage TLS configuration. The goals of this class are as follows: 1. To provide a method of specifying TLS configuration that avoids the risk of errors in typing (this excludes the use of a simple dictionary). 2. To provide an object that can be safely compared to other configuration objects to detect changes in TLS configuration, for use with the SNI callback. This class is not an ABC, primarily because it is not expected to have implementation-specific behaviour. The responsibility for transforming a ``TLSConfiguration`` object into a useful set of configuration for a given TLS implementation belongs to the Context objects discussed below. This class has one other notable property: it is immutable. This is a desirable trait for a few reasons. The most important one is that it allows these objects to be used as dictionary keys, which is potentially extremely valuable for certain TLS backends and their SNI configuration. On top of this, it frees implementations from needing to worry about their configuration objects being changed under their feet, which allows them to avoid needing to carefully synchronize changes between their concrete data structures and the configuration object. This object is extendable: that is, future releases of Python may add configuration fields to this object as they become useful. For backwards-compatibility purposes, new fields are only appended to this object. Existing fields will never be removed, renamed, or reordered. The ``TLSConfiguration`` object would be defined by the following code:: ServerNameCallback = Callable[[TLSBufferObject, Optional[str], TLSConfiguration], Any] _configuration_fields = [ 'validate_certificates', 'certificate_chain', 'ciphers', 'inner_protocols', 'lowest_supported_version', 'highest_supported_version', 'trust_store', 'sni_callback', ] _DEFAULT_VALUE = object() class TLSConfiguration(namedtuple('TLSConfiguration', _configuration_fields)): """ An immutable TLS Configuration object. This object has the following properties: :param validate_certificates bool: Whether to validate the TLS certificates. This switch operates at a very broad scope: either validation is enabled, in which case all forms of validation are performed including hostname validation if possible, or validation is disabled, in which case no validation is performed. Not all backends support having their certificate validation disabled. If a backend does not support having their certificate validation disabled, attempting to set this property to ``False`` will throw a ``TLSError`` when this object is passed into a context object. :param certificate_chain Tuple[Tuple[Certificate],PrivateKey]: The certificate, intermediate certificate, and the corresponding private key for the leaf certificate. These certificates will be offered to the remote peer during the handshake if required. The first Certificate in the list must be the leaf certificate. All subsequent certificates will be offered as intermediate additional certificates. :param ciphers Tuple[Union[CipherSuite, int]]: The available ciphers for TLS connections created with this configuration, in priority order. :param inner_protocols Tuple[Union[NextProtocol, bytes]]: Protocols that connections created with this configuration should advertise as supported during the TLS handshake. These may be advertised using either or both of ALPN or NPN. This list of protocols should be ordered by preference. :param lowest_supported_version TLSVersion: The minimum version of TLS that should be allowed on TLS connections using this configuration. :param highest_supported_version TLSVersion: The maximum version of TLS that should be allowed on TLS connections using this configuration. :param trust_store TrustStore: The trust store that connections using this configuration will use to validate certificates. :param sni_callback Optional[ServerNameCallback]: A callback function that will be called after the TLS Client Hello handshake message has been received by the TLS server when the TLS client specifies a server name indication. Only one callback can be set per ``TLSConfiguration``. If the ``sni_callback`` is ``None`` then the callback is disabled. If the ``TLSConfiguration`` is used for a ``ClientContext`` then this setting will be ignored. The ``callback`` function will be called with three arguments: the first will be the ``TLSBufferObject`` for the connection; the second will be a string that represents the server name that the client is intending to communicate (or ``None`` if the TLS Client Hello does not contain a server name); and the third argument will be the original ``TLSConfiguration`` that configured the connection. The server name argument will be the IDNA *decoded* server name. The ``callback`` must return a ``TLSConfiguration`` to allow negotiation to continue. Other return values signal errors. Attempting to control what error is signaled by the underlying TLS implementation is not specified in this API, but is up to the concrete implementation to handle. The Context will do its best to apply the ``TLSConfiguration`` changes from its original configuration to the incoming connection. This will usually include changing the certificate chain, but may also include changes to allowable ciphers or any other configuration settings. """ __slots__ = () def __new__(cls, validate_certificates: Optional[bool] = None, certificate_chain: Optional[Tuple[Tuple[Certificate], PrivateKey]] = None, ciphers: Optional[Tuple[Union[CipherSuite, int]]] = None, inner_protocols: Optional[Tuple[Union[NextProtocol, bytes]]] = None, lowest_supported_version: Optional[TLSVersion] = None, highest_supported_version: Optional[TLSVersion] = None, trust_store: Optional[TrustStore] = None, sni_callback: Optional[ServerNameCallback] = None): if validate_certificates is None: validate_certificates = True if ciphers is None: ciphers = DEFAULT_CIPHER_LIST if inner_protocols is None: inner_protocols = [] if lowest_supported_version is None: lowest_supported_version = TLSVersion.TLSv1 if highest_supported_version is None: highest_supported_version = TLSVersion.MAXIMUM_SUPPORTED return super().__new__( cls, validate_certificates, certificate_chain, ciphers, inner_protocols, lowest_supported_version, highest_supported_version, trust_store, sni_callback ) def update(self, validate_certificates=_DEFAULT_VALUE, certificate_chain=_DEFAULT_VALUE, ciphers=_DEFAULT_VALUE, inner_protocols=_DEFAULT_VALUE, lowest_supported_version=_DEFAULT_VALUE, highest_supported_version=_DEFAULT_VALUE, trust_store=_DEFAULT_VALUE, sni_callback=_DEFAULT_VALUE): """ Create a new ``TLSConfiguration``, overriding some of the settings on the original configuration with the new settings. """ if validate_certificates is _DEFAULT_VALUE: validate_certificates = self.validate_certificates if certificate_chain is _DEFAULT_VALUE: certificate_chain = self.certificate_chain if ciphers is _DEFAULT_VALUE: ciphers = self.ciphers if inner_protocols is _DEFAULT_VALUE: inner_protocols = self.inner_protocols if lowest_supported_version is _DEFAULT_VALUE: lowest_supported_version = self.lowest_supported_version if highest_supported_version is _DEFAULT_VALUE: highest_supported_version = self.highest_supported_version if trust_store is _DEFAULT_VALUE: trust_store = self.trust_store if sni_callback is _DEFAULT_VALUE: sni_callback = self.sni_callback return self.__class__( validate_certificates, certificate_chain, ciphers, inner_protocols, lowest_supported_version, highest_supported_version, trust_store, sni_callback ) Context ~~~~~~~ We define two Context abstract base classes. These ABCs define objects that allow configuration of TLS to be applied to specific connections. They can be thought of as factories for ``TLSWrappedSocket`` and ``TLSWrappedBuffer`` objects. Unlike the current ``ssl`` module, we provide two context classes instead of one. Specifically, we provide the ``ClientContext`` and ``ServerContext`` classes. This simplifies the APIs (for example, there is no sense in the server providing the ``server_hostname`` parameter to ``ssl.SSLContext.wrap_socket``, but because there is only one context class that parameter is still available), and ensures that implementations know as early as possible which side of a TLS connection they will serve. Additionally, it allows implementations to opt-out of one or either side of the connection. For example, SecureTransport on macOS is not really intended for server use and has an enormous amount of functionality missing for server-side use. This would allow SecureTransport implementations to simply not define a concrete subclass of ``ServerContext`` to signal their lack of support. One of the other major differences to the current ``ssl`` module is that a number of flags and options have been removed. Most of these are self-evident, but it is worth noting that ``auto_handshake`` has been removed from ``wrap_socket``. This was removed because it fundamentally represents an odd design wart that saves very minimal effort at the cost of a complexity increase both for users and implementers. This PEP requires that all users call ``do_handshake`` explicitly after connecting. As much as possible implementers should aim to make these classes immutable: that is, they should prefer not to allow users to mutate their internal state directly, instead preferring to create new contexts from new TLSConfiguration objects. Obviously, the ABCs cannot enforce this constraint, and so they do not attempt to. The ``Context`` abstract base class has the following class definition:: TLSBufferObject = Union[TLSWrappedSocket, TLSWrappedBuffer] class _BaseContext(metaclass=ABCMeta): @abstractmethod def __init__(self, configuration: TLSConfiguration): """ Create a new context object from a given TLS configuration. """ @property @abstractmethod def configuration(self) -> TLSConfiguration: """ Returns the TLS configuration that was used to create the context. """ class ClientContext(_BaseContext): def wrap_socket(self, socket: socket.socket, server_hostname: Optional[str]) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The socket object originally passed to this method should not be used again: attempting to use it in any way will lead to undefined behaviour, especially across different TLS implementations. To get the original socket object back once it has been wrapped in TLS, see the ``unwrap`` method of the TLSWrappedSocket. The parameter ``server_hostname`` specifies the hostname of the service which we are connecting to. This allows a single server to host multiple SSL-based services with distinct certificates, quite similarly to HTTP virtual hosts. This is also used to validate the TLS certificate for the given hostname. If hostname validation is not desired, then pass ``None`` for this parameter. This parameter has no default value because opting-out of hostname validation is dangerous, and should not be the default behaviour. """ buffer = self.wrap_buffers(server_hostname) return TLSWrappedSocket(socket, buffer) @abstractmethod def wrap_buffers(self, server_hostname: Optional[str]) -> TLSWrappedBuffer: """ Create an in-memory stream for TLS, using memory buffers to store incoming and outgoing ciphertext. The TLS routines will read received TLS data from one buffer, and write TLS data that needs to be emitted to another buffer. The implementation details of how this buffering works are up to the individual TLS implementation. This allows TLS libraries that have their own specialised support to continue to do so, while allowing those without to use whatever Python objects they see fit. The ``server_hostname`` parameter has the same meaning as in ``wrap_socket``. """ class ServerContext(_BaseContext): def wrap_socket(self, socket: socket.socket) -> TLSWrappedSocket: """ Wrap an existing Python socket object ``socket`` and return a ``TLSWrappedSocket`` object. ``socket`` must be a ``SOCK_STREAM`` socket: all other socket types are unsupported. The returned SSL socket is tied to the context, its settings and certificates. The socket object originally passed to this method should not be used again: attempting to use it in any way will lead to undefined behaviour, especially across different TLS implementations. To get the original socket object back once it has been wrapped in TLS, see the ``unwrap`` method of the TLSWrappedSocket. """ buffer = self.wrap_buffers() return TLSWrappedSocket(socket, buffer) @abstractmethod def wrap_buffers(self) -> TLSWrappedBuffer: """ Create an in-memory stream for TLS, using memory buffers to store incoming and outgoing ciphertext. The TLS routines will read received TLS data from one buffer, and write TLS data that needs to be emitted to another buffer. The implementation details of how this buffering works are up to the individual TLS implementation. This allows TLS libraries that have their own specialised support to continue to do so, while allowing those without to use whatever Python objects they see fit. """ Buffer ~~~~~~ The buffer-wrapper ABC will be defined by the ``TLSWrappedBuffer`` ABC, which has the following definition:: class TLSWrappedBuffer(metaclass=ABCMeta): @abstractmethod def read(self, amt: int) -> bytes: """ Read up to ``amt`` bytes of data from the input buffer and return the result as a ``bytes`` instance. Once EOF is reached, all further calls to this method return the empty byte string ``b''``. May read "short": that is, fewer bytes may be returned than were requested. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. May raise ``RaggedEOF`` if the connection has been closed without a graceful TLS shutdown. Whether this is an exception that should be ignored or not is up to the specific application. As at any time a re-negotiation is possible, a call to ``read()`` can also cause write operations. """ @abstractmethod def readinto(self, buffer: Any, amt: int) -> int: """ Read up to ``amt`` bytes of data from the input buffer into ``buffer``, which must be an object that implements the buffer protocol. Returns the number of bytes read. Once EOF is reached, all further calls to this method return the empty byte string ``b''``. Raises ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. May read "short": that is, fewer bytes may be read than were requested. May raise ``RaggedEOF`` if the connection has been closed without a graceful TLS shutdown. Whether this is an exception that should be ignored or not is up to the specific application. As at any time a re-negotiation is possible, a call to ``readinto()`` can also cause write operations. """ @abstractmethod def write(self, buf: Any) -> int: """ Write ``buf`` in encrypted form to the output buffer and return the number of bytes written. The ``buf`` argument must be an object supporting the buffer interface. Raise ``WantReadError`` or ``WantWriteError`` if there is insufficient data in either the input or output buffer and the operation would have caused data to be written or read. In either case, users should endeavour to resolve that situation and then re-call this method. When re-calling this method users *should* re-use the exact same ``buf`` object, as some backends require that the exact same buffer be used. This operation may write "short": that is, fewer bytes may be written than were in the buffer. As at any time a re-negotiation is possible, a call to ``write()`` can also cause read operations. """ @abstractmethod def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. """ @abstractmethod def cipher(self) -> Optional[Union[CipherSuite, int]]: """ Returns the CipherSuite entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. If the cipher negotiated is not defined in CipherSuite, returns the 16-bit integer representing that cipher directly. """ @abstractmethod def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If the negotiated protocol is one of the protocols defined in the ``NextProtocol`` enum, the value from that enum will be returned. Otherwise, the raw bytestring of the negotiated protocol will be returned. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property @abstractmethod def context(self) -> Context: """ The ``Context`` object this buffer is tied to. """ @abstractproperty def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ @abstractmethod def shutdown(self) -> None: """ Performs a clean TLS shut down. This should generally be used whenever possible to signal to the remote peer that the content is finished. """ @abstractmethod def receive_from_network(self, data): """ Receives some TLS data from the network and stores it in an internal buffer. """ @abstractmethod def peek_outgoing(self, amt): """ Returns the next ``amt`` bytes of data that should be written to the network from the outgoing data buffer, without removing it from the internal buffer. """ @abstractmethod def consume_outgoing(self, amt): """ Discard the next ``amt`` bytes from the outgoing data buffer. This should be used when ``amt`` bytes have been sent on the network, to signal that the data no longer needs to be buffered. """ Socket ~~~~~~ The socket-wrapper class will be a concrete class that accepts two items in its constructor: a regular socket object, and a ``TLSWrappedBuffer`` object. This object will be too large to recreate in this PEP, but will be submitted as part of the work to build the module. The wrapped socket will implement all of the socket API, though it will have stub implementations of methods that only work for sockets with types other than ``SOCK_STREAM`` (e.g. ``sendto``/``recvfrom``). That limitation can be lifted as-and-when support for DTLS is added to this module. In addition, the socket class will include the following *extra* methods on top of the regular socket methods:: class TLSWrappedSocket: def do_handshake(self) -> None: """ Performs the TLS handshake. Also performs certificate validation and hostname verification. This must be called after the socket has connected (either via ``connect`` or ``accept``), before any other operation is performed on the socket. """ def cipher(self) -> Optional[Union[CipherSuite, int]]: """ Returns the CipherSuite entry for the cipher that has been negotiated on the connection. If no connection has been negotiated, returns ``None``. If the cipher negotiated is not defined in CipherSuite, returns the 16-bit integer representing that cipher directly. """ def negotiated_protocol(self) -> Optional[Union[NextProtocol, bytes]]: """ Returns the protocol that was selected during the TLS handshake. This selection may have been made using ALPN, NPN, or some future negotiation mechanism. If the negotiated protocol is one of the protocols defined in the ``NextProtocol`` enum, the value from that enum will be returned. Otherwise, the raw bytestring of the negotiated protocol will be returned. If ``Context.set_inner_protocols()`` was not called, if the other party does not support protocol negotiation, if this socket does not support any of the peer's proposed protocols, or if the handshake has not happened yet, ``None`` is returned. """ @property def context(self) -> Context: """ The ``Context`` object this socket is tied to. """ def negotiated_tls_version(self) -> Optional[TLSVersion]: """ The version of TLS that has been negotiated on this connection. """ def unwrap(self) -> socket.socket: """ Cleanly terminate the TLS connection on this wrapped socket. Once called, this ``TLSWrappedSocket`` can no longer be used to transmit data. Returns the socket that was wrapped with TLS. """ Cipher Suites ~~~~~~~~~~~~~ Supporting cipher suites in a truly library-agnostic fashion is a remarkably difficult undertaking. Different TLS implementations often have *radically* different APIs for specifying cipher suites, but more problematically these APIs frequently differ in capability as well as in style. Some examples are shown below: OpenSSL ^^^^^^^ OpenSSL uses a well-known cipher string format. This format has been adopted as a configuration language by most products that use OpenSSL, including Python. This format is relatively easy to read, but has a number of downsides: it is a string, which makes it remarkably easy to provide bad inputs; it lacks much detailed validation, meaning that it is possible to configure OpenSSL in a way that doesn't allow it to negotiate any cipher at all; and it allows specifying cipher suites in a number of different ways that make it tricky to parse. The biggest problem with this format is that there is no formal specification for it, meaning that the only way to parse a given string the way OpenSSL would is to get OpenSSL to parse it. OpenSSL's cipher strings can look like this:: 'ECDH+AESGCM:ECDH+CHACHA20:DH+AESGCM:DH+CHACHA20:ECDH+AES256:DH+AES256:ECDH+AES128:DH+AES:RSA+AESGCM:RSA+AES:!aNULL:!eNULL:!MD5' This string demonstrates some of the complexity of the OpenSSL format. For example, it is possible for one entry to specify multiple cipher suites: the entry ``ECDH+AESGCM`` means "all ciphers suites that include both elliptic-curve Diffie-Hellman key exchange and AES in Galois Counter Mode". More explicitly, that will expand to four cipher suites:: "ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256" That makes parsing a complete OpenSSL cipher string extremely tricky. Add to the fact that there are other meta-characters, such as "!" (exclude all cipher suites that match this criterion, even if they would otherwise be included: "!MD5" means that no cipher suites using the MD5 hash algorithm should be included), "-" (exclude matching ciphers if they were already included, but allow them to be re-added later if they get included again), and "+" (include the matching ciphers, but place them at the end of the list), and you get an *extremely* complex format to parse. On top of this complexity it should be noted that the actual result depends on the OpenSSL version, as an OpenSSL cipher string is valid so long as it contains at least one cipher that OpenSSL recognises. OpenSSL also uses different names for its ciphers than the names used in the relevant specifications. See the manual page for ``ciphers(1)`` for more details. The actual API inside OpenSSL for the cipher string is simple:: char *cipher_list = ; int rc = SSL_CTX_set_cipher_list(context, cipher_list); This means that any format that is used by this module must be able to be converted to an OpenSSL cipher string for use with OpenSSL. SecureTransport ^^^^^^^^^^^^^^^ SecureTransport is the macOS system TLS library. This library is substantially more restricted than OpenSSL in many ways, as it has a much more restricted class of users. One of these substantial restrictions is in controlling supported cipher suites. Ciphers in SecureTransport are represented by a C ``enum``. This enum has one entry per cipher suite, with no aggregate entries, meaning that it is not possible to reproduce the meaning of an OpenSSL cipher string like "ECDH+AESGCM" without hand-coding which categories each enum member falls into. However, the names of most of the enum members are in line with the formal names of the cipher suites: that is, the cipher suite that OpenSSL calls "ECDHE-ECDSA-AES256-GCM-SHA384" is called "TLS_ECDHE_ECDHSA_WITH_AES_256_GCM_SHA384" in SecureTransport. The API for configuring cipher suites inside SecureTransport is simple:: SSLCipherSuite ciphers[] = {TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384, ...}; OSStatus status = SSLSetEnabledCiphers(context, ciphers, sizeof(ciphers)); SChannel ^^^^^^^^ SChannel is the Windows system TLS library. SChannel has extremely restrictive support for controlling available TLS cipher suites, and additionally adopts a third method of expressing what TLS cipher suites are supported. Specifically, SChannel defines a set of ``ALG_ID`` constants (C unsigned ints). Each of these constants does not refer to an entire cipher suite, but instead an individual algorithm. Some examples are ``CALG_3DES`` and ``CALG_AES_256``, which refer to the bulk encryption algorithm used in a cipher suite, ``CALG_DH_EPHEM`` and ``CALG_RSA_KEYX`` which refer to part of the key exchange algorithm used in a cipher suite, ``CALG_SHA1`` and ``CALG_MD5`` which refer to the message authentication code used in a cipher suite, and ``CALG_ECDSA`` and ``CALG_RSA_SIGN`` which refer to the signing portions of the key exchange algorithm. This can be thought of as the half of OpenSSL's functionality that SecureTransport doesn't have: SecureTransport only allows specifying exact cipher suites, while SChannel only allows specifying *parts* of the cipher suite, while OpenSSL allows both. Determining which cipher suites are allowed on a given connection is done by providing a pointer to an array of these ``ALG_ID`` constants. This means that any suitable API must allow the Python code to determine which ``ALG_ID`` constants must be provided. Network Security Services (NSS) ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ NSS is Mozilla's crypto and TLS library. It's used in Firefox, Thunderbird, and as alternative to OpenSSL in multiple libraries, e.g. curl. By default, NSS comes with secure configuration of allowed ciphers. On some platforms such as Fedora, the list of enabled ciphers is globally configured in a system policy. Generally, applications should not modify cipher suites unless they have specific reasons to do so. NSS has both process global and per-connection settings for cipher suites. It does not have a concept of SSLContext like OpenSSL. A SSLContext-like behavior can be easily emulated. Specifically, ciphers can be enabled or disabled globally with ```SSL_CipherPrefSetDefault(PRInt32 cipher, PRBool enabled)```, and ```SSL_CipherPrefSet(PRFileDesc *fd, PRInt32 cipher, PRBool enabled)``` for a connection. The cipher ```PRInt32``` number is a signed 32bit integer that directly corresponds to an registered IANA id, e.g. ```0x1301``` is ```TLS_AES_128_GCM_SHA256```. Contrary to OpenSSL, the preference order of ciphers is fixed and cannot be modified at runtime. Like SecureTransport, NSS has no API for aggregated entries. Some consumers of NSS have implemented custom mappings from OpenSSL cipher names and rules to NSS ciphers, e.g. ```mod_nss```. Proposed Interface ^^^^^^^^^^^^^^^^^^ The proposed interface for the new module is influenced by the combined set of limitations of the above implementations. Specifically, as every implementation *except* OpenSSL requires that each individual cipher be provided, there is no option but to provide that lowest-common denominator approach. The simplest approach is to provide an enumerated type that includes a large subset of the cipher suites defined for TLS. The values of the enum members will be their two-octet cipher identifier as used in the TLS handshake, stored as a 16 bit integer. The names of the enum members will be their IANA-registered cipher suite names. As of now, the `IANA cipher suite registry`_ contains over 320 cipher suites. A large portion of the cipher suites are irrelevant for TLS connections to network services. Other suites specify deprecated and insecure algorithms that are no longer provided by recent versions of implementations. The enum does not contain ciphers with: * key exchange: NULL, Kerberos (KRB5), pre-shared key (PSK), secure remote transport (TLS-SRP) * authentication: NULL, anonymous, export grade, Kerberos (KRB5), pre-shared key (PSK), secure remote transport (TLS-SRP), DSA cert (DSS) * encryption: NULL, ARIA, DES, RC2, export grade 40bit * PRF: MD5 * SCSV cipher suites 3DES, RC4, SEED, and IDEA are included for legacy applications. Further more five additional cipher suites from the TLS 1.3 draft (draft-ietf-tls-tls13-18) are included, too. TLS 1.3 does not share any cipher suites with TLS 1.2 and earlier. The resulting enum will contain roughly 110 suites. Because of these limitations, and because the enum doesn't contain every defined cipher, and also to allow for forward-looking applications, all parts of this API that accept ``CipherSuite`` objects will also accept raw 16-bit integers directly. Rather than populate this enum by hand, we have a `TLS enum script`_ that builds it from Christian Heimes' `tlsdb JSON file`_ (warning: large file) and `IANA cipher suite registry`_. The TLSDB also opens up the possibility of extending the API with additional querying function, such as determining which TLS versions support which ciphers, if that functionality is found to be useful or necessary. If users find this approach to be onerous, a future extension to this API can provide helpers that can reintroduce OpenSSL's aggregation functionality. :: class CipherSuite(IntEnum): TLS_RSA_WITH_RC4_128_SHA = 0x0005 TLS_RSA_WITH_IDEA_CBC_SHA = 0x0007 TLS_RSA_WITH_3DES_EDE_CBC_SHA = 0x000a TLS_DH_RSA_WITH_3DES_EDE_CBC_SHA = 0x0010 TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA = 0x0016 TLS_RSA_WITH_AES_128_CBC_SHA = 0x002f TLS_DH_RSA_WITH_AES_128_CBC_SHA = 0x0031 TLS_DHE_RSA_WITH_AES_128_CBC_SHA = 0x0033 TLS_RSA_WITH_AES_256_CBC_SHA = 0x0035 TLS_DH_RSA_WITH_AES_256_CBC_SHA = 0x0037 TLS_DHE_RSA_WITH_AES_256_CBC_SHA = 0x0039 TLS_RSA_WITH_AES_128_CBC_SHA256 = 0x003c TLS_RSA_WITH_AES_256_CBC_SHA256 = 0x003d TLS_DH_RSA_WITH_AES_128_CBC_SHA256 = 0x003f TLS_RSA_WITH_CAMELLIA_128_CBC_SHA = 0x0041 TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA = 0x0043 TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA = 0x0045 TLS_DHE_RSA_WITH_AES_128_CBC_SHA256 = 0x0067 TLS_DH_RSA_WITH_AES_256_CBC_SHA256 = 0x0069 TLS_DHE_RSA_WITH_AES_256_CBC_SHA256 = 0x006b TLS_RSA_WITH_CAMELLIA_256_CBC_SHA = 0x0084 TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA = 0x0086 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA = 0x0088 TLS_RSA_WITH_SEED_CBC_SHA = 0x0096 TLS_DH_RSA_WITH_SEED_CBC_SHA = 0x0098 TLS_DHE_RSA_WITH_SEED_CBC_SHA = 0x009a TLS_RSA_WITH_AES_128_GCM_SHA256 = 0x009c TLS_RSA_WITH_AES_256_GCM_SHA384 = 0x009d TLS_DHE_RSA_WITH_AES_128_GCM_SHA256 = 0x009e TLS_DHE_RSA_WITH_AES_256_GCM_SHA384 = 0x009f TLS_DH_RSA_WITH_AES_128_GCM_SHA256 = 0x00a0 TLS_DH_RSA_WITH_AES_256_GCM_SHA384 = 0x00a1 TLS_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0x00ba TLS_DH_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0x00bc TLS_DHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0x00be TLS_RSA_WITH_CAMELLIA_256_CBC_SHA256 = 0x00c0 TLS_DH_RSA_WITH_CAMELLIA_256_CBC_SHA256 = 0x00c2 TLS_DHE_RSA_WITH_CAMELLIA_256_CBC_SHA256 = 0x00c4 TLS_AES_128_GCM_SHA256 = 0x1301 TLS_AES_256_GCM_SHA384 = 0x1302 TLS_CHACHA20_POLY1305_SHA256 = 0x1303 TLS_AES_128_CCM_SHA256 = 0x1304 TLS_AES_128_CCM_8_SHA256 = 0x1305 TLS_ECDH_ECDSA_WITH_RC4_128_SHA = 0xc002 TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA = 0xc003 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA = 0xc004 TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA = 0xc005 TLS_ECDHE_ECDSA_WITH_RC4_128_SHA = 0xc007 TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA = 0xc008 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA = 0xc009 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA = 0xc00a TLS_ECDH_RSA_WITH_RC4_128_SHA = 0xc00c TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA = 0xc00d TLS_ECDH_RSA_WITH_AES_128_CBC_SHA = 0xc00e TLS_ECDH_RSA_WITH_AES_256_CBC_SHA = 0xc00f TLS_ECDHE_RSA_WITH_RC4_128_SHA = 0xc011 TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA = 0xc012 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA = 0xc013 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA = 0xc014 TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256 = 0xc023 TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384 = 0xc024 TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256 = 0xc025 TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384 = 0xc026 TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256 = 0xc027 TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384 = 0xc028 TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256 = 0xc029 TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384 = 0xc02a TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256 = 0xc02b TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384 = 0xc02c TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256 = 0xc02d TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384 = 0xc02e TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256 = 0xc02f TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384 = 0xc030 TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256 = 0xc031 TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384 = 0xc032 TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_CBC_SHA256 = 0xc072 TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_CBC_SHA384 = 0xc073 TLS_ECDH_ECDSA_WITH_CAMELLIA_128_CBC_SHA256 = 0xc074 TLS_ECDH_ECDSA_WITH_CAMELLIA_256_CBC_SHA384 = 0xc075 TLS_ECDHE_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0xc076 TLS_ECDHE_RSA_WITH_CAMELLIA_256_CBC_SHA384 = 0xc077 TLS_ECDH_RSA_WITH_CAMELLIA_128_CBC_SHA256 = 0xc078 TLS_ECDH_RSA_WITH_CAMELLIA_256_CBC_SHA384 = 0xc079 TLS_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc07a TLS_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc07b TLS_DHE_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc07c TLS_DHE_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc07d TLS_DH_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc07e TLS_DH_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc07f TLS_ECDHE_ECDSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc086 TLS_ECDHE_ECDSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc087 TLS_ECDH_ECDSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc088 TLS_ECDH_ECDSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc089 TLS_ECDHE_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc08a TLS_ECDHE_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc08b TLS_ECDH_RSA_WITH_CAMELLIA_128_GCM_SHA256 = 0xc08c TLS_ECDH_RSA_WITH_CAMELLIA_256_GCM_SHA384 = 0xc08d TLS_RSA_WITH_AES_128_CCM = 0xc09c TLS_RSA_WITH_AES_256_CCM = 0xc09d TLS_DHE_RSA_WITH_AES_128_CCM = 0xc09e TLS_DHE_RSA_WITH_AES_256_CCM = 0xc09f TLS_RSA_WITH_AES_128_CCM_8 = 0xc0a0 TLS_RSA_WITH_AES_256_CCM_8 = 0xc0a1 TLS_DHE_RSA_WITH_AES_128_CCM_8 = 0xc0a2 TLS_DHE_RSA_WITH_AES_256_CCM_8 = 0xc0a3 TLS_ECDHE_ECDSA_WITH_AES_128_CCM = 0xc0ac TLS_ECDHE_ECDSA_WITH_AES_256_CCM = 0xc0ad TLS_ECDHE_ECDSA_WITH_AES_128_CCM_8 = 0xc0ae TLS_ECDHE_ECDSA_WITH_AES_256_CCM_8 = 0xc0af TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256 = 0xcca8 TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256 = 0xcca9 TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256 = 0xccaa Enum members can be mapped to OpenSSL cipher names:: >>> import ssl >>> ctx = ssl.SSLContext(ssl.PROTOCOL_TLS) >>> ctx.set_ciphers('ALL:COMPLEMENTOFALL') >>> ciphers = {c['id'] & 0xffff: c['name'] for c in ctx.get_ciphers()} >>> ciphers[CipherSuite.TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256] 'ECDHE-RSA-AES128-GCM-SHA256' For SecureTransport, these enum members directly refer to the values of the cipher suite constants. For example, SecureTransport defines the cipher suite enum member ``TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384`` as having the value ``0xC02C``. Not coincidentally, that is identical to its value in the above enum. This makes mapping between SecureTransport and the above enum very easy indeed. For SChannel there is no easy direct mapping, due to the fact that SChannel configures ciphers, instead of cipher suites. This represents an ongoing concern with SChannel, which is that it is very difficult to configure in a specific manner compared to other TLS implementations. For the purposes of this PEP, any SChannel implementation will need to determine which ciphers to choose based on the enum members. This may be more open than the actual cipher suite list actually wants to allow, or it may be more restrictive, depending on the choices of the implementation. This PEP recommends that it be more restrictive, but of course this cannot be enforced. Protocol Negotiation ~~~~~~~~~~~~~~~~~~~~ Both NPN and ALPN allow for protocol negotiation as part of the HTTP/2 handshake. While NPN and ALPN are, at their fundamental level, built on top of bytestrings, string-based APIs are frequently problematic as they allow for errors in typing that can be hard to detect. For this reason, this module would define a type that protocol negotiation implementations can pass and be passed. This type would wrap a bytestring to allow for aliases for well-known protocols. This allows us to avoid the problems inherent in typos for well-known protocols, while allowing the full extensibility of the protocol negotiation layer if needed by letting users pass byte strings directly. :: class NextProtocol(Enum): H2 = b'h2' H2C = b'h2c' HTTP1 = b'http/1.1' WEBRTC = b'webrtc' C_WEBRTC = b'c-webrtc' FTP = b'ftp' STUN = b'stun.nat-discovery' TURN = b'stun.turn' TLS Versions ~~~~~~~~~~~~ It is often useful to be able to restrict the versions of TLS you're willing to support. There are many security advantages in refusing to use old versions of TLS, and some misbehaving servers will mishandle TLS clients advertising support for newer versions. The following enumerated type can be used to gate TLS versions. Forward-looking applications should almost never set a maximum TLS version unless they absolutely must, as a TLS backend that is newer than the Python that uses it may support TLS versions that are not in this enumerated type. Additionally, this enumerated type defines two additional flags that can always be used to request either the lowest or highest TLS version supported by an implementation. :: class TLSVersion(Enum): MINIMUM_SUPPORTED = auto() SSLv2 = auto() SSLv3 = auto() TLSv1 = auto() TLSv1_1 = auto() TLSv1_2 = auto() TLSv1_3 = auto() MAXIMUM_SUPPORTED = auto() Errors ~~~~~~ This module would define four base classes for use with error handling. Unlike many of the the other classes defined here, these classes are not abstract, as they have no behaviour. They exist simply to signal certain common behaviours. Backends should subclass these exceptions in their own packages, but needn't define any behaviour for them. In general, concrete implementations should subclass these exceptions rather than throw them directly. This makes it moderately easier to determine which concrete TLS implementation is in use during debugging of unexpected errors. However, this is not mandatory. The definitions of the errors are below:: class TLSError(Exception): """ The base exception for all TLS related errors from any backend. Catching this error should be sufficient to catch *all* TLS errors, regardless of what backend is used. """ class WantWriteError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is written to the network, or until the output buffer is drained. This error is should only be raised when it is completely impossible to write any data. If a partial write is achievable then this should not be raised. """ class WantReadError(TLSError): """ A special signaling exception used only when non-blocking or buffer-only I/O is used. This error signals that the requested operation cannot complete until more data is read from the network, or until more data is available in the input buffer. This error should only be raised when it is completely impossible to write any data. If a partial write is achievable then this should not be raised. """ class RaggedEOF(TLSError): """ A special signaling exception used when a TLS connection has been closed gracelessly: that is, when a TLS CloseNotify was not received from the peer before the underlying TCP socket reached EOF. This is a so-called "ragged EOF". This exception is not guaranteed to be raised in the face of a ragged EOF: some implementations may not be able to detect or report the ragged EOF. This exception is not always a problem. Ragged EOFs are a concern only when protocols are vulnerable to length truncation attacks. Any protocol that can detect length truncation attacks at the application layer (e.g. HTTP/1.1 and HTTP/2) is not vulnerable to this kind of attack and so can ignore this exception. """ Certificates ~~~~~~~~~~~~ This module would define an abstract X509 certificate class. This class would have almost no behaviour, as the goal of this module is not to provide all possible relevant cryptographic functionality that could be provided by X509 certificates. Instead, all we need is the ability to signal the source of a certificate to a concrete implementation. For that reason, this certificate implementation defines only constructors. In essence, the certificate object in this module could be as abstract as a handle that can be used to locate a specific certificate. Concrete implementations may choose to provide alternative constructors, e.g. to load certificates from HSMs. If a common interface emerges for doing this, this module may be updated to provide a standard constructor for this use-case as well. Concrete implementations should aim to have Certificate objects be hashable if at all possible. This will help ensure that TLSConfiguration objects used with an individual concrete implementation are also hashable. :: class Certificate(metaclass=ABCMeta): @abstractclassmethod def from_buffer(cls, buffer: bytes): """ Creates a Certificate object from a byte buffer. This byte buffer may be either PEM-encoded or DER-encoded. If the buffer is PEM encoded it *must* begin with the standard PEM preamble (a series of dashes followed by the ASCII bytes "BEGIN CERTIFICATE" and another series of dashes). In the absence of that preamble, the implementation may assume that the certificate is DER-encoded instead. """ @abstractclassmethod def from_file(cls, path: Union[pathlib.Path, AnyStr]): """ Creates a Certificate object from a file on disk. This method may be a convenience method that wraps ``open`` and ``from_buffer``, but some TLS implementations may be able to provide more-secure or faster methods of loading certificates that do not involve Python code. """ Private Keys ~~~~~~~~~~~~ This module would define an abstract private key class. Much like the Certificate class, this class has almost no behaviour in order to give as much freedom as possible to the concrete implementations to treat keys carefully. This class has all the caveats of the ``Certificate`` class. :: class PrivateKey(metaclass=ABCMeta): @abstractclassmethod def from_buffer(cls, buffer: bytes, password: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]] = None): """ Creates a PrivateKey object from a byte buffer. This byte buffer may be either PEM-encoded or DER-encoded. If the buffer is PEM encoded it *must* begin with the standard PEM preamble (a series of dashes followed by the ASCII bytes "BEGIN", the key type, and another series of dashes). In the absence of that preamble, the implementation may assume that the certificate is DER-encoded instead. The key may additionally be encrypted. If it is, the ``password`` argument can be used to decrypt the key. The ``password`` argument may be a function to call to get the password for decrypting the private key. It will only be called if the private key is encrypted and a password is necessary. It will be called with no arguments, and it should return either bytes or bytearray containing the password. Alternatively a bytes, or bytearray value may be supplied directly as the password argument. It will be ignored if the private key is not encrypted and no password is needed. """ @abstractclassmethod def from_file(cls, path: Union[pathlib.Path, bytes, str], password: Optional[Union[Callable[[], Union[bytes, bytearray]], bytes, bytearray]] = None): """ Creates a PrivateKey object from a file on disk. This method may be a convenience method that wraps ``open`` and ``from_buffer``, but some TLS implementations may be able to provide more-secure or faster methods of loading certificates that do not involve Python code. The ``password`` parameter behaves exactly as the equivalent parameter on ``from_buffer``. """ Trust Store ~~~~~~~~~~~ As discussed above, loading a trust store represents an issue because different TLS implementations vary wildly in how they allow users to select trust stores. For this reason, we need to provide a model that assumes very little about the form that trust stores take. This problem is the same as the one that the Certificate and PrivateKey types need to solve. For this reason, we use the exact same model, by creating an opaque type that can encapsulate the various means that TLS backends may open a trust store. A given TLS implementation is not required to implement all of the constructors. However, it is strongly recommended that a given TLS implementation provide the ``system`` constructor if at all possible, as this is the most common validation trust store that is used. Concrete implementations may also add their own constructors. Concrete implementations should aim to have TrustStore objects be hashable if at all possible. This will help ensure that TLSConfiguration objects used with an individual concrete implementation are also hashable. :: class TrustStore(metaclass=ABCMeta): @abstractclassmethod def system(cls) -> TrustStore: """ Returns a TrustStore object that represents the system trust database. """ @abstractclassmethod def from_pem_file(cls, path: Union[pathlib.Path, bytes, str]) -> TrustStore: """ Initializes a trust store from a single file full of PEMs. """ Runtime Access ~~~~~~~~~~~~~~ A not-uncommon use case for library users is to want to allow the library to control the TLS configuration, but to want to select what backend is in use. For example, users of Requests may want to be able to select between OpenSSL or a platform-native solution on Windows and macOS, or between OpenSSL and NSS on some Linux platforms. These users, however, may not care about exactly how their TLS configuration is done. This poses a problem: given an arbitrary concrete implementation, how can a library work out how to load certificates into the trust store? There are two options: either all concrete implementations can be required to fit into a specific naming scheme, or we can provide an API that makes it possible to grab these objects. This PEP proposes that we use the second approach. This grants the greatest freedom to concrete implementations to structure their code as they see fit, requiring only that they provide a single object that has the appropriate properties in place. Users can then pass this "backend" object to libraries that support it, and those libraries can take care of configuring and using the concrete implementation. All concrete implementations must provide a method of obtaining a ``Backend`` object. The ``Backend`` object can be a global singleton or can be created by a callable if there is an advantage in doing that. The ``Backend`` object has the following definition:: Backend = namedtuple( 'Backend', ['client_context', 'server_context', 'certificate', 'private_key', 'trust_store'] ) Each of the properties must provide the concrete implementation of the relevant ABC. This ensures that code like this will work for any backend:: trust_store = backend.trust_store.system() Changes to the Standard Library =============================== The portions of the standard library that interact with TLS should be revised to use these ABCs. This will allow them to function with other TLS backends. This includes the following modules: - asyncio - ftplib - http - imaplib - nntplib - poplib - smtplib - urllib Migration of the ssl module --------------------------- Naturally, we will need to extend the ``ssl`` module itself to conform to these ABCs. This extension will take the form of new classes, potentially in an entirely new module. This will allow applications that take advantage of the current ``ssl`` module to continue to do so, while enabling the new APIs for applications and libraries that want to use them. In general, migrating from the ``ssl`` module to the new ABCs is not expected to be one-to-one. This is normally acceptable: most tools that use the ``ssl`` module hide it from the user, and so refactoring to use the new module should be invisible. However, a specific problem comes from libraries or applications that leak exceptions from the ``ssl`` module, either as part of their defined API or by accident (which is easily done). Users of those tools may have written code that tolerates and handles exceptions from the ``ssl`` module being raised: migrating to the ABCs presented here would potentially cause the exceptions defined above to be thrown instead, and existing ``except`` blocks will not catch them. For this reason, part of the migration of the ``ssl`` module would require that the exceptions in the ``ssl`` module alias those defined above. That is, they would require the following statements to all succeed:: assert ssl.SSLError is tls.TLSError assert ssl.SSLWantReadError is tls.WantReadError assert ssl.SSLWantWriteError is tls.WantWriteError The exact mechanics of how this will be done are beyond the scope of this PEP, as they are made more complex due to the fact that the current ``ssl`` exceptions are defined in C code, but more details can be found in `an email sent to the Security-SIG by Christian Heimes`_. Future ====== Major future TLS features may require revisions of these ABCs. These revisions should be made cautiously: many backends may not be able to move forward swiftly, and will be invalidated by changes in these ABCs. This is acceptable, but wherever possible features that are specific to individual implementations should not be added to the ABCs. The ABCs should restrict themselves to high-level descriptions of IETF-specified features. However, well-justified extensions to this API absolutely should be made. The focus of this API is to provide a unifying lowest-common-denominator configuration option for the Python community. TLS is not a static target, and as TLS evolves so must this API. Credits ======= This document has received extensive review from a number of individuals in the community who have substantially helped shape it. Detailed review was provided by: * Alex Chan * Alex Gaynor * Antoine Pitrou * Ashwini Oruganti * Donald Stufft * Ethan Furman * Glyph * Hynek Schlawack * Jim J Jewett * Nathaniel J. Smith * Nick Coghlan * Paul Kehrer * Steve Dower * Steven Fackler * Wes Turner * Will Bond Further review was provided by the Security-SIG and python-ideas mailing lists. Copyright ========= This document has been placed in the public domain. .. _ssl module: https://docs.python.org/3/library/ssl.html .. _OpenSSL Library: https://www.openssl.org/ .. _PyOpenSSL: https://pypi.org/project/pyOpenSSL/ .. _certifi: https://pypi.org/project/certifi/ .. _SSLContext: https://docs.python.org/3/library/ssl.html#ssl.SSLContext .. _SSLSocket: https://docs.python.org/3/library/ssl.html#ssl.SSLSocket .. _SSLObject: https://docs.python.org/3/library/ssl.html#ssl.SSLObject .. _SSLError: https://docs.python.org/3/library/ssl.html#ssl.SSLError .. _MSDN articles: https://msdn.microsoft.com/en-us/library/windows/desktop/mt490158(v=vs.85).aspx .. _TLS enum script: https://github.com/tiran/tlsdb/blob/master/tlspep_ciphersuite.py .. _tlsdb JSON file: https://github.com/tiran/tlsdb/blob/master/tlsdb.json .. _IANA cipher suite registry: https://www.iana.org/assignments/tls-parameters/tls-parameters.xhtml#tls-parameters-4 .. _an email sent to the Security-SIG by Christian Heimes: https://mail.python.org/pipermail/security-sig/2017-January/000213.html .. _s2n: https://github.com/awslabs/s2n .. _working to add it: https://github.com/awslabs/s2n/issues/358 .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From barry at python.org Mon Feb 13 10:10:26 2017 From: barry at python.org (Barry Warsaw) Date: Mon, 13 Feb 2017 10:10:26 -0500 Subject: [Python-Dev] [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... In-Reply-To: <20170212011136.GA17693@gazelle> References: <589e61bcbaba_7f783fb29586dc2449911@hookshot-fe2-cp1-prd.iad.github.net.mail> <20170212011136.GA17693@gazelle> Message-ID: <20170213101026.4cee089a@subdivisions.wooz.org> On Feb 11, 2017, at 08:11 PM, Andrew M. Kuchling wrote: >Are we planning to include the diffs in python-checkins e-mail, or are >they gone for good? > >(While they made the list's messages and digests larger, I liked >having the diffs because you could page through them to see what >changes were made without having to go through to another site.) Yep, this was recently discussed and I think the plan is to bring them back for python-checkins, but that it's not possible to enable them on PRs due to GH limitations. Cheers, -Barry From brett at snarky.ca Mon Feb 13 12:54:51 2017 From: brett at snarky.ca (Brett Cannon) Date: Mon, 13 Feb 2017 17:54:51 +0000 Subject: [Python-Dev] [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... In-Reply-To: <20170213101026.4cee089a@subdivisions.wooz.org> References: <589e61bcbaba_7f783fb29586dc2449911@hookshot-fe2-cp1-prd.iad.github.net.mail> <20170212011136.GA17693@gazelle> <20170213101026.4cee089a@subdivisions.wooz.org> Message-ID: On Mon, 13 Feb 2017 at 07:10 Barry Warsaw wrote: > On Feb 11, 2017, at 08:11 PM, Andrew M. Kuchling wrote: > > >Are we planning to include the diffs in python-checkins e-mail, or are > >they gone for good? > > > >(While they made the list's messages and digests larger, I liked > >having the diffs because you could page through them to see what > >changes were made without having to go through to another site.) > > Yep, this was recently discussed and I think the plan is to bring them back > for python-checkins, but that it's not possible to enable them on PRs due > to > GH limitations. > Serhiy asked about it and I said that the email integration we are using from GitHub doesn't allow for customizing the email it sends out beyond how is listed in the From line. If people find a different integration that they want to use for sending email then that can be set up. -------------- next part -------------- An HTML attachment was scrubbed... URL: From berker.peksag at gmail.com Mon Feb 13 13:25:34 2017 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Mon, 13 Feb 2017 21:25:34 +0300 Subject: [Python-Dev] [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... In-Reply-To: References: <589e61bcbaba_7f783fb29586dc2449911@hookshot-fe2-cp1-prd.iad.github.net.mail> <20170212011136.GA17693@gazelle> <20170213101026.4cee089a@subdivisions.wooz.org> Message-ID: --Berker On Mon, Feb 13, 2017 at 8:54 PM, Brett Cannon wrote: > > > On Mon, 13 Feb 2017 at 07:10 Barry Warsaw wrote: >> >> On Feb 11, 2017, at 08:11 PM, Andrew M. Kuchling wrote: >> >> >Are we planning to include the diffs in python-checkins e-mail, or are >> >they gone for good? >> > >> >(While they made the list's messages and digests larger, I liked >> >having the diffs because you could page through them to see what >> >changes were made without having to go through to another site.) >> >> Yep, this was recently discussed and I think the plan is to bring them >> back >> for python-checkins, but that it's not possible to enable them on PRs due >> to >> GH limitations. > > > Serhiy asked about it and I said that the email integration we are using > from GitHub doesn't allow for customizing the email it sends out beyond how > is listed in the From line. If people find a different integration that they > want to use for sending email then that can be set up. I wrote a webhook that includes diffs: An example output can be seen at https://github.com/python/core-workflow/issues/24#issuecomment-279162079 Since it listens all 'push' events, it's possible to catch all commits and send them to python-checkins. --Berker From esr at thyrsus.com Mon Feb 13 14:05:55 2017 From: esr at thyrsus.com (Eric S. Raymond) Date: Mon, 13 Feb 2017 14:05:55 -0500 (EST) Subject: [Python-Dev] Heads-up, I've rejoined Message-ID: <20170213190555.7073113A019A@snark.thyrsus.com> Some of the older Pythonistas will remember my previous time on this list, and possibly that I faded away quietly under time pressure from other projects and the whole being-famous nonsense. I'm back, for now primarily to listen. I have some questions about the future direction of Python which I'll bring up when I have time for the vigorous discussion that will certainly ensue. -- >>esr>> From brett at python.org Mon Feb 13 15:08:55 2017 From: brett at python.org (Brett Cannon) Date: Mon, 13 Feb 2017 20:08:55 +0000 Subject: [Python-Dev] Update on labels to be used on GitHub Message-ID: We now have two sets of labels for representing cherry-picking statuses: "backport to N.M" and "cherry-pick for N.M". The former are to help keep track of what branches a PR should be cherry-picked *to* and can be removed once the PR for a cherry-pick has been created (you can add a comment like "cherry-picked from GH-NN" to make the link between the PRs)*.* The latter labels are to signify that the PR is a cherry-pick *itself*. As with all aspects of the new GitHub workflow, we will give this a solid try for about a month and then we can re-evaluate whether we like this approach or want a more traditional prefix format in the title to represent the same thing. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Feb 13 17:35:58 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 13 Feb 2017 23:35:58 +0100 Subject: [Python-Dev] Update on labels to be used on GitHub In-Reply-To: References: Message-ID: 2017-02-13 21:08 GMT+01:00 Brett Cannon : > We now have two sets of labels for representing cherry-picking statuses: > "backport to N.M" and "cherry-pick for N.M". The former are to help keep > track of what branches a PR should be cherry-picked to and can be removed > once the PR for a cherry-pick has been created (you can add a comment like > "cherry-picked from GH-NN" to make the link between the PRs). The first time I looked at the "backport to 3.6" flag, I understood that the change was written on the 3.6 branch and asked to rebase the change on 3.6. If I was confused, I'm sure that someone else will be confused as well :-) What do you think of renaming the label to "need 3.6 backport" (or "need backport to 3.6")? Victor From brett at python.org Tue Feb 14 12:32:37 2017 From: brett at python.org (Brett Cannon) Date: Tue, 14 Feb 2017 17:32:37 +0000 Subject: [Python-Dev] Update on labels to be used on GitHub In-Reply-To: References: Message-ID: On Mon, 13 Feb 2017 at 14:43 Victor Stinner wrote: > 2017-02-13 21:08 GMT+01:00 Brett Cannon : > > We now have two sets of labels for representing cherry-picking statuses: > > "backport to N.M" and "cherry-pick for N.M". The former are to help keep > > track of what branches a PR should be cherry-picked to and can be removed > > once the PR for a cherry-pick has been created (you can add a comment > like > > "cherry-picked from GH-NN" to make the link between the PRs). > > The first time I looked at the "backport to 3.6" flag, I understood > that the change was written on the 3.6 branch and asked to rebase the > change on 3.6. If I was confused, I'm sure that someone else will be > confused as well :-) > > What do you think of renaming the label to "need 3.6 backport" (or > "need backport to 3.6")? > That's fine by me. Anyone else have an opinion? -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Tue Feb 14 12:43:09 2017 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Tue, 14 Feb 2017 11:43:09 -0600 Subject: [Python-Dev] Update on labels to be used on GitHub In-Reply-To: References: Message-ID: On Tue, Feb 14, 2017 at 11:32 AM, Brett Cannon wrote: > On Mon, 13 Feb 2017 at 14:43 Victor Stinner > wrote: >> >> 2017-02-13 21:08 GMT+01:00 Brett Cannon : >> > We now have two sets of labels for representing cherry-picking statuses: >> > "backport to N.M" and "cherry-pick for N.M". The former are to help keep >> > track of what branches a PR should be cherry-picked to and can be >> > removed >> > once the PR for a cherry-pick has been created (you can add a comment >> > like >> > "cherry-picked from GH-NN" to make the link between the PRs). >> >> The first time I looked at the "backport to 3.6" flag, I understood >> that the change was written on the 3.6 branch and asked to rebase the >> change on 3.6. If I was confused, I'm sure that someone else will be >> confused as well :-) >> >> What do you think of renaming the label to "need 3.6 backport" (or >> "need backport to 3.6")? > > > That's fine by me. Anyone else have an opinion? To propose another color for the bikeshed, what about having labels for each branch along with "needs backport" and "cherry-pick" labels? PRs that should be backported get "needs backport" and each branch that should get the patch, and cherry-pick PRs get exactly one branch label and the "cherry-pick" label. I think this would make it easier to see which branch a particular PR is against from the PR list. -- Zach From victor.stinner at gmail.com Tue Feb 14 12:58:17 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 14 Feb 2017 18:58:17 +0100 Subject: [Python-Dev] Update on labels to be used on GitHub In-Reply-To: References: Message-ID: 2017-02-14 18:43 GMT+01:00 Zachary Ware : > To propose another color for the bikeshed, what about having labels > for each branch along with "needs backport" and "cherry-pick" labels? Oh right, labels have colors. I propose pink for "needs backport" and yellow for "cherry-pick"! Just kidding, sorry ;-) Victor From python at mrabarnett.plus.com Tue Feb 14 13:12:42 2017 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 14 Feb 2017 18:12:42 +0000 Subject: [Python-Dev] Update on labels to be used on GitHub In-Reply-To: References: Message-ID: <0d036738-2a47-0538-2f3f-9353ab5931c1@mrabarnett.plus.com> On 2017-02-14 17:58, Victor Stinner wrote: > 2017-02-14 18:43 GMT+01:00 Zachary Ware : >> To propose another color for the bikeshed, what about having labels >> for each branch along with "needs backport" and "cherry-pick" labels? > > Oh right, labels have colors. I propose pink for "needs backport" and > yellow for "cherry-pick"! > > Just kidding, sorry ;-) > Surely the colour for "cherry-pick" should be the colour of cherries! :-) From brett at python.org Tue Feb 14 14:29:42 2017 From: brett at python.org (Brett Cannon) Date: Tue, 14 Feb 2017 19:29:42 +0000 Subject: [Python-Dev] Update on labels to be used on GitHub In-Reply-To: <0d036738-2a47-0538-2f3f-9353ab5931c1@mrabarnett.plus.com> References: <0d036738-2a47-0538-2f3f-9353ab5931c1@mrabarnett.plus.com> Message-ID: On Tue, 14 Feb 2017 at 10:13 MRAB wrote: > On 2017-02-14 17:58, Victor Stinner wrote: > > 2017-02-14 18:43 GMT+01:00 Zachary Ware : > >> To propose another color for the bikeshed, what about having labels > >> for each branch along with "needs backport" and "cherry-pick" labels? > > > > Oh right, labels have colors. I propose pink for "needs backport" and > > yellow for "cherry-pick"! > > > > Just kidding, sorry ;-) > > > Surely the colour for "cherry-pick" should be the colour of cherries! :-) > Since the wording keeps throwing people I have renamed "backport to N.M" to "needs backport to N.M". I left the "cherry-pick for N.M" labels but made them a shade of pink (I associate cherry red with "bug" too much to switch over to red entirely :) . -------------- next part -------------- An HTML attachment was scrubbed... URL: From www.2849007069 at outlook.com Tue Feb 14 19:20:05 2017 From: www.2849007069 at outlook.com (=?gb2312?B?s8Igys8=?=) Date: Wed, 15 Feb 2017 00:20:05 +0000 Subject: [Python-Dev] Python-Dev Digest In-Reply-To: References: Message-ID: help ________________________________ From: Python-Dev on behalf of python-dev-request at python.org Sent: Wednesday, February 15, 2017 1:00:05 AM To: python-dev at python.org Subject: Python-Dev Digest, Vol 163, Issue 20 Send Python-Dev mailing list submissions to python-dev at python.org To subscribe or unsubscribe via the World Wide Web, visit https://mail.python.org/mailman/listinfo/python-dev or, via email, send a message with subject or body 'help' to python-dev-request at python.org You can reach the person managing the list at python-dev-owner at python.org When replying, please edit your Subject line so it is more specific than "Re: Contents of Python-Dev digest..." Today's Topics: 1. Re: [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... (Brett Cannon) 2. Re: [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... (Berker Peksa?) 3. Heads-up, I've rejoined (Eric S. Raymond) 4. Update on labels to be used on GitHub (Brett Cannon) 5. Re: Update on labels to be used on GitHub (Victor Stinner) ---------------------------------------------------------------------- Message: 1 Date: Mon, 13 Feb 2017 17:54:51 +0000 From: Brett Cannon To: python-dev at python.org, python-checkins at python.org Cc: amk at amk.ca Subject: Re: [Python-Dev] [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... Message-ID: Content-Type: text/plain; charset="utf-8" On Mon, 13 Feb 2017 at 07:10 Barry Warsaw wrote: > On Feb 11, 2017, at 08:11 PM, Andrew M. Kuchling wrote: > > >Are we planning to include the diffs in python-checkins e-mail, or are > >they gone for good? > > > >(While they made the list's messages and digests larger, I liked > >having the diffs because you could page through them to see what > >changes were made without having to go through to another site.) > > Yep, this was recently discussed and I think the plan is to bring them back > for python-checkins, but that it's not possible to enable them on PRs due > to > GH limitations. > Serhiy asked about it and I said that the email integration we are using from GitHub doesn't allow for customizing the email it sends out beyond how is listed in the From line. If people find a different integration that they want to use for sending email then that can be set up. -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 2 Date: Mon, 13 Feb 2017 21:25:34 +0300 From: Berker Peksa? To: Python Dev , python-checkins at python.org Subject: Re: [Python-Dev] [Python-checkins] [python/cpython] 4538dd: Fix bpo-29528 Use a secure variable to stop spam (... Message-ID: Content-Type: text/plain; charset=UTF-8 --Berker On Mon, Feb 13, 2017 at 8:54 PM, Brett Cannon wrote: > > > On Mon, 13 Feb 2017 at 07:10 Barry Warsaw wrote: >> >> On Feb 11, 2017, at 08:11 PM, Andrew M. Kuchling wrote: >> >> >Are we planning to include the diffs in python-checkins e-mail, or are >> >they gone for good? >> > >> >(While they made the list's messages and digests larger, I liked >> >having the diffs because you could page through them to see what >> >changes were made without having to go through to another site.) >> >> Yep, this was recently discussed and I think the plan is to bring them >> back >> for python-checkins, but that it's not possible to enable them on PRs due >> to >> GH limitations. > > > Serhiy asked about it and I said that the email integration we are using > from GitHub doesn't allow for customizing the email it sends out beyond how > is listed in the From line. If people find a different integration that they > want to use for sending email then that can be set up. I wrote a webhook that includes diffs: An example output can be seen at https://github.com/python/core-workflow/issues/24#issuecomment-279162079 Since it listens all 'push' events, it's possible to catch all commits and send them to python-checkins. --Berker ------------------------------ Message: 3 Date: Mon, 13 Feb 2017 14:05:55 -0500 (EST) From: esr at thyrsus.com (Eric S. Raymond) To: python-dev at python.org Subject: [Python-Dev] Heads-up, I've rejoined Message-ID: <20170213190555.7073113A019A at snark.thyrsus.com> Some of the older Pythonistas will remember my previous time on this list, and possibly that I faded away quietly under time pressure from other projects and the whole being-famous nonsense. I'm back, for now primarily to listen. I have some questions about the future direction of Python which I'll bring up when I have time for the vigorous discussion that will certainly ensue. -- >>esr>> ------------------------------ Message: 4 Date: Mon, 13 Feb 2017 20:08:55 +0000 From: Brett Cannon To: python-dev Subject: [Python-Dev] Update on labels to be used on GitHub Message-ID: Content-Type: text/plain; charset="utf-8" We now have two sets of labels for representing cherry-picking statuses: "backport to N.M" and "cherry-pick for N.M". The former are to help keep track of what branches a PR should be cherry-picked *to* and can be removed once the PR for a cherry-pick has been created (you can add a comment like "cherry-picked from GH-NN" to make the link between the PRs)*.* The latter labels are to signify that the PR is a cherry-pick *itself*. As with all aspects of the new GitHub workflow, we will give this a solid try for about a month and then we can re-evaluate whether we like this approach or want a more traditional prefix format in the title to represent the same thing. -------------- next part -------------- An HTML attachment was scrubbed... URL: ------------------------------ Message: 5 Date: Mon, 13 Feb 2017 23:35:58 +0100 From: Victor Stinner To: Brett Cannon Cc: python-dev Subject: Re: [Python-Dev] Update on labels to be used on GitHub Message-ID: Content-Type: text/plain; charset=UTF-8 2017-02-13 21:08 GMT+01:00 Brett Cannon : > We now have two sets of labels for representing cherry-picking statuses: > "backport to N.M" and "cherry-pick for N.M". The former are to help keep > track of what branches a PR should be cherry-picked to and can be removed > once the PR for a cherry-pick has been created (you can add a comment like > "cherry-picked from GH-NN" to make the link between the PRs). The first time I looked at the "backport to 3.6" flag, I understood that the change was written on the 3.6 branch and asked to rebase the change on 3.6. If I was confused, I'm sure that someone else will be confused as well :-) What do you think of renaming the label to "need 3.6 backport" (or "need backport to 3.6")? Victor ------------------------------ Subject: Digest Footer _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev ------------------------------ End of Python-Dev Digest, Vol 163, Issue 20 ******************************************* -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhoyt at gmail.com Wed Feb 15 11:12:48 2017 From: benhoyt at gmail.com (Ben Hoyt) Date: Wed, 15 Feb 2017 11:12:48 -0500 Subject: [Python-Dev] cgi.FieldStorage with multipart/form-data tries to decode binary file as UTF-8 if "filename=" not specified Message-ID: I posted this on StackOverflow [1], but I'm posting it here as well, as I believe this is a bug (or at least quirk) in cgi.FieldStorage where you can't access a file upload properly if "filename=" is not present in the MIME part's Content-Disposition header. There are a couple of related bugs open (and closed) on bugs.python.ord, but not quite this issue. Is it legitimate for cgi.FieldStorage to use the presence of "filename=" to determine "this is a binary file" (in which case this is not a bug and my client is just buggy), or is this a bug? I lean towards the latter as the spec indicates that the filename is optional [2]. Copying from my StackOverflow question, including a test/repro case: When I use `cgi.FieldStorage` to parse a `multipart/form-data` request (or any web framework like Pyramid which uses `cgi.FieldStorage`) I have trouble processing file uploads from certain clients which don't provide a `filename=file.ext` in the part's `Content-Disposition` header. If the `filename=` option is missing, `FieldStorage()` tries to decode the contents of the file as UTF-8 and return a string. And obviously many files are binary and not UTF-8 and as such give bogus results. For example: >>> import cgi >>> import io >>> body = (b'--KQNTvuH-itP09uVKjjZiegh7\r\n' + ... b'Content-Disposition: form-data; name=payload\r\n\r\n' + ... b'\xff\xd8\xff\xe0\x00\x10JFIF') >>> env = { ... 'REQUEST_METHOD': 'POST', ... 'CONTENT_TYPE': 'multipart/form-data; boundary=KQNTvuH-itP09uVKjjZiegh7', ... 'CONTENT_LENGTH': len(body), ... } >>> fs = cgi.FieldStorage(fp=io.BytesIO(body), environ=env) >>> (fs['payload'].filename, fs['payload'].file.read()) (None, '????\x00\x10JFIF') Browsers, and *most* HTTP libraries do include the `filename=` option for file uploads, but I'm currently dealing with a client that doesn't (and omitting the `filename` does seem to be valid according to the spec). Currently I'm using a pretty hacky workaround by subclassing `FieldStorage` and replacing the relevant `Content-Disposition` header with one that does have the filename: import cgi import os class FileFieldStorage(cgi.FieldStorage): """To use, subclass FileFieldStorage and override _file_fields with a tuple of the names of the file field(s). You can also override _file_name with the filename to add. """ _file_fields = () _file_name = 'file_name' def __init__(self, fp=None, headers=None, outerboundary=b'', environ=os.environ, keep_blank_values=0, strict_parsing=0, limit=None, encoding='utf-8', errors='replace'): if self._file_fields and headers and headers.get('content-disposition'): content_disposition = headers['content-disposition'] key, pdict = cgi.parse_header(content_disposition) if (key == 'form-data' and pdict.get('name') in self._file_fields and 'filename' not in pdict): del headers['content-disposition'] quoted_file_name = self._file_name.replace('"', '\\"') headers['content-disposition'] = '{}; filename="{}"'.format( content_disposition, quoted_file_name) super().__init__(fp=fp, headers=headers, outerboundary=outerboundary, environ=environ, keep_blank_values=keep_blank_values, strict_parsing=strict_parsing, limit=limit, encoding=encoding, errors=errors) Using the `body` and `env` in my first test, this works now: >>> class TestFieldStorage(FileFieldStorage): ... _file_fields = ('payload',) >>> fs = TestFieldStorage(fp=io.BytesIO(body), environ=env) >>> (fs['payload'].filename, fs['payload'].file.read()) ('file_name', b'\xff\xd8\xff\xe0\x00\x10JFIF') Is there some way to avoid this hack and tell `FieldStorage` not to decode as UTF-8? It would be nice if you could provide `encoding=None` or something, but it doesn't look like it supports that. Thanks, Ben. [1] https://stackoverflow.com/questions/42213318/cgi-fieldstorage-with-multipart-form-data-tries-to-decode-binary-file-as-utf-8-e [2] https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1 -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Wed Feb 15 12:35:39 2017 From: brett at python.org (Brett Cannon) Date: Wed, 15 Feb 2017 17:35:39 +0000 Subject: [Python-Dev] cgi.FieldStorage with multipart/form-data tries to decode binary file as UTF-8 if "filename=" not specified In-Reply-To: References: Message-ID: On Wed, 15 Feb 2017 at 08:14 Ben Hoyt wrote: > I posted this on StackOverflow [1], but I'm posting it here as well, as I > believe this is a bug (or at least quirk) in cgi.FieldStorage where you > can't access a file upload properly if "filename=" is not present in the > MIME part's Content-Disposition header. There are a couple of related bugs > open (and closed) on bugs.python.ord, but not quite this issue. > > Is it legitimate for cgi.FieldStorage to use the presence of "filename=" > to determine "this is a binary file" (in which case this is not a bug and > my client is just buggy), or is this a bug? I lean towards the latter as > the spec indicates that the filename is optional [2]. > Assuming this isn't a recent change in semantics I would say this is now a quick considering how old the module is and people probably rely on its current semantics. -Brett > > Copying from my StackOverflow question, including a test/repro case: > > When I use `cgi.FieldStorage` to parse a `multipart/form-data` request (or > any web framework like Pyramid which uses `cgi.FieldStorage`) I have > trouble processing file uploads from certain clients which don't provide a > `filename=file.ext` in the part's `Content-Disposition` header. > > If the `filename=` option is missing, `FieldStorage()` tries to decode the > contents of the file as UTF-8 and return a string. And obviously many files > are binary and not UTF-8 and as such give bogus results. > > For example: > > >>> import cgi > >>> import io > >>> body = (b'--KQNTvuH-itP09uVKjjZiegh7\r\n' + > ... b'Content-Disposition: form-data; name=payload\r\n\r\n' + > ... b'\xff\xd8\xff\xe0\x00\x10JFIF') > >>> env = { > ... 'REQUEST_METHOD': 'POST', > ... 'CONTENT_TYPE': 'multipart/form-data; > boundary=KQNTvuH-itP09uVKjjZiegh7', > ... 'CONTENT_LENGTH': len(body), > ... } > >>> fs = cgi.FieldStorage(fp=io.BytesIO(body), environ=env) > >>> (fs['payload'].filename, fs['payload'].file.read()) > (None, '????\x00\x10JFIF') > > Browsers, and *most* HTTP libraries do include the `filename=` option for > file uploads, but I'm currently dealing with a client that doesn't (and > omitting the `filename` does seem to be valid according to the spec). > > Currently I'm using a pretty hacky workaround by subclassing > `FieldStorage` and replacing the relevant `Content-Disposition` header with > one that does have the filename: > > import cgi > import os > > class FileFieldStorage(cgi.FieldStorage): > """To use, subclass FileFieldStorage and override _file_fields > with a tuple > of the names of the file field(s). You can also override > _file_name with > the filename to add. > """ > > _file_fields = () > _file_name = 'file_name' > > def __init__(self, fp=None, headers=None, outerboundary=b'', > environ=os.environ, keep_blank_values=0, > strict_parsing=0, > limit=None, encoding='utf-8', errors='replace'): > > if self._file_fields and headers and > headers.get('content-disposition'): > content_disposition = headers['content-disposition'] > key, pdict = cgi.parse_header(content_disposition) > if (key == 'form-data' and pdict.get('name') in > self._file_fields and > 'filename' not in pdict): > del headers['content-disposition'] > quoted_file_name = self._file_name.replace('"', '\\"') > headers['content-disposition'] = '{}; > filename="{}"'.format( > content_disposition, quoted_file_name) > > super().__init__(fp=fp, headers=headers, > outerboundary=outerboundary, > environ=environ, > keep_blank_values=keep_blank_values, > strict_parsing=strict_parsing, limit=limit, > encoding=encoding, errors=errors) > > Using the `body` and `env` in my first test, this works now: > > >>> class TestFieldStorage(FileFieldStorage): > ... _file_fields = ('payload',) > >>> fs = TestFieldStorage(fp=io.BytesIO(body), environ=env) > >>> (fs['payload'].filename, fs['payload'].file.read()) > ('file_name', b'\xff\xd8\xff\xe0\x00\x10JFIF') > > Is there some way to avoid this hack and tell `FieldStorage` not to decode > as UTF-8? It would be nice if you could provide `encoding=None` or > something, but it doesn't look like it supports that. > > Thanks, > Ben. > > > [1] > https://stackoverflow.com/questions/42213318/cgi-fieldstorage-with-multipart-form-data-tries-to-decode-binary-file-as-utf-8-e > [2] https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From benhoyt at gmail.com Wed Feb 15 14:41:09 2017 From: benhoyt at gmail.com (Ben Hoyt) Date: Wed, 15 Feb 2017 14:41:09 -0500 Subject: [Python-Dev] cgi.FieldStorage with multipart/form-data tries to decode binary file as UTF-8 if "filename=" not specified In-Reply-To: References: Message-ID: That's reasonable. This isn't an issue on Python 2.x because everything is handled as bytes (str on 2.x). It looks like cgi.FieldStorage() only got Unicode support in Python 3.x at all fairly "late" in the game, for 3.2 or 3.3. It was in that commit (https://github.com/python/cpy thon/commit/5c23b8e6ea7cdb0002842d16dbce4b4d716dd35a by Victor Stinner) where the check for "it's a binary file if it has the 'filename' parameter" was added. It wasn't an issue before that because Unicode wasn't really handled. It seems to me this could be added to FieldStorage pretty easily in a backwards-compatible way, by one of the following: 1) Adding a file_fields keyword argument to FieldStorage.__init__ which would allow you to specify that "these fields are definitely files, please treat them as binary". So in the case of my example on the SO question, you'd do FieldStorage(..., file_fields=('payload',)) 2) Add a .binary_file (or .file_binary) property to FieldStorage which would be like .file but for sure give you a binary file. 3) Allowing you to specify encoding=None, which would mean don't decode fields to str, return everything as bytes. This might not be so nice, as then other non-file fields would be returned as bytes too, and the caller would have to decode those manually. Any thoughts on which of #1-3 might be best? I also realized that to work around this, instead of my fairly complicated content-disposition headers fix, I can subclass FieldStorage and just override "filename" with a property, so that FieldStorage.__init__ thinks that field does have a filename: class MyFieldStorage(cgi.FieldStorage): @property def filename(self): return 'file_name' if self.name == 'payload' else self._filename @filename.setter def filename(self, value): self._filename = value Also, on StackOverflow someone responded that you can also work around this by passing errors='surrogateescape' and then file_input.read().encode('utf-8', 'surrogateescape') ... pretty hacky, but at least you can get the bytes back. -Ben On Wed, Feb 15, 2017 at 12:35 PM, Brett Cannon wrote: > > > On Wed, 15 Feb 2017 at 08:14 Ben Hoyt wrote: > >> I posted this on StackOverflow [1], but I'm posting it here as well, as I >> believe this is a bug (or at least quirk) in cgi.FieldStorage where you >> can't access a file upload properly if "filename=" is not present in the >> MIME part's Content-Disposition header. There are a couple of related bugs >> open (and closed) on bugs.python.ord, but not quite this issue. >> >> Is it legitimate for cgi.FieldStorage to use the presence of "filename=" >> to determine "this is a binary file" (in which case this is not a bug and >> my client is just buggy), or is this a bug? I lean towards the latter as >> the spec indicates that the filename is optional [2]. >> > > Assuming this isn't a recent change in semantics I would say this is now a > quick considering how old the module is and people probably rely on its > current semantics. > > -Brett > > >> >> Copying from my StackOverflow question, including a test/repro case: >> >> When I use `cgi.FieldStorage` to parse a `multipart/form-data` request >> (or any web framework like Pyramid which uses `cgi.FieldStorage`) I have >> trouble processing file uploads from certain clients which don't provide a >> `filename=file.ext` in the part's `Content-Disposition` header. >> >> If the `filename=` option is missing, `FieldStorage()` tries to decode >> the contents of the file as UTF-8 and return a string. And obviously many >> files are binary and not UTF-8 and as such give bogus results. >> >> For example: >> >> >>> import cgi >> >>> import io >> >>> body = (b'--KQNTvuH-itP09uVKjjZiegh7\r\n' + >> ... b'Content-Disposition: form-data; name=payload\r\n\r\n' + >> ... b'\xff\xd8\xff\xe0\x00\x10JFIF') >> >>> env = { >> ... 'REQUEST_METHOD': 'POST', >> ... 'CONTENT_TYPE': 'multipart/form-data; boundary=KQNTvuH- >> itP09uVKjjZiegh7', >> ... 'CONTENT_LENGTH': len(body), >> ... } >> >>> fs = cgi.FieldStorage(fp=io.BytesIO(body), environ=env) >> >>> (fs['payload'].filename, fs['payload'].file.read()) >> (None, '????\x00\x10JFIF') >> >> Browsers, and *most* HTTP libraries do include the `filename=` option for >> file uploads, but I'm currently dealing with a client that doesn't (and >> omitting the `filename` does seem to be valid according to the spec). >> >> Currently I'm using a pretty hacky workaround by subclassing >> `FieldStorage` and replacing the relevant `Content-Disposition` header with >> one that does have the filename: >> >> import cgi >> import os >> >> class FileFieldStorage(cgi.FieldStorage): >> """To use, subclass FileFieldStorage and override _file_fields >> with a tuple >> of the names of the file field(s). You can also override >> _file_name with >> the filename to add. >> """ >> >> _file_fields = () >> _file_name = 'file_name' >> >> def __init__(self, fp=None, headers=None, outerboundary=b'', >> environ=os.environ, keep_blank_values=0, >> strict_parsing=0, >> limit=None, encoding='utf-8', errors='replace'): >> >> if self._file_fields and headers and headers.get('content- >> disposition'): >> content_disposition = headers['content-disposition'] >> key, pdict = cgi.parse_header(content_disposition) >> if (key == 'form-data' and pdict.get('name') in >> self._file_fields and >> 'filename' not in pdict): >> del headers['content-disposition'] >> quoted_file_name = self._file_name.replace('"', '\\"') >> headers['content-disposition'] = '{}; >> filename="{}"'.format( >> content_disposition, quoted_file_name) >> >> super().__init__(fp=fp, headers=headers, >> outerboundary=outerboundary, >> environ=environ, >> keep_blank_values=keep_blank_values, >> strict_parsing=strict_parsing, limit=limit, >> encoding=encoding, errors=errors) >> >> Using the `body` and `env` in my first test, this works now: >> >> >>> class TestFieldStorage(FileFieldStorage): >> ... _file_fields = ('payload',) >> >>> fs = TestFieldStorage(fp=io.BytesIO(body), environ=env) >> >>> (fs['payload'].filename, fs['payload'].file.read()) >> ('file_name', b'\xff\xd8\xff\xe0\x00\x10JFIF') >> >> Is there some way to avoid this hack and tell `FieldStorage` not to >> decode as UTF-8? It would be nice if you could provide `encoding=None` or >> something, but it doesn't look like it supports that. >> >> Thanks, >> Ben. >> >> >> [1] https://stackoverflow.com/questions/42213318/cgi- >> fieldstorage-with-multipart-form-data-tries-to-decode- >> binary-file-as-utf-8-e >> [2] https://www.w3.org/Protocols/rfc2616/rfc2616-sec19.html#sec19.5.1 >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> brett%40python.org >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From freddyrietdijk at fridh.nl Thu Feb 16 12:03:16 2017 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Thu, 16 Feb 2017 18:03:16 +0100 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: Hi, Are there anymore suggestions how to improve the determinism of the Python 3 interpreter? As I mentioned, it seems only sets cause unreproducible bytecode. Sets have no order. But when generating the bytecode, I would expect there would still be an order since the code isn't actually executed, right? On Fri, Feb 10, 2017 at 12:03 PM, INADA Naoki wrote: > On Fri, Feb 10, 2017 at 7:58 PM, Freddy Rietdijk > wrote: > > For Python 3.5 PYTHONHASHSEED doesn't seem to be sufficient, these items > > still seem indeterministic. > > To be sure, I ran `PYTHONHASHSEED=1 $out/bin/python -m compileall -f > $out` > > where $out is the path where I installed Python. > > > > Do you have an idea why in [3], this is Python 2.7, the timestamps are > still > > incorrect? I think they're all required for `compileall` and somehow it > > doesn't seem capable of taking into account DETERMINISTIC_BUILD. > Explicitly > > removing those pyc and pyo files and recompiling them to bytecode still > > results in timestamp issues for these 4 files. > > Sorry, I have no motivation about Python 2 anymore. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From the.hiker at gmail.com Thu Feb 16 15:33:19 2017 From: the.hiker at gmail.com (Patrick Wallinger) Date: Thu, 16 Feb 2017 14:33:19 -0600 Subject: [Python-Dev] Is this a bug or a feature? Message-ID: I'm brand new to python and was trying to install the newest version (3.6.0) on my ubuntu 16.04.1 but couldn't get it to build error free. I tried searching the bugs but didn't see a match for what I am seeing. Thanks for any help, Patrick Here is the report: patrick at ubuntu-MacBook:~/Programs/Python-3.6.0$ ./python -m test -v test_venv == CPython 3.6.0 (default, Feb 16 2017, 13:20:29) [GCC 5.4.0 20160609] == Linux-4.4.0-62-generic-x86_64-with-debian-stretch-sid little-endian == hash algorithm: siphash24 64bit == cwd: /home/patrick/Programs/Python-3.6.0/build/test_python_20536 == encodings: locale=UTF-8, FS=utf-8 Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, hash_randomization=1, isolated=0) Run tests sequentially 0:00:00 [1/1] test_venv test_defaults (test.test_venv.BasicTest) ... ok test_executable (test.test_venv.BasicTest) ... ok test_executable_symlinks (test.test_venv.BasicTest) ... ok test_isolation (test.test_venv.BasicTest) ... ok test_overwrite_existing (test.test_venv.BasicTest) ... ok test_prefixes (test.test_venv.BasicTest) ... ok test_prompt (test.test_venv.BasicTest) ... ok test_symlinking (test.test_venv.BasicTest) ... ok test_unoverwritable_fails (test.test_venv.BasicTest) ... ok test_upgrade (test.test_venv.BasicTest) ... ok test_devnull (test.test_venv.EnsurePipTest) ... ok test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok test_with_pip (test.test_venv.EnsurePipTest) ... FAIL ====================================================================== FAIL: test_with_pip (test.test_venv.EnsurePipTest) ---------------------------------------------------------------------- Traceback (most recent call last): File "/home/patrick/Programs/Python-3.6.0/Lib/test/test_venv.py", line 372, in test_with_pip with_pip=True) subprocess.CalledProcessError: Command '['/tmp/tmp51me85og/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. During handling of the above exception, another exception occurred: Traceback (most recent call last): File "/home/patrick/Programs/Python-3.6.0/Lib/test/test_venv.py", line 378, in test_with_pip self.fail(msg.format(exc, details)) AssertionError: Command '['/tmp/tmp51me85og/bin/python', '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. **Subprocess Output** Traceback (most recent call last): File "/home/patrick/Programs/Python-3.6.0/Lib/runpy.py", line 193, in _run_module_as_main "__main__", mod_spec) File "/home/patrick/Programs/Python-3.6.0/Lib/runpy.py", line 85, in _run_code exec(code, run_globals) File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__main__.py", line 4, in ensurepip._main() File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", line 189, in _main default_pip=args.default_pip, File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", line 102, in bootstrap _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", line 27, in _run_pip import pip zipimport.ZipImportError: can't decompress data; zlib not available ---------------------------------------------------------------------- Ran 14 tests in 1.278s FAILED (failures=1) test test_venv failed test_venv failed 1 test failed: test_venv Total duration: 1 sec Tests result: FAILURE -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Feb 16 16:43:36 2017 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 17 Feb 2017 08:43:36 +1100 Subject: [Python-Dev] Is this a bug or a feature? In-Reply-To: References: Message-ID: On Fri, Feb 17, 2017 at 7:33 AM, Patrick Wallinger wrote: > zipimport.ZipImportError: can't decompress data; zlib not available > There may very well be a bug here of the form of "zlib dependency is considered soft, but then something else breaks". However, in your current situation, my recommendation would be to install the zlib development libraries and retry the build. You may be able to get that with: $ sudo apt-get build-dep python3 or possibly search your package manager for "zlib*-dev" (on my Debian, it's zlib1g-dev). That should give you the full de/compression suite, which will in turn make ensurepip work. ChrisA From phd at phdru.name Thu Feb 16 16:46:43 2017 From: phd at phdru.name (Oleg Broytman) Date: Thu, 16 Feb 2017 22:46:43 +0100 Subject: [Python-Dev] Is this a bug or a feature? In-Reply-To: References: Message-ID: <20170216214643.GA2307@phdru.name> Hello. We are sorry but we cannot help you. This mailing list is to work on developing Python (adding new features to Python itself and fixing bugs); if you're having problems learning, understanding or using Python, please find another forum. Probably python-list/comp.lang.python mailing list/news group is the best place; there are Python developers who participate in it; you may get a faster, and probably more complete, answer there. See http://www.python.org/community/ for other lists/news groups/fora. Thank you for understanding. On Thu, Feb 16, 2017 at 02:33:19PM -0600, Patrick Wallinger wrote: > I'm brand new to python and was trying to install the newest version > (3.6.0) on my ubuntu 16.04.1 but couldn't get it to build error free. I > tried searching the bugs but didn't see a match for what I am seeing. > > Thanks for any help, > Patrick > > Here is the report: > patrick at ubuntu-MacBook:~/Programs/Python-3.6.0$ ./python -m test -v > test_venv [skip] > zipimport.ZipImportError: can't decompress data; zlib not available Seems yoy don't have zlib or haven't had zlib headers when you compiled python. Install zlib and zlib-dev packages and recompile python. Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From the.hiker at gmail.com Thu Feb 16 22:09:34 2017 From: the.hiker at gmail.com (Patrick Wallinger) Date: Thu, 16 Feb 2017 21:09:34 -0600 Subject: [Python-Dev] Is this a bug or a feature? In-Reply-To: <20170216214643.GA2307@phdru.name> References: <20170216214643.GA2307@phdru.name> Message-ID: Oleg, After installing a fresh zlib package from here: http://packages.ubuntu.com/xenial/zlib1g I was able to build and install Python-3.6.0 seems to me if the zlib module sent out with the 3.6.0 release package doesn't build (which it doesn't) then that's a bug. thanks for your time, Patrick On Thu, Feb 16, 2017 at 3:46 PM, Oleg Broytman wrote: > Hello. > > We are sorry but we cannot help you. This mailing list is to work on > developing Python (adding new features to Python itself and fixing bugs); > if you're having problems learning, understanding or using Python, please > find another forum. Probably python-list/comp.lang.python mailing list/news > group is the best place; there are Python developers who participate in it; > you may get a faster, and probably more complete, answer there. See > http://www.python.org/community/ for other lists/news groups/fora. Thank > you for understanding. > > > On Thu, Feb 16, 2017 at 02:33:19PM -0600, Patrick Wallinger < > the.hiker at gmail.com> wrote: > > I'm brand new to python and was trying to install the newest version > > (3.6.0) on my ubuntu 16.04.1 but couldn't get it to build error free. I > > tried searching the bugs but didn't see a match for what I am seeing. > > > > Thanks for any help, > > Patrick > > > > Here is the report: > > patrick at ubuntu-MacBook:~/Programs/Python-3.6.0$ ./python -m test -v > > test_venv > [skip] > > zipimport.ZipImportError: can't decompress data; zlib not available > > Seems yoy don't have zlib or haven't had zlib headers when you > compiled python. Install zlib and zlib-dev packages and recompile > python. > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mal at egenix.com Fri Feb 17 03:56:59 2017 From: mal at egenix.com (M.-A. Lemburg) Date: Fri, 17 Feb 2017 09:56:59 +0100 Subject: [Python-Dev] Is this a bug or a feature? In-Reply-To: References: Message-ID: <477bf160-1897-8945-6fd3-624f0c3dec99@egenix.com> Please report such problems on our bug tracker: http://bugs.python.org/ In your case, Python doesn't appear to find the (right) libz shared library, which is a bit odd, but it's better to track this down on the tracker :-) On 16.02.2017 21:33, Patrick Wallinger wrote: > I'm brand new to python and was trying to install the newest version > (3.6.0) on my ubuntu 16.04.1 but couldn't get it to build error free. I > tried searching the bugs but didn't see a match for what I am seeing. > > Thanks for any help, > Patrick > > Here is the report: > patrick at ubuntu-MacBook:~/Programs/Python-3.6.0$ ./python -m test -v > test_venv > == CPython 3.6.0 (default, Feb 16 2017, 13:20:29) [GCC 5.4.0 20160609] > == Linux-4.4.0-62-generic-x86_64-with-debian-stretch-sid little-endian > == hash algorithm: siphash24 64bit > == cwd: /home/patrick/Programs/Python-3.6.0/build/test_python_20536 > == encodings: locale=UTF-8, FS=utf-8 > Testing with flags: sys.flags(debug=0, inspect=0, interactive=0, > optimize=0, dont_write_bytecode=0, no_user_site=0, no_site=0, > ignore_environment=0, verbose=0, bytes_warning=0, quiet=0, > hash_randomization=1, isolated=0) > Run tests sequentially > 0:00:00 [1/1] test_venv > test_defaults (test.test_venv.BasicTest) ... ok > test_executable (test.test_venv.BasicTest) ... ok > test_executable_symlinks (test.test_venv.BasicTest) ... ok > test_isolation (test.test_venv.BasicTest) ... ok > test_overwrite_existing (test.test_venv.BasicTest) ... ok > test_prefixes (test.test_venv.BasicTest) ... ok > test_prompt (test.test_venv.BasicTest) ... ok > test_symlinking (test.test_venv.BasicTest) ... ok > test_unoverwritable_fails (test.test_venv.BasicTest) ... ok > test_upgrade (test.test_venv.BasicTest) ... ok > test_devnull (test.test_venv.EnsurePipTest) ... ok > test_explicit_no_pip (test.test_venv.EnsurePipTest) ... ok > test_no_pip_by_default (test.test_venv.EnsurePipTest) ... ok > test_with_pip (test.test_venv.EnsurePipTest) ... FAIL > > ====================================================================== > FAIL: test_with_pip (test.test_venv.EnsurePipTest) > ---------------------------------------------------------------------- > Traceback (most recent call last): > File "/home/patrick/Programs/Python-3.6.0/Lib/test/test_venv.py", line > 372, in test_with_pip > with_pip=True) > subprocess.CalledProcessError: Command '['/tmp/tmp51me85og/bin/python', > '-Im', 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit > status 1. > > During handling of the above exception, another exception occurred: > > Traceback (most recent call last): > File "/home/patrick/Programs/Python-3.6.0/Lib/test/test_venv.py", line > 378, in test_with_pip > self.fail(msg.format(exc, details)) > AssertionError: Command '['/tmp/tmp51me85og/bin/python', '-Im', > 'ensurepip', '--upgrade', '--default-pip']' returned non-zero exit status 1. > > **Subprocess Output** > Traceback (most recent call last): > File "/home/patrick/Programs/Python-3.6.0/Lib/runpy.py", line 193, in > _run_module_as_main > "__main__", mod_spec) > File "/home/patrick/Programs/Python-3.6.0/Lib/runpy.py", line 85, in > _run_code > exec(code, run_globals) > File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__main__.py", > line 4, in > ensurepip._main() > File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", > line 189, in _main > default_pip=args.default_pip, > File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", > line 102, in bootstrap > _run_pip(args + [p[0] for p in _PROJECTS], additional_paths) > File "/home/patrick/Programs/Python-3.6.0/Lib/ensurepip/__init__.py", > line 27, in _run_pip > import pip > zipimport.ZipImportError: can't decompress data; zlib not available > > > ---------------------------------------------------------------------- > Ran 14 tests in 1.278s > > FAILED (failures=1) > test test_venv failed > test_venv failed > > 1 test failed: > test_venv > > Total duration: 1 sec > Tests result: FAILURE > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mal%40egenix.com > -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Feb 17 2017) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From status at bugs.python.org Fri Feb 17 12:09:14 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 17 Feb 2017 18:09:14 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170217170914.DBD8A56669@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-02-10 - 2017-02-17) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5837 (+38) closed 35511 (+28) total 41348 (+66) Open issues with patches: 2488 Issues opened (51) ================== #29442: Use argparse and drop dirty optparse hacks in setup.py http://bugs.python.org/issue29442 reopened by Chi Hsuan Yen #29529: Backport Travis configuration http://bugs.python.org/issue29529 opened by brett.cannon #29530: Windows buildbots broken by the migration to GitHub (meta issu http://bugs.python.org/issue29530 opened by haypo #29532: functools.partial is not compatible between 2.7 and 3.5 http://bugs.python.org/issue29532 opened by naoyuki #29533: urllib2 works slowly with proxy on windows http://bugs.python.org/issue29533 opened by juliadolgova #29535: datetime hash is deterministic in some cases http://bugs.python.org/issue29535 opened by arigo #29537: Alternative fix for BUILD_MAP_UNPACK_WITH_CALL opcode in 3.5 http://bugs.python.org/issue29537 opened by serhiy.storchaka #29539: [smtplib] collect response data for all recipients http://bugs.python.org/issue29539 opened by David Ford (FirefighterBlu3) #29540: Add compact=True flag to json.dump/dumps http://bugs.python.org/issue29540 opened by Alex Gordon #29541: Python3 error while building on Alt-F http://bugs.python.org/issue29541 opened by Honza Sk??pala #29545: Python behavioral difference between Linux and AIX http://bugs.python.org/issue29545 opened by gireeshpunathil #29546: A more helpful ImportError message http://bugs.python.org/issue29546 opened by barry #29547: Add c code coverage to codecov http://bugs.python.org/issue29547 opened by belopolsky #29548: Recommend PyObject_Call* APIs over PyEval_Call*() APIs http://bugs.python.org/issue29548 opened by inada.naoki #29549: Improve docstring for str.index http://bugs.python.org/issue29549 opened by rhettinger #29550: Mac build-installer touch step fails after github conversion http://bugs.python.org/issue29550 opened by db3l #29551: unittest: TestSuite.debug() does not like subTest() http://bugs.python.org/issue29551 opened by wrohdewald #29553: Argparser does not display closing parentheses in nested mutex http://bugs.python.org/issue29553 opened by christofsteel #29554: profile/pstat doc clariification http://bugs.python.org/issue29554 opened by mbussonn #29555: Update Python Software Foundation Copyright Year http://bugs.python.org/issue29555 opened by orsenthil #29556: Remove unused #include http://bugs.python.org/issue29556 opened by Chi Hsuan Yen #29557: binhex documentation claims unknown bug http://bugs.python.org/issue29557 opened by David Wilemski #29558: Provide run_until_complete inside loop http://bugs.python.org/issue29558 opened by txomon #29559: Detect mouse over lines on canvas while mouse button is down http://bugs.python.org/issue29559 opened by Ted Shaneyfelt2 #29560: Turtle graphics fill behavior differs between versions http://bugs.python.org/issue29560 opened by John Simmons #29561: Interactive mode gives sys.ps2 not sys.ps1 after comment-only http://bugs.python.org/issue29561 opened by JDLH #29562: test_getgroups of test_posix fails (on OS X 10.10) http://bugs.python.org/issue29562 opened by JDLH #29564: ResourceWarning: suggest to enable tracemalloc in the message http://bugs.python.org/issue29564 opened by David Ford (FirefighterBlu3) #29565: Still broken ctypes calling convention on MSVC / 64-bit Window http://bugs.python.org/issue29565 opened by Igor Kudrin #29566: binhex() creates files with mixed line endings http://bugs.python.org/issue29566 opened by serhiy.storchaka #29568: undefined parsing behavior with the old style string formattin http://bugs.python.org/issue29568 opened by Jerry Dimitrov #29569: threading.Timer class: Continue periodical execution till acti http://bugs.python.org/issue29569 opened by slytomcat #29570: Windows Buildbot 2.7 is broken http://bugs.python.org/issue29570 opened by haypo #29571: test_re is failing when local is set for `en_IN` http://bugs.python.org/issue29571 opened by jaysinh.shukla #29572: Upgrade installers to OpenSSL 1.0.2k http://bugs.python.org/issue29572 opened by zach.ware #29573: NamedTemporaryFile with delete=True should not fail if file al http://bugs.python.org/issue29573 opened by richardxia #29574: python-3.6.0.tgz permissions borked http://bugs.python.org/issue29574 opened by Dave_Anderson #29575: doc 17.2.1: basic Pool example is too basic http://bugs.python.org/issue29575 opened by j5w6 #29577: Enum: mixin classes don't mix well with already mixed Enums http://bugs.python.org/issue29577 opened by ethan.furman #29579: Windows Python 3.7 installer broken by README.txt renamed to R http://bugs.python.org/issue29579 opened by haypo #29580: "Built-in Functions" not being functions http://bugs.python.org/issue29580 opened by Stefan Pochmann #29581: __init_subclass__ causes TypeError when used with standard lib http://bugs.python.org/issue29581 opened by Nate Soares #29582: Add Python coverage make target(s) http://bugs.python.org/issue29582 opened by zach.ware #29585: site.py imports relatively large `sysconfig` module. http://bugs.python.org/issue29585 opened by inada.naoki #29586: Cannot run pip in fresh install of py 3.5.3 http://bugs.python.org/issue29586 opened by Adrian Chan #29587: Generator/coroutine 'throw' discards exc_info state, which is http://bugs.python.org/issue29587 opened by njs #29588: importing ssl can fail with NameError: name 'PROTOCOL_TLS' is http://bugs.python.org/issue29588 opened by jdemeyer #29589: test_asyncio & test_multiprocessing_forkserver failed http://bugs.python.org/issue29589 opened by Dima Zhukov #29590: Incorrect stack traces when re-entering a generator/coroutine http://bugs.python.org/issue29590 opened by njs #29591: Various security vulnerabilities in bundled expat http://bugs.python.org/issue29591 opened by Natanael Copa #29592: abs_paths() in site.py is slow http://bugs.python.org/issue29592 opened by inada.naoki Most recent 15 issues with no replies (15) ========================================== #29592: abs_paths() in site.py is slow http://bugs.python.org/issue29592 #29590: Incorrect stack traces when re-entering a generator/coroutine http://bugs.python.org/issue29590 #29589: test_asyncio & test_multiprocessing_forkserver failed http://bugs.python.org/issue29589 #29581: __init_subclass__ causes TypeError when used with standard lib http://bugs.python.org/issue29581 #29575: doc 17.2.1: basic Pool example is too basic http://bugs.python.org/issue29575 #29572: Upgrade installers to OpenSSL 1.0.2k http://bugs.python.org/issue29572 #29570: Windows Buildbot 2.7 is broken http://bugs.python.org/issue29570 #29569: threading.Timer class: Continue periodical execution till acti http://bugs.python.org/issue29569 #29566: binhex() creates files with mixed line endings http://bugs.python.org/issue29566 #29560: Turtle graphics fill behavior differs between versions http://bugs.python.org/issue29560 #29559: Detect mouse over lines on canvas while mouse button is down http://bugs.python.org/issue29559 #29555: Update Python Software Foundation Copyright Year http://bugs.python.org/issue29555 #29554: profile/pstat doc clariification http://bugs.python.org/issue29554 #29551: unittest: TestSuite.debug() does not like subTest() http://bugs.python.org/issue29551 #29549: Improve docstring for str.index http://bugs.python.org/issue29549 Most recent 15 issues waiting for review (15) ============================================= #29571: test_re is failing when local is set for `en_IN` http://bugs.python.org/issue29571 #29568: undefined parsing behavior with the old style string formattin http://bugs.python.org/issue29568 #29557: binhex documentation claims unknown bug http://bugs.python.org/issue29557 #29555: Update Python Software Foundation Copyright Year http://bugs.python.org/issue29555 #29540: Add compact=True flag to json.dump/dumps http://bugs.python.org/issue29540 #29537: Alternative fix for BUILD_MAP_UNPACK_WITH_CALL opcode in 3.5 http://bugs.python.org/issue29537 #29533: urllib2 works slowly with proxy on windows http://bugs.python.org/issue29533 #29532: functools.partial is not compatible between 2.7 and 3.5 http://bugs.python.org/issue29532 #29523: latest setuptools breaks virtualenvs due to unbundling depende http://bugs.python.org/issue29523 #29509: redundant interning in PyObject_GetAttrString http://bugs.python.org/issue29509 #29507: Use FASTCALL in call_method() to avoid temporary tuple http://bugs.python.org/issue29507 #29476: Simplify set_add_entry() http://bugs.python.org/issue29476 #29475: option to not follow symlinks when globbing http://bugs.python.org/issue29475 #29469: AST-level Constant folding http://bugs.python.org/issue29469 #29465: Add _PyObject_FastCall() to reduce stack consumption http://bugs.python.org/issue29465 Top 10 most discussed issues (10) ================================= #29514: Add a test case that prevents magic number changes in minor re http://bugs.python.org/issue29514 30 msgs #29548: Recommend PyObject_Call* APIs over PyEval_Call*() APIs http://bugs.python.org/issue29548 23 msgs #29470: ssl: SNI callbacks should not modify context objects http://bugs.python.org/issue29470 14 msgs #29585: site.py imports relatively large `sysconfig` module. http://bugs.python.org/issue29585 11 msgs #29540: Add compact=True flag to json.dump/dumps http://bugs.python.org/issue29540 10 msgs #29564: ResourceWarning: suggest to enable tracemalloc in the message http://bugs.python.org/issue29564 10 msgs #29521: Minor warning messages when compiling documentation http://bugs.python.org/issue29521 9 msgs #29442: Use argparse and drop dirty optparse hacks in setup.py http://bugs.python.org/issue29442 8 msgs #29586: Cannot run pip in fresh install of py 3.5.3 http://bugs.python.org/issue29586 8 msgs #19675: Pool dies with excessive workers, but does not cleanup http://bugs.python.org/issue19675 7 msgs Issues closed (28) ================== #5667: Interpreter fails to initialize on build dir when IO encoding http://bugs.python.org/issue5667 closed by haypo #18779: Misleading documentations and comments in regular expression H http://bugs.python.org/issue18779 closed by akuchling #23109: French quotes in the documentation are often ungrammatical http://bugs.python.org/issue23109 closed by akuchling #27536: Convert readme to reStructuredText http://bugs.python.org/issue27536 closed by zach.ware #28929: Provide a link from documentation back to its source file http://bugs.python.org/issue28929 closed by Mariatta #28941: Update the link to Source Code in Python Docs from hg to githu http://bugs.python.org/issue28941 closed by Mariatta #29438: use after free in key sharing dict http://bugs.python.org/issue29438 closed by inada.naoki #29474: Grammatical errors in weakref.WeakValueDictionary docs http://bugs.python.org/issue29474 closed by Mariatta #29481: 3.6.0 doc describes 3.6.1 feature - typing.Deque http://bugs.python.org/issue29481 closed by Mariatta #29511: Add 'find' as build-in method for lists http://bugs.python.org/issue29511 closed by terry.reedy #29518: 'for' loop not automatically breaking (index error on line of http://bugs.python.org/issue29518 closed by brett.cannon #29524: Move functions to call objects into a new Objects/call.c file http://bugs.python.org/issue29524 closed by haypo #29527: Travis: doc job is broken http://bugs.python.org/issue29527 closed by haypo #29528: Encrypt IRC channel details for Travis http://bugs.python.org/issue29528 closed by zach.ware #29531: Update Doc/README.txt to README.rst http://bugs.python.org/issue29531 closed by Roger Sachan #29534: _decimal difference with _pydecimal http://bugs.python.org/issue29534 closed by mark.dickinson #29536: test_hashlib failure on Ubuntu 16.04 http://bugs.python.org/issue29536 closed by christian.heimes #29538: bugs.python.org does not redirect to HTTPS http://bugs.python.org/issue29538 closed by berker.peksag #29542: python -m spacy.en.download undefined symbol: _ZSt24__throw_ou http://bugs.python.org/issue29542 closed by berker.peksag #29543: Allow login to bugs.python.org with email address http://bugs.python.org/issue29543 closed by berker.peksag #29544: Allow login to bugs.python.org with Google account http://bugs.python.org/issue29544 closed by berker.peksag #29552: Issue in Dictionary fromkeys http://bugs.python.org/issue29552 closed by prudvibt81 #29563: Update Devguide about building documentation. http://bugs.python.org/issue29563 closed by brett.cannon #29567: Allow argparse to be used in setup.py http://bugs.python.org/issue29567 closed by Chi Hsuan Yen #29576: Improve some deprecations in the importlib http://bugs.python.org/issue29576 closed by brett.cannon #29578: "python.exe t2.py" doesn't work the same on Python-3.6 as Pyth http://bugs.python.org/issue29578 closed by paul.moore #29583: Python 3.6 won't install on Windows 2012 R2 http://bugs.python.org/issue29583 closed by eryksun #29584: 2D list with Bugs when defined by using Multiplication http://bugs.python.org/issue29584 closed by ezio.melotti From armin.rigo at gmail.com Sun Feb 19 03:30:47 2017 From: armin.rigo at gmail.com (Armin Rigo) Date: Sun, 19 Feb 2017 09:30:47 +0100 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: Hi Freddy, On 16 February 2017 at 18:03, Freddy Rietdijk wrote: > As I mentioned, it seems only sets cause unreproducible > bytecode. Sets have no order. But when generating the bytecode, I would > expect there would still be an order since the code isn't actually executed, > right? No, the sets are built as real sets and then marshalled to .pyc files in a separate step. So on CPython an essentially random order will end up in the .pyc file. Even CPython 3.6 gives a deterministic order to dictionaries but not sets. You could ensure sets are marshalled in a known order by changing the marshalling code, e.g. to emit them in sorted order (on Python 2.x; on 3.x it is more messy because different types are more often non-comparable). A bient?t, Armin. From freddyrietdijk at fridh.nl Sun Feb 19 06:35:13 2017 From: freddyrietdijk at fridh.nl (Freddy Rietdijk) Date: Sun, 19 Feb 2017 12:35:13 +0100 Subject: [Python-Dev] Deterministic builds of the interpreter In-Reply-To: References: Message-ID: Hi Armin, Thank you for your explanation. I've now managed to build 2.7 and 3.5 deterministic by recompiling the bytecode at the end of the build (and excluding 2to3). Freddy On Sun, Feb 19, 2017 at 9:30 AM, Armin Rigo wrote: > Hi Freddy, > > On 16 February 2017 at 18:03, Freddy Rietdijk > wrote: > > As I mentioned, it seems only sets cause unreproducible > > bytecode. Sets have no order. But when generating the bytecode, I would > > expect there would still be an order since the code isn't actually > executed, > > right? > > No, the sets are built as real sets and then marshalled to .pyc files > in a separate step. So on CPython an essentially random order will > end up in the .pyc file. Even CPython 3.6 gives a deterministic order > to dictionaries but not sets. You could ensure sets are marshalled in > a known order by changing the marshalling code, e.g. to emit them in > sorted order (on Python 2.x; on 3.x it is more messy because different > types are more often non-comparable). > > > A bient?t, > > Armin. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.dartiailh at gmail.com Mon Feb 20 13:44:36 2017 From: m.dartiailh at gmail.com (Matthieu Dartiailh) Date: Mon, 20 Feb 2017 19:44:36 +0100 Subject: [Python-Dev] CALL_FUNCTION_EX arg and stack_effect Message-ID: Hi, I have a question about the use of CALL_FUNCTION_EX in https://github.com/python/cpython/blob/master/Python/compile.c#L3624. Looking at the code it appears that the argument will be either 1 or 0 depending on whether or not the function is taking keywords arguments (which means that CALL_FUNCTION_EX cannot be used on function taking no argument). Executing that opcode will remove from the stack the function code, the positional arguments (packed in a tuple) and the potential keyword arguments packed in a dict and push the return value. So the stack effect will be either -1 or -2 (could be 0 if the possibility to pass 0 arguments existed). Looking at the stack effect computation (https://github.com/python/cpython/blob/master/Python/compile.c#L1047), it appears that the stack effect will be 0 if the argument is 0, -1 for either 1 or 2, and -2 for 3. Which means that the code generated at https://github.com/python/cpython/blob/master/Python/compile.c#L3624 can never allow to compute the right stack effect using the stack_effect function (as it will return either 0 or -1 instead of -1 and -2) I would say that this is a bug and that the oparg should be 1 + 2 if keywords arguments are present at line 3624. I am not sure what consequence this can have on CPython but it means the bytecode becomes weird as a the stack looks like it can grow during a list comprehension (calling a function f using * syntax) : BUILD_LIST 0 1 LOAD_FAST .0 1 FOR_ITER 22 1 ---> after each jump it looks like the stack is higher by one STORE_FAST i -1 LOAD_GLOBAL f 1 LOAD_DEREF a 1 LOAD_FAST i 1 BINARY_SUBSCR None -1 CALL_FUNCTION_EX 0 0 LIST_APPEND 2 -1 JUMP_ABSOLUTE 4 0 RETURN_VALUE None -1 What do you think ? Should I open an issue on https://bugs.python.org/ ? Best regards Matthieu From nad at python.org Mon Feb 20 15:12:47 2017 From: nad at python.org (Ned Deily) Date: Mon, 20 Feb 2017 15:12:47 -0500 Subject: [Python-Dev] IMPORTANT: Python 3.6.1 Maintenance Release Release Candidate in 7 days (2017-02-27) Message-ID: <9DEC04ED-5B6E-45CE-AC83-4A9CD8EBF829@python.org> It seems like last year already since the release of 3.6.0. I guess that's because it was last year, 2016-12-22 to be exact! Now we're approaching the end of the first quarter and, according to PEP 494, it's time to start producing the first maintenance release for the 3.6 series. The schedule calls for the release candidate to be produced on Monday 2017-02-27 UTC. As was the case with the 3.6.0 release cycle, the plan is for the release candidate to be the same as the final release, that is, no additional changes go in after the release candidate except for any showstopper critical problems that might be discovered with rc1. So please plan to get any security fixes, bug fixes, and documentation changes you think should be in 3.6.1 merged in before 2017-02-27. I will send out another reminder a couple of days before. The 3.6.1 final is planned for two weeks following rc1, that is, on 2017-03-13. I expect the next 3.6 maintenance release (3.6.2) will follow about 3 months later, so most likely in 2017-06 after PyCon US. 3.6.1 will be the first release using our new GitHub-based development process (thanks, Brett and team!). If you are planning to push something for 3.6.1 and haven't yet tried out the new workflow or are not yet familiar with GitHub pull requests, you should probably give yourself some extra time. As always, the Developer's Guide is the primary reference for the development workflow; not surprisingly, with such a major change, there are likely still some parts of the guide that could use further changes and clarifications. You can help by reviewing the devguide's open issues and pull requests in its repository and adding to them as you work through issues. If you have comments on or improvement suggestions for the new workflow, the place to discuss them is on the core-workflow mailing list. Thanks again for all of your efforts in bringing 3.6.0 into the world and for helping now to make it even better! https://www.python.org/dev/peps/pep-0494/ http://cpython-devguide.readthedocs.io https://mail.python.org/mailman/listinfo/core-workflow -- Ned Deily nad at python.org -- [] From nospam at curso.re Mon Feb 20 16:01:21 2017 From: nospam at curso.re (nospam at curso.re) Date: Mon, 20 Feb 2017 21:01:21 +0000 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) Message-ID: <87wpcktw4e.fsf@example.com> Hello, I have just noticed that an FTP injection advisory has been made public on the oss-security list. The author says that he an exploit exists but it won't be published until the code is patched You may be already aware, but it would be good to understand what is the position of the core developers about this. The advisory is linked below (with some excerpts in this message): http://blog.blindspotsecurity.com/2017/02/advisory-javapython-ftp-injections.html Protocol injection flaws like this have been an area of research of mine for the past few couple of years and as it turns out, this FTP protocol injection allows one to fool a victim's firewall into allowing TCP connections from the Internet to the vulnerable host's system on any "high" port (1024-65535). A nearly identical vulnerability exists in Python's urllib2 and urllib libraries. In the case of Java, this attack can be carried out against desktop users even if those desktop users do not have the Java browser plugin enabled. As of 2017-02-20, the vulnerabilities discussed here have not been patched by the associated vendors, despite advance warning and ample time to do so. [...] Python's built-in URL fetching library (urllib2 in Python 2 and urllib in Python 3) is vulnerable to a nearly identical protocol stream injection, but this injection appears to be limited to attacks via directory names specified in the URL. [...] The Python security team was notified in January 2016. Information provided included an outline of the possibility of FTP/firewall attacks. Despite repeated follow-ups, there has been no apparent action on their part. Best regards, -- Stefano P.S. I am posting from gmane, I hope that this is OK. From armin.rigo at gmail.com Tue Feb 21 02:23:54 2017 From: armin.rigo at gmail.com (Armin Rigo) Date: Tue, 21 Feb 2017 08:23:54 +0100 Subject: [Python-Dev] CALL_FUNCTION_EX arg and stack_effect In-Reply-To: References: Message-ID: Hi Matthieu, On 20 February 2017 at 19:44, Matthieu Dartiailh wrote: > What do you think ? Should I open an issue on https://bugs.python.org/ ? Possibly related: http://bugs.python.org/issue24340 A bient?t, Armin. From victor.stinner at gmail.com Wed Feb 22 07:10:17 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 22 Feb 2017 13:10:17 +0100 Subject: [Python-Dev] Translated Python documentation Message-ID: Hi, I'm a member of the french #python-fr IRC channel on Freenode: it's common to meet people who don't speak english and so are unable to read the Python official documentation. Python wants to be widely available, for all users, in any language: that's also why Python 3 now allows any non-ASCII identifiers: https://www.python.org/dev/peps/pep-3131/#rationale There are a least 3 groups of people who are translating the Python documentation in their mother language (french, japanese, spanish). They tried to make it more official, but their attempt didn't go far yet. I'm writing this email to propose to officially support translated versions of the documentation. For me, the most impotant point would be to give access to the translated documentation from docs.python.org. For example, have a dropdown list with available languages. IMHO a reference in this domain is PHP: PHP documentation is translated to at least 10 languages. See for example the "Change language: [...]" list at: http://php.net/echo I'm not asking you to take any technical decision here, I'm just asking for an official general "support" of translated documentation. References to translated documentations: Transiflex project: https://www.transifex.com/python-doc/ Fran?ais (French, FR): doc: https://www.afpy.org/doc/python/ source: https://github.com/AFPy/python_doc_fr mailing list: http://lists.afpy.org/mailman/listinfo/traductions Japanese (JP): doc: http://docs.python.jp/3/ source: https://github.com/python-doc-ja/python-doc-ja Spanish: doc: http://docs.python.org.ar/tutorial/3/index.html Previous discussions: [Python-ideas] Cross link documentation translations (January, 2016): https://mail.python.org/pipermail/python-ideas/2016-January/038010.html [Python-ideas] https://docs.python.org/fr/ ? (March 2016) https://mail.python.org/pipermail/python-ideas/2016-March/038879.html Victor From victor.stinner at gmail.com Wed Feb 22 07:27:57 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 22 Feb 2017 13:27:57 +0100 Subject: [Python-Dev] Split Misc/NEWS into individual files Message-ID: Hi, As an user, I like Misc/NEWS changelog. I like reading https://docs.python.org/dev/whatsnew/changelog.html#changelog after a minor Python release to see all bugfixes. As a developer, I deeply hate this file :-) It's very painful to: * find the latest Python version in this giant file * find the right section * ... handle merge conflicts!!! * understand how this thing works on multiple Python versions The last point was always a mystery for me. If we make a change in Python 3.5.3, merge it into the 3.6 branch: should we document the change in 3.5.3 and 3.6.0 changelog? What if 3.6.0 is released *before* 3.5.3? What if 3.6.0 is released *after* 3.5.3? IMHO developers should not have to think about the release date when writing a changelog, we need a tool doing that for us. OpenStack happily fixed this issue one or two years ago with "reno": https://pypi.python.org/pypi/reno It's a simple tool building a changelog from a list of files: each file describe a change of a commit. The filename is the commit identifier. Simple, isn't it? OpenStack has something like 100x more commits than CPython (the overall OpenStack project). It's just a giant project and so many people and pull requests which can remain open for months. Very obvious win of individual files: no more merge conflict in two concurrent changes, since each commit adds its own file. My idea is not new, it's already discussed in the Python core-workflow: https://github.com/python/core-workflow/issues/6 I'm opening a thread on python-dev because Misc/NEWS quickly became a blocker issue with the Python new workflow. What do you think of the overall idea? Victor From tseaver at palladion.com Wed Feb 22 10:19:37 2017 From: tseaver at palladion.com (Tres Seaver) Date: Wed, 22 Feb 2017 10:19:37 -0500 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: Message-ID: -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 On 02/22/2017 07:10 AM, Victor Stinner wrote: > Hi, > > I'm a member of the french #python-fr IRC channel on Freenode: it's > common to meet people who don't speak english and so are unable to > read the Python official documentation. Python wants to be widely > available, for all users, in any language: that's also why Python 3 > now allows any non-ASCII identifiers: > https://www.python.org/dev/peps/pep-3131/#rationale > > There are a least 3 groups of people who are translating the Python > documentation in their mother language (french, japanese, spanish). > They tried to make it more official, but their attempt didn't go far > yet. I'm writing this email to propose to officially support > translated versions of the documentation. > > For me, the most impotant point would be to give access to the > translated documentation from docs.python.org. For example, have a > dropdown list with available languages. > > IMHO a reference in this domain is PHP: PHP documentation is > translated to at least 10 languages. See for example the "Change > language: [...]" list at: > > http://php.net/echo > > I'm not asking you to take any technical decision here, I'm just > asking for an official general "support" of translated documentation. +1 for the idea. Tres. - -- =================================================================== Tres Seaver +1 540-429-0999 tseaver at palladion.com Palladion Software "Excellence by Design" http://palladion.com -----BEGIN PGP SIGNATURE----- Version: GnuPG v1 iQIcBAEBAgAGBQJYrawEAAoJEPKpaDSJE9HYevwP/0KsRu1KNkBGFVffA8WNBDqT khZta8AogUnvIkubNxDWvtMgo+UDgXdpW0D0/uCI5ON8O6TVeAQp8JB5giZkKt6e VSoXjPaD4sJJRjWCFj+z33DCu/HlVE5+qt60cRQ6kwWWKWm6Aoh+yiu1g0CDQIN0 miOmc+6rZWTptYUDYX3j/gq1W/XxiaEDIJTf9D8GUQ7YZA6AyTkzUm1o9OY8PJdg lSsTxkjuriQPLJq6IVXMZxOUPKgU/O9HLL9r3XfRqA/sIle6pz6z6qKIQOHj246E lKiSXwm2Gik+XrIk0YfIG+ZezlWJuRyyA+7wksxQmX/Azl85iQ/Nb5/zwVMTfpF1 NIT0++bpefM5X/Y7CLusupNIQ90k5dSMjX9uPOBoK/ZIJ7I6QglBVxr97CHx1FCn lJ4brOkBnGCtz9mJ50FgM4YqYuUUWjw8mZxpjv5URl8ouPdZnhmuWmiRJLgezmbV n0uqjr9XvEdXEYidRzL39LAWcERJIj1CfyyCgDISYw8eaZsrn29lrk0F8x17XAIP Z0iIF4cYg0Z1oXqmWelzljDyo66GIMybof3R2pHKCiPlfZeE3b+AwgU2eyI+6lnj iVfQi5ok4XuYMydmJhjfWbmz8xb0jkqQ7un3TRb6w8MFOS1WxxYAsVJnesA6Jk3c g4leycwmSg9dTQ/TSRZj =6/rU -----END PGP SIGNATURE----- From solipsis at pitrou.net Wed Feb 22 10:40:58 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 22 Feb 2017 16:40:58 +0100 Subject: [Python-Dev] Translated Python documentation References: Message-ID: <20170222164058.7b0333ef@fsol> Hi Victor, On Wed, 22 Feb 2017 13:10:17 +0100 Victor Stinner wrote: > > IMHO a reference in this domain is PHP: PHP documentation is > translated to at least 10 languages. See for example the "Change > language: [...]" list at: > > http://php.net/echo > > I'm not asking you to take any technical decision here, I'm just > asking for an official general "support" of translated documentation. As long as you are asking for "moral" support and not actually vouching for the accuracy of third-party translations, then +1 from me. ? bient?t Antoine. From victor.stinner at gmail.com Wed Feb 22 11:15:53 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 22 Feb 2017 17:15:53 +0100 Subject: [Python-Dev] Translated Python documentation In-Reply-To: <20170222164058.7b0333ef@fsol> References: <20170222164058.7b0333ef@fsol> Message-ID: 2017-02-22 16:40 GMT+01:00 Antoine Pitrou : > As long as you are asking for "moral" support and not actually > vouching for the accuracy of third-party translations, then +1 from me. The main complain about these translations is the accuracy. My bet is that making these translations "official" and more visible (at docs.python.org) would make them more popular, and so indirectly help to recruit new contributors. Slowly, the number of translated pages should increase, but the overall translation quality should also increase. That's how free software are developed, no? :-) Victor From eric at trueblade.com Wed Feb 22 11:26:26 2017 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 22 Feb 2017 11:26:26 -0500 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: References: Message-ID: <09c0a605-41a9-b045-82e6-0b2be170f928@trueblade.com> On 2/22/17 7:27 AM, Victor Stinner wrote: > OpenStack happily fixed this issue one or two years ago with "reno": > https://pypi.python.org/pypi/reno > > It's a simple tool building a changelog from a list of files: each > file describe a change of a commit. The filename is the commit > identifier. Simple, isn't it? I've read through the reno design document, and I think it would work for Python: https://docs.openstack.org/developer/reno/design.html > What do you think of the overall idea? I'm for it. I'd be interested in hearing what current and former release managers say. Eric. From barry at python.org Wed Feb 22 11:42:57 2017 From: barry at python.org (Barry Warsaw) Date: Wed, 22 Feb 2017 11:42:57 -0500 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: References: Message-ID: <20170222114257.62c4c88a@subdivisions.wooz.org> On Feb 22, 2017, at 01:27 PM, Victor Stinner wrote: >OpenStack happily fixed this issue one or two years ago with "reno": >https://pypi.python.org/pypi/reno reno looks interesting but there is at least one thing I think would need changing for us. Aside: I'm not super psyched about its requiring YAML files for its input. Their structure can be fairly arcane for drive-by contributors, but I suppose with good documentation and a CI gate, we can live with that. I'm also not thrilled with having to learn yet another command line tool to submit changes to Python. The main dissonance for us is I think the sections that reno supports https://docs.openstack.org/developer/reno/usage.html Our release notes have different sections: * Core and Builtins * Extension Modules * Library * Windows * C API * Documentation * Build * Tools/Demos * Tests (others?) I don't know if those are configurable in reno or it would require a fork, but I'd like to preserve that organizational structure. If reno can also help wrap long lines, enforce/encourage bpo-* mentions, and clean up whitespace, then I'm for trying it out. -Barry From facundobatista at gmail.com Wed Feb 22 12:58:03 2017 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 22 Feb 2017 14:58:03 -0300 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: Message-ID: On Wed, Feb 22, 2017 at 9:10 AM, Victor Stinner wrote: > There are a least 3 groups of people who are translating the Python > documentation in their mother language (french, japanese, spanish). We translated (and even printed/published) the Tutorial (we're currently maintaining the Py2 and Py3 branches). But only the tutorial, that is a smaller and steadier doc. > For me, the most impotant point would be to give access to the > translated documentation from docs.python.org. For example, have a > dropdown list with available languages. To clarify: you're proposing to *serve* the translated files from d.p.o? Or just a link to where those are served now? Thanks! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From storchaka at gmail.com Wed Feb 22 13:04:19 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 22 Feb 2017 20:04:19 +0200 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On 22.02.17 18:15, Victor Stinner wrote: > 2017-02-22 16:40 GMT+01:00 Antoine Pitrou : >> As long as you are asking for "moral" support and not actually >> vouching for the accuracy of third-party translations, then +1 from me. > > The main complain about these translations is the accuracy. > > My bet is that making these translations "official" and more visible > (at docs.python.org) would make them more popular, and so indirectly > help to recruit new contributors. Slowly, the number of translated > pages should increase, but the overall translation quality should also > increase. That's how free software are developed, no? :-) What percent of lines is changed between bugfix releases? Feature releases? My largest apprehension is that the documentation can be outdated and quickly become degraded if main contributors left it. From doug at doughellmann.com Wed Feb 22 13:45:02 2017 From: doug at doughellmann.com (Doug Hellmann) Date: Wed, 22 Feb 2017 13:45:02 -0500 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: <20170222114257.62c4c88a@subdivisions.wooz.org> References: <20170222114257.62c4c88a@subdivisions.wooz.org> Message-ID: <1487788395-sup-6534@lrrr.local> Excerpts from Barry Warsaw's message of 2017-02-22 11:42:57 -0500: > On Feb 22, 2017, at 01:27 PM, Victor Stinner wrote: > > >OpenStack happily fixed this issue one or two years ago with "reno": > >https://pypi.python.org/pypi/reno > > reno looks interesting but there is at least one thing I think would need > changing for us. Aside: I'm not super psyched about its requiring YAML files > for its input. Their structure can be fairly arcane for drive-by > contributors, but I suppose with good documentation and a CI gate, we can live > with that. > > I'm also not thrilled with having to learn yet another command line tool to > submit changes to Python. > > The main dissonance for us is I think the sections that reno supports > > https://docs.openstack.org/developer/reno/usage.html > > Our release notes have different sections: > > * Core and Builtins > * Extension Modules > * Library > * Windows > * C API > * Documentation > * Build > * Tools/Demos > * Tests > (others?) > > I don't know if those are configurable in reno or it would require a fork, but > I'd like to preserve that organizational structure. If reno can also help > wrap long lines, enforce/encourage bpo-* mentions, and clean up whitespace, > then I'm for trying it out. > > -Barry FWIW, I've been planning to work on a demonstration patch using reno for cpython so we can identify feature gaps like this. I have permission to work on this as part of my day job, to make reno suitable for both communities. I'm at a conference this week, but will try to put something together after I get back to work, and definitely before the language summit at PyCon. We have a patch up for review to support configuring the sections, including the keys within the input file and the titles in the output, so that should take care of that gap. The use of lists within each section is also something I think we could either make smart or explicitly configurable, and that would simplify the file format to just section keys and an RST blob. The "report" command assembles the RST embedded in the YAML and then passes it through docutils to produce text output. In the process, it does wrap lines, standardize bullets for lists, etc. For an example of the output, you can look at the release announcement for nova from earlier today [1]. The content of that email up to the git log output at the end was produced by reno based on the input files in [2]. Doug [1] http://lists.openstack.org/pipermail/release-announce/2017-February/000784.html [2] http://git.openstack.org/cgit/openstack/nova/tree/releasenotes/notes From facundobatista at gmail.com Wed Feb 22 14:20:44 2017 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 22 Feb 2017 16:20:44 -0300 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On Wed, Feb 22, 2017 at 3:04 PM, Serhiy Storchaka wrote: > My largest apprehension is that the documentation can be outdated and > quickly become degraded if main contributors left it. This is why we focus in the Tutorial only, but also: a) The tutorial is mostly the "entry point" to Python's doc, so being it in Spanish for Spanish speakers lowers the barrier a lot b) At some point you need to understand English docs, so translating the library reference should not be that important Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From victor.stinner at gmail.com Wed Feb 22 15:59:17 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 22 Feb 2017 21:59:17 +0100 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: <20170222114257.62c4c88a@subdivisions.wooz.org> References: <20170222114257.62c4c88a@subdivisions.wooz.org> Message-ID: 2017-02-22 17:42 GMT+01:00 Barry Warsaw : > Our release notes have different sections: > > * Core and Builtins > * Extension Modules > * Library > * Windows > * C API > * Documentation > * Build > * Tools/Demos > * Tests > (others?) While we are talking about changing things... I would propose to require to specify a module name if a change is in the Library section, and then maybe even group changes per module. Victor From victor.stinner at gmail.com Wed Feb 22 16:01:01 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 22 Feb 2017 22:01:01 +0100 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: 2017-02-22 19:04 GMT+01:00 Serhiy Storchaka : > What percent of lines is changed between bugfix releases? Feature releases? > > My largest apprehension is that the documentation can be outdated and > quickly become degraded if main contributors left it. If the original text (english) changes, untranslated text is displayed, not outdated text. The french translation uses gettext. I don't know how other translations are implemented. Victor From nad at python.org Wed Feb 22 16:29:57 2017 From: nad at python.org (Ned Deily) Date: Wed, 22 Feb 2017 16:29:57 -0500 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: References: Message-ID: <9868CB93-F573-4CA9-9C17-7CA1883891CF@python.org> On Feb 22, 2017, at 07:27, Victor Stinner wrote: > [...]My idea is not new, it's already discussed in the Python core-workflow: > https://github.com/python/core-workflow/issues/6 > > I'm opening a thread on python-dev because Misc/NEWS quickly became a > blocker issue with the Python new workflow. > > What do you think of the overall idea? I think we clearly need to do something about Misc/NEWS but this issue is already under discussion in the core-workflow working group and I think we shouldn't be moving it to python-dev until there is a final concrete proposal. I know it is high on Brett's list but I believe he's taking a few days of holiday right so he may not be able to respond immediately. Let's move this discussion over to the core-workflow mailing list. It can wait a few days. https://mail.python.org/mailman/listinfo/core-workflow -- Ned Deily nad at python.org -- [] From storchaka at gmail.com Wed Feb 22 17:53:07 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 23 Feb 2017 00:53:07 +0200 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: References: <20170222114257.62c4c88a@subdivisions.wooz.org> Message-ID: On 22.02.17 22:59, Victor Stinner wrote: > While we are talking about changing things... I would propose to > require to specify a module name if a change is in the Library > section, and then maybe even group changes per module. Some changes can be related to a couple of modules. From tjreedy at udel.edu Wed Feb 22 18:17:46 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 22 Feb 2017 18:17:46 -0500 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: References: <20170222114257.62c4c88a@subdivisions.wooz.org> Message-ID: On 2/22/2017 3:59 PM, Victor Stinner wrote: > 2017-02-22 17:42 GMT+01:00 Barry Warsaw : >> Our release notes have different sections: >> >> * Core and Builtins >> * Extension Modules >> * Library * IDLE >> * Windows >> * C API >> * Documentation >> * Build >> * Tools/Demos >> * Tests >> (others?) > > While we are talking about changing things... I would propose to > require to specify a module name if a change is in the Library > section, and then maybe even group changes per module. +1 I would like tkinter changes grouped together. There have been similar requests to more finely divide tracker issues. Since there are about 400 modules, an alternative would be to group by package or doc chapter. The 30 chapters after the 5 built-in chapters are a much more manageable number for both the tracker and NEWS. Having IDLE issues (which groups 60 related modules) split out tremendously helps IDLE users. But it need not be the only such grouping. -- Terry Jan Reedy From tjreedy at udel.edu Wed Feb 22 18:20:49 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 22 Feb 2017 18:20:49 -0500 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: References: <20170222114257.62c4c88a@subdivisions.wooz.org> Message-ID: On 2/22/2017 5:53 PM, Serhiy Storchaka wrote: > On 22.02.17 22:59, Victor Stinner wrote: >> While we are talking about changing things... I would propose to >> require to specify a module name if a change is in the Library >> section, and then maybe even group changes per module. > > Some changes can be related to a couple of modules. There should be a 'General' categories for changes not limited to modules within a grouping (see previous message). Many of your multi-module patches would fit in 'General'. -- Terry Jan Reedy From victor.stinner at gmail.com Wed Feb 22 20:15:20 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 23 Feb 2017 02:15:20 +0100 Subject: [Python-Dev] Report on Python vulnerabilities Message-ID: Hi, I created a tool to generate a report on Python vulnerabilities: http://python-security.readthedocs.io/vulnerabilities.html I collected data of 41 vulnerabilities since 2007 (first Python CVE: CVE-2007-4965). If you would like to add data of a new vulnerability, enhance the report, ... : see the GitHub project. https://github.com/haypo/python-security The main data file is vulnerabilities.yml (YAML). I also filled manually the python_releases.txt: file: list of all release dates from Python 2.5.0 to Python 3.6.0. The tool compute the first Python release in each branch which includes the fix from a list of commits. The tool should help to track if vulnerabilities are fixed in all supported Python versions (branches accepting security fixes). I also started to collect some notes about Python security in general, evolution of th ssl module, etc. in the same documentation. Victor From victor.stinner at gmail.com Wed Feb 22 20:17:03 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 23 Feb 2017 02:17:03 +0100 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: References: <20170222114257.62c4c88a@subdivisions.wooz.org> Message-ID: 2017-02-22 23:53 GMT+01:00 Serhiy Storchaka : > Some changes can be related to a couple of modules. I suggest to repeat either repeat the change in each module section, or put the change in a general section. The tool should support a list of modules. Victor From steve at pearwood.info Thu Feb 23 23:36:57 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 24 Feb 2017 15:36:57 +1100 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) In-Reply-To: <87wpcktw4e.fsf@example.com> References: <87wpcktw4e.fsf@example.com> Message-ID: <20170224043657.GE5689@ando.pearwood.info> I haven't seen any response to the following alleged security vulnerability. I am not qualified to judge the merits of this, but it does seem worrying that (alledgedly) the Python security team hasn't responded for over 12 months. Is anyone able to comment? Thanks, Steve On Mon, Feb 20, 2017 at 09:01:21PM +0000, nospam at curso.re wrote: > Hello, > > I have just noticed that an FTP injection advisory has been made public > on the oss-security list. > > The author says that he an exploit exists but it won't be published > until the code is patched > > You may be already aware, but it would be good to understand what is the > position of the core developers about this. > > The advisory is linked below (with some excerpts in this message): > > http://blog.blindspotsecurity.com/2017/02/advisory-javapython-ftp-injections.html > > Protocol injection flaws like this have been an area of research of mine > for the past few couple of years and as it turns out, this FTP protocol > injection allows one to fool a victim's firewall into allowing TCP > connections from the Internet to the vulnerable host's system on any > "high" port (1024-65535). A nearly identical vulnerability exists in > Python's urllib2 and urllib libraries. In the case of Java, this attack > can be carried out against desktop users even if those desktop users do > not have the Java browser plugin enabled. > As of 2017-02-20, the vulnerabilities discussed here have not been patched > by the associated vendors, despite advance warning and ample time to do > so. > [...] > Python's built-in URL fetching library (urllib2 in Python 2 and urllib in > Python 3) is vulnerable to a nearly identical protocol stream injection, > but this injection appears to be limited to attacks via directory names > specified in the URL. > [...] > The Python security team was notified in January 2016. Information > provided included an outline of the possibility of FTP/firewall attacks. > Despite repeated follow-ups, there has been no apparent action on their > part. > > Best regards, > > -- Stefano > > P.S. > I am posting from gmane, I hope that this is OK. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve%40pearwood.info > From benjamin at python.org Fri Feb 24 02:51:45 2017 From: benjamin at python.org (Benjamin Peterson) Date: Thu, 23 Feb 2017 23:51:45 -0800 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) In-Reply-To: <20170224043657.GE5689@ando.pearwood.info> References: <87wpcktw4e.fsf@example.com> <20170224043657.GE5689@ando.pearwood.info> Message-ID: <1487922705.3562826.891373264.5AA7922D@webmail.messagingengine.com> On Thu, Feb 23, 2017, at 20:36, Steven D'Aprano wrote: > I haven't seen any response to the following alleged security > vulnerability. > > I am not qualified to judge the merits of this, but it does seem > worrying that (alledgedly) the Python security team hasn't responded for > over 12 months. Like all CPython developers, the Python security team are all volunteers. That combined with the fact that dealing with security issues is one of the least fun programming tasks means issues are sometimes dropped. Perhaps some organization with a stake Python security would like to financially support Python security team members. As for this, particular issue, we should determine if there's a tracker issue yet and continue discussion there. From vadmium+py at gmail.com Fri Feb 24 03:09:22 2017 From: vadmium+py at gmail.com (Martin Panter) Date: Fri, 24 Feb 2017 08:09:22 +0000 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) In-Reply-To: <1487922705.3562826.891373264.5AA7922D@webmail.messagingengine.com> References: <87wpcktw4e.fsf@example.com> <20170224043657.GE5689@ando.pearwood.info> <1487922705.3562826.891373264.5AA7922D@webmail.messagingengine.com> Message-ID: On 24 February 2017 at 07:51, Benjamin Peterson wrote: > As for this, particular issue, we should determine if there's a tracker > issue yet and continue discussion there. That would be . From ncoghlan at gmail.com Fri Feb 24 04:55:20 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Feb 2017 19:55:20 +1000 Subject: [Python-Dev] PEP 543: A Unified TLS API for Python In-Reply-To: <0AC4A5C1-B6D3-41FF-80E0-2B26266A5956@lukasa.co.uk> References: <0AC4A5C1-B6D3-41FF-80E0-2B26266A5956@lukasa.co.uk> Message-ID: On 14 February 2017 at 00:26, Cory Benfield wrote: > All, > > Python has never exposed the ability to use TLS backends other than > OpenSSL in the standard library. As Christian and I discussed back at the > Developer Summit at PyCon US 2016, the ssl module would more properly be > called the openssl module, due to exposing many OpenSSL-specific concepts > and behaviours in its API. This has meant that the Python ecosystem is > overwhelmingly also an OpenSSL ecosystem, which is problematic on Windows > and macOS (and Unices that for whatever reason aren?t interested in > shipping an OpenSSL), as it has meant that Python needs to bring its own > OpenSSL, and that it is troublesome to interact with the system trust > database. > > The first step to resolving this would be to provide a new module that > exposes TLS concepts in a much more generic manner. There are two possible > approaches to this. The first is to have this module be a generic concrete > implementation that can be compiled against multiple TLS backends (like > curl). This would require that all the relevant bindings be built into the > standard library from the get-go, which provides a substantial maintenance > burden on a team of people that are already understaffed maintaining the > ssl module alone. The second approach is to define a generic high-level TLS > interface that provides a minimal usable interface that can be implemented > by both first- and third-party modules. This would allow the standard > library to continue to ship with only exactly what it needs (for example, > OpenSSL, SecureTransport and SChannel), but would give those who want to > use more esoteric TLS choices (NSS, GnuTLS, mbedTLS, and BoringSSL are some > examples) an API that they can implement that will guarantee that complying > modules can use the appropriate TLS backend. > > To that end, I?ve composed a draft PEP that would define this API: PEP > 543. This draft can be found online here[1], and I have reproduced it below > for those who want to reply inline. I should apologise in advance that this > PEP is quite large: it is this large because it lays out a substantial > proportion of the new code that would be added directly in the PEP. This is > to help de-risk the work by showing as much of it up-front during the PEP > stage, rather than having it emerge over the implementation process. > > Please note that this proposal is not intended to resolve the current > problem pip has with TLSv1.2, and so is not subject to the tight time > constraints associated with that problem. That problem will be solved by > building a temporary shim into urllib3 (and therefore pip) that can use > SecureTransport bindings on the Mac. This proposal is intended as a > solution to the more-general problem of supporting different TLS backends > in Python, and so is larger in scope and less urgent. > > I should also mention that there will be a tendency to want to make this > API all things to all people from the get-go. I?m going to strongly resist > attempts to extend this API too much, because each additional bit of API > surface makes it harder for us to encourage module authors to conform to > this API. I will encourage people to extend this API over time as needed, > but to begin with I think it is most important that basic TLS clients and > servers can be constructed with this API. More specialised features should > be considered future enhancements, rather than being tacked on to this > initial PEP. > > Please let me know what you think. I?d like significant feedback from the > community before I ask the BDFL to pronounce on this PEP, as this is the > kind of change that will affect a large amount of the Python ecosystem. > It?s important to me that we minimise the risk to ourselves and the > community by getting this as right as possible, and while we can help with > that by limiting scope, it?s also true that we need as many eyes as > possible. > > Please let me know what you think. > Heh, I guess you must have already caught everyone with a strong opinion about this by way of security-sig :) On the procedural front, the main open question is whether or not Guido wants to review and approve this PEP himself, or if he'd prefer to delegate that task. Assuming the latter case, I think it may make sense for Christian to handle the BDFL-Delegate responsibilities as well - I know he's a co-author of the PEP, but I also think he's the most qualified to make the final decision on the suitability of this design. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Feb 24 05:01:41 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Feb 2017 11:01:41 +0100 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) References: <87wpcktw4e.fsf@example.com> <20170224043657.GE5689@ando.pearwood.info> <1487922705.3562826.891373264.5AA7922D@webmail.messagingengine.com> Message-ID: <20170224110141.69b22c89@fsol> On Thu, 23 Feb 2017 23:51:45 -0800 Benjamin Peterson wrote: > > Like all CPython developers, the Python security team are all > volunteers. That combined with the fact that dealing with security > issues is one of the least fun programming tasks means issues are > sometimes dropped. > > Perhaps some organization with a stake Python security would like to > financially support Python security team members. > > As for this, particular issue, we should determine if there's a tracker > issue yet and continue discussion there. Just for the record, I find the mailing-list scheme used by PSRT quite difficult to deal with. For many people it's easy to lose track of e-mails received more than one week ago, so the necessary followup to security issues received by e-mail suffers. It's a bit sad that regular issues benefit from a full-fledged Roundup instance to allow for easy tracking of open issues (including comments and proposed fixes), but security issues are restricted to such a primitive communication setup which makes it so difficult to get work done. AFAIK, other projects have full-fledged private bug trackers for their security issues (or access-restricted sections in the main bug tracker, where the software supports it). Regards Antoine. From ncoghlan at gmail.com Fri Feb 24 05:02:08 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Feb 2017 20:02:08 +1000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On 23 February 2017 at 02:15, Victor Stinner wrote: > 2017-02-22 16:40 GMT+01:00 Antoine Pitrou : > > As long as you are asking for "moral" support and not actually > > vouching for the accuracy of third-party translations, then +1 from me. > > The main complain about these translations is the accuracy. > > My bet is that making these translations "official" and more visible > (at docs.python.org) would make them more popular, and so indirectly > help to recruit new contributors. Slowly, the number of translated > pages should increase, but the overall translation quality should also > increase. That's how free software are developed, no? :-) > +1 from me for these reasons, and those Facundo gives: we want folks to be able to learn at least the basics of Python *before* they learn English (even if learning English remains a pre-requisite for tapping into the full capabilities of both the language and its ecosystem). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Feb 24 05:03:21 2017 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 24 Feb 2017 11:03:21 +0100 Subject: [Python-Dev] PEP 543: A Unified TLS API for Python References: <0AC4A5C1-B6D3-41FF-80E0-2B26266A5956@lukasa.co.uk> Message-ID: <20170224110321.4e4c4b61@fsol> On Fri, 24 Feb 2017 19:55:20 +1000 Nick Coghlan wrote: > > On the procedural front, the main open question is whether or not Guido > wants to review and approve this PEP himself, or if he'd prefer to delegate > that task. > > Assuming the latter case, I think it may make sense for Christian to handle > the BDFL-Delegate responsibilities as well - I know he's a co-author of the > PEP, but I also think he's the most qualified to make the final decision on > the suitability of this design. Obviously not speaking for Guido, but I trust Christian to make a good judgement on this PEP. Regards Antoine. From ncoghlan at gmail.com Fri Feb 24 05:06:04 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Feb 2017 20:06:04 +1000 Subject: [Python-Dev] Report on Python vulnerabilities In-Reply-To: References: Message-ID: On 23 February 2017 at 11:15, Victor Stinner wrote: > Hi, > > I created a tool to generate a report on Python vulnerabilities: > > http://python-security.readthedocs.io/vulnerabilities.html > > I collected data of 41 vulnerabilities since 2007 (first Python CVE: > CVE-2007-4965). > Very handy! > If you would like to add data of a new vulnerability, enhance the > report, ... : see the GitHub project. > > https://github.com/haypo/python-security > > The main data file is vulnerabilities.yml (YAML). I also filled > manually the python_releases.txt: file: list of all release dates from > Python 2.5.0 to Python 3.6.0. > > The tool compute the first Python release in each branch which > includes the fix from a list of commits. > The main idea that comes to mind is finding a way to add a "Fixed In" column to the summary table to get a quick overview of which versions were affected. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From cory at lukasa.co.uk Fri Feb 24 05:21:15 2017 From: cory at lukasa.co.uk (Cory Benfield) Date: Fri, 24 Feb 2017 10:21:15 +0000 Subject: [Python-Dev] PEP 543: A Unified TLS API for Python In-Reply-To: References: <0AC4A5C1-B6D3-41FF-80E0-2B26266A5956@lukasa.co.uk> Message-ID: <5C941491-F1F1-4CA7-A766-5C26687B5C35@lukasa.co.uk> > On 24 Feb 2017, at 09:55, Nick Coghlan wrote: > > Heh, I guess you must have already caught everyone with a strong opinion about this by way of security-sig :) So it seems. =D > On the procedural front, the main open question is whether or not Guido wants to review and approve this PEP himself, or if he'd prefer to delegate that task. > > Assuming the latter case, I think it may make sense for Christian to handle the BDFL-Delegate responsibilities as well - I know he's a co-author of the PEP, but I also think he's the most qualified to make the final decision on the suitability of this design. Either way, I think I?d like to confirm this works by writing two more modules. First, I?d like to actually write most of the tls module into something that can live on PyPI, in no small part because backporting it would be useful for tools like urllib3. Secondly, I?d like to write a shim for the standard library ssl module so that I have a clearer picture of whether we want to shim it in Python code or whether we should take this opportunity to write new bindings to the C code. The goal here, essentially, is to have a PoC that the API is at least suitable for tools like urllib3/requests before we commit to it too much, and the best way to do that is to prove that the stdlib can meet that API and that at least one other implementation can do as well. I?ll use SecureTransport for that proof point, but I?ll make available a branch of urllib3 that has patches applied to use the new API so that anyone who wants to investigate shimming a different TLS implementation has got a test suite they can try to run. Obviously that?s slightly orthogonal to PEP acceptance, but I wanted to keep people up-to-speed with my thinking. Cory -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Feb 24 05:31:08 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 24 Feb 2017 20:31:08 +1000 Subject: [Python-Dev] PEP 543: A Unified TLS API for Python In-Reply-To: <5C941491-F1F1-4CA7-A766-5C26687B5C35@lukasa.co.uk> References: <0AC4A5C1-B6D3-41FF-80E0-2B26266A5956@lukasa.co.uk> <5C941491-F1F1-4CA7-A766-5C26687B5C35@lukasa.co.uk> Message-ID: On 24 February 2017 at 20:21, Cory Benfield wrote: > > Either way, I think I?d like to confirm this works by writing two more > modules. First, I?d like to actually write most of the tls module into > something that can live on PyPI, in no small part because backporting it > would be useful for tools like urllib3. Secondly, I?d like to write a shim > for the standard library ssl module so that I have a clearer picture of > whether we want to shim it in Python code or whether we should take this > opportunity to write new bindings to the C code. > > The goal here, essentially, is to have a PoC that the API is at least > suitable for tools like urllib3/requests before we commit to it too much, > and the best way to do that is to prove that the stdlib can meet that API > and that at least one other implementation can do as well. I?ll use > SecureTransport for that proof point, but I?ll make available a branch of > urllib3 that has patches applied to use the new API so that anyone who > wants to investigate shimming a different TLS implementation has got a test > suite they can try to run. > > Obviously that?s slightly orthogonal to PEP acceptance, but I wanted to > keep people up-to-speed with my thinking. > Having a sufficiently complete reference implementation in place to convince both the author(s) and the reviewers that the proposal solves the problem it claims to solve is part of the PEP process, so your planned approach makes a lot of sense to me. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Fri Feb 24 05:52:01 2017 From: tritium-list at sdamon.com (tritium-list at sdamon.com) Date: Fri, 24 Feb 2017 05:52:01 -0500 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) In-Reply-To: <20170224110141.69b22c89@fsol> References: <87wpcktw4e.fsf@example.com> <20170224043657.GE5689@ando.pearwood.info> <1487922705.3562826.891373264.5AA7922D@webmail.messagingengine.com> <20170224110141.69b22c89@fsol> Message-ID: <1b4101d28e8c$07f26f20$17d74d60$@hotmail.com> Ask the infrastructure team for a tracker instance. That would probably be more fruitful of an outlet than in the thread of this one issue. (I'm not trying to be flippant, I think a private issue tracker for vulnerabilities is a really good idea, I just don't think that bemoaning the lack of one in a thread about an FTP issue is likely to get much done.) > -----Original Message----- > From: Python-Dev [mailto:python-dev-bounces+tritium- > list=sdamon.com at python.org] On Behalf Of Antoine Pitrou > Sent: Friday, February 24, 2017 5:02 AM > To: python-dev at python.org > Subject: Re: [Python-Dev] Python FTP Injections Allow for Firewall Bypass > (oss-security advisory) > > On Thu, 23 Feb 2017 23:51:45 -0800 > Benjamin Peterson wrote: > > > > Like all CPython developers, the Python security team are all > > volunteers. That combined with the fact that dealing with security > > issues is one of the least fun programming tasks means issues are > > sometimes dropped. > > > > Perhaps some organization with a stake Python security would like to > > financially support Python security team members. > > > > As for this, particular issue, we should determine if there's a tracker > > issue yet and continue discussion there. > > Just for the record, I find the mailing-list scheme used by PSRT quite > difficult to deal with. For many people it's easy to lose track of > e-mails received more than one week ago, so the necessary followup to > security issues received by e-mail suffers. > > It's a bit sad that regular issues benefit from a full-fledged > Roundup instance to allow for easy tracking of open issues (including > comments and proposed fixes), but security issues are restricted to such > a primitive communication setup which makes it so difficult to get work > done. > > AFAIK, other projects have full-fledged private bug trackers for their > security issues (or access-restricted sections in the main bug tracker, > where the software supports it). > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > list%40sdamon.com From tritium-list at sdamon.com Fri Feb 24 06:01:59 2017 From: tritium-list at sdamon.com (tritium-list at sdamon.com) Date: Fri, 24 Feb 2017 06:01:59 -0500 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> My gut splits the difference on this issue; I suggest an approach to meet in the middle ? a version of the docs written in simplified English (Not quite Up Goer Five simplified, but simplified.) It has the practical benefit of having more eyes to look at the docs to check for accuracy, it targets both English learners and children, and is something the current contributors to the documentation can do. For any language you want to support other than English, you need a translator who is A: a python expert, B: fluent in English, and C: fluent in the target language. ?And then you need another one to check what was written. These are practical problems. There are extant services to support this, they are expensive in either money or time, and the docs produced usually lag behind English quite a bit. From: Python-Dev [mailto:python-dev-bounces+tritium-list=sdamon.com at python.org] On Behalf Of Nick Coghlan Sent: Friday, February 24, 2017 5:02 AM To: Victor Stinner Cc: Antoine Pitrou ; Python Dev Subject: Re: [Python-Dev] Translated Python documentation On 23 February 2017 at 02:15, Victor Stinner > wrote: 2017-02-22 16:40 GMT+01:00 Antoine Pitrou >: > As long as you are asking for "moral" support and not actually > vouching for the accuracy of third-party translations, then +1 from me. The main complain about these translations is the accuracy. My bet is that making these translations "official" and more visible (at docs.python.org ) would make them more popular, and so indirectly help to recruit new contributors. Slowly, the number of translated pages should increase, but the overall translation quality should also increase. That's how free software are developed, no? :-) +1 from me for these reasons, and those Facundo gives: we want folks to be able to learn at least the basics of Python *before* they learn English (even if learning English remains a pre-requisite for tapping into the full capabilities of both the language and its ecosystem). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.f.hilliard at gmail.com Fri Feb 24 06:12:06 2017 From: d.f.hilliard at gmail.com (Jim F.Hilliard) Date: Fri, 24 Feb 2017 13:12:06 +0200 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: I've recently been translating the Tutorial in Greek as a side thing that also helps me catch some doc bugs :-). So it's an obvious +1 from me. I really like this idea and would like to see it come to fruition. I think some issues need to be addressed first, though: Where should these translated docs live and how does one make it clear to users reading them that doc bugs shouldn't be submitted to the main bug tracker/github repo? these are just two that popped in my head. It would be great if this went forward, many people will find the path to learning Python much much easier to walk if the documentation was written in a language they are comfortable in. Best Regards, Jim Fasarakis Hilliard *d.f.hilliard at gmail.com | dimitrisjim.github.io * On Fri, Feb 24, 2017 at 12:02 PM, Nick Coghlan wrote: > On 23 February 2017 at 02:15, Victor Stinner > wrote: > >> 2017-02-22 16:40 GMT+01:00 Antoine Pitrou : >> > As long as you are asking for "moral" support and not actually >> > vouching for the accuracy of third-party translations, then +1 from me. >> >> The main complain about these translations is the accuracy. >> >> My bet is that making these translations "official" and more visible >> (at docs.python.org) would make them more popular, and so indirectly >> help to recruit new contributors. Slowly, the number of translated >> pages should increase, but the overall translation quality should also >> increase. That's how free software are developed, no? :-) >> > > +1 from me for these reasons, and those Facundo gives: we want folks to be > able to learn at least the basics of Python *before* they learn English > (even if learning English remains a pre-requisite for tapping into the full > capabilities of both the language and its ecosystem). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/d. > f.hilliard%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From d.f.hilliard at gmail.com Fri Feb 24 06:15:40 2017 From: d.f.hilliard at gmail.com (Jim F.Hilliard) Date: Fri, 24 Feb 2017 13:15:40 +0200 Subject: [Python-Dev] Translated Python documentation In-Reply-To: <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> Message-ID: I've recently been translating the Tutorial in Greek as a side thing that also helps me catch some doc bugs :-). So it's an obvious +1 from me. I really like this idea and would like to see it come to fruition. I think some issues need to be addressed first, though: Where should these translated docs live and how does one make it clear to users reading them that doc bugs shouldn't be submitted to the main bug tracker/github repo? these are just two that popped in my head. It would be great if this went forward, many people will find the path to learning Python much much easier to walk if, at least the tutorial, was written in a language they are comfortable in. -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Fri Feb 24 06:49:08 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 24 Feb 2017 20:49:08 +0900 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> Message-ID: > Where should these translated docs live and how does one make it clear to > users reading them that doc bugs shouldn't be submitted to the main bug > tracker/github repo? > We setup github page. See https://python-doc-ja.github.io/py36/ (note that version switcher won't work because this html is build for docs.python.jp.) I reserved "python-docs" organization. So we can use URL like https://python-doc.github.io//3.6/ (Because the organization name looks "something official", I won't use it until we get consensus about it.) For issue tracker, bugs.html must mention about translated document, before mention to normal issue tracker. From christian at python.org Fri Feb 24 06:55:33 2017 From: christian at python.org (Christian Heimes) Date: Fri, 24 Feb 2017 12:55:33 +0100 Subject: [Python-Dev] Python FTP Injections Allow for Firewall Bypass (oss-security advisory) In-Reply-To: <20170224110141.69b22c89@fsol> References: <87wpcktw4e.fsf@example.com> <20170224043657.GE5689@ando.pearwood.info> <1487922705.3562826.891373264.5AA7922D@webmail.messagingengine.com> <20170224110141.69b22c89@fsol> Message-ID: <5445d58d-4699-9836-4598-06c204f09948@python.org> On 2017-02-24 11:01, Antoine Pitrou wrote: > On Thu, 23 Feb 2017 23:51:45 -0800 > Benjamin Peterson wrote: >> >> Like all CPython developers, the Python security team are all >> volunteers. That combined with the fact that dealing with security >> issues is one of the least fun programming tasks means issues are >> sometimes dropped. >> >> Perhaps some organization with a stake Python security would like to >> financially support Python security team members. >> >> As for this, particular issue, we should determine if there's a tracker >> issue yet and continue discussion there. > > Just for the record, I find the mailing-list scheme used by PSRT quite > difficult to deal with. For many people it's easy to lose track of > e-mails received more than one week ago, so the necessary followup to > security issues received by e-mail suffers. > > It's a bit sad that regular issues benefit from a full-fledged > Roundup instance to allow for easy tracking of open issues (including > comments and proposed fixes), but security issues are restricted to such > a primitive communication setup which makes it so difficult to get work > done. > > AFAIK, other projects have full-fledged private bug trackers for their > security issues (or access-restricted sections in the main bug tracker, > where the software supports it). Amen! Antoine's and Benjamin's reply are the gist of my security talk at the last language summit, https://lwn.net/Articles/691308/ . A dedicated bug tracker or embargoed tickets would help the most. It would also make it much easier to track and measure our response time. A paid position would also help with the organizational overhead. Personally, I'm good in finding and fixing security issues. The actual communication, reporting and press releases are not my strength. Victor's incredible work on http://python-security.readthedocs.io/vulnerabilities.html is going to help, too. Christian From songofacandy at gmail.com Fri Feb 24 07:04:10 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 24 Feb 2017 21:04:10 +0900 Subject: [Python-Dev] Can I commit a hack for translation? (Was: Translated Python documentation Message-ID: Thanks, Victor and all. This thread is very encouraging for me. Currently, I have a suspended pull request: https://github.com/python/cpython/pull/195 This pull request (1) fixes "CPython implementation detail:" label is disappear when it's body is translated, and (2) make the label translatable. For (2), I managed to "make gettext" in Doc/ extract the label text into "sphinx.pot". So we can translate the label at Transifex. But I used ugly hack for it. This pull request "dummy.html" file which isn't used for building html. It only contains "{% trans %}CPython implementation detail:{% endtrans %}". We already did the hack in downstream fork of Japanese translation team. https://github.com/python-doc-ja/cpython-doc-intl/blob/intl-3.6/Doc/tools/templates/dummy.html There are one upvote and one downvote for the pull request. This is why I suspend the pull request. Please vote! a) Commit all. b) Commit (1), and call gettext() for the "CPython implementation detail:". (downstream fork should only add dummy.html). c) Commit only (1). (downstream fork should modify pyspecific extension and add dummy.html) From berker.peksag at gmail.com Fri Feb 24 07:20:41 2017 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Fri, 24 Feb 2017 15:20:41 +0300 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On Thu, Feb 23, 2017 at 12:01 AM, Victor Stinner wrote: > 2017-02-22 19:04 GMT+01:00 Serhiy Storchaka : >> What percent of lines is changed between bugfix releases? Feature releases? >> >> My largest apprehension is that the documentation can be outdated and >> quickly become degraded if main contributors left it. > > If the original text (english) changes, untranslated text is > displayed, not outdated text. I think that's much worse than showing the outdated text. I don't see any point on showing half English and half French text if the reader can't understand the other half of it. As someone who have spent a lot of time reviewing and committing documentation patches, I'm strongly against on marking documentation translations as official. The Python documentation updates frequently and it's simply not possible to keep them sync with the official documentation. See https://github.com/python/cpython/commits/master/Doc for the commit history of the official documentation. You can easily compare it with the translations by looking their GitHub repositories. Also, there are a lot of better educational materials (e.g. Django Girls Tutorial) for people who don't speak English and have no previous programming experience. Even the tutorial contains several references to different programming concepts and programming languages such as C++. --Berker From victor.stinner at gmail.com Fri Feb 24 07:26:50 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 24 Feb 2017 13:26:50 +0100 Subject: [Python-Dev] Report on Python vulnerabilities In-Reply-To: References: Message-ID: 2017-02-24 11:06 GMT+01:00 Nick Coghlan : > The main idea that comes to mind is finding a way to add a "Fixed In" column > to the summary table to get a quick overview of which versions were > affected. I had this column, just I just removed it before my email to python-dev because the table was too wide. I fixed my script: I reimplemented the tabulate for my needs, to support multiline table cells. I also enhanced the compilation process for ReadTheDocs to be able to remove the generated vulnerabilities.rst file from Git. Victor From victor.stinner at gmail.com Fri Feb 24 07:35:50 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 24 Feb 2017 13:35:50 +0100 Subject: [Python-Dev] Can I commit a hack for translation? (Was: Translated Python documentation In-Reply-To: References: Message-ID: This change has zero impact on docs.python.org, it only help teams working on translation, so go ahead. Victor From berker.peksag at gmail.com Fri Feb 24 08:04:51 2017 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Fri, 24 Feb 2017 16:04:51 +0300 Subject: [Python-Dev] Can I commit a hack for translation? (Was: Translated Python documentation In-Reply-To: References: Message-ID: On Fri, Feb 24, 2017 at 3:35 PM, Victor Stinner wrote: > This change has zero impact on docs.python.org, it only help teams > working on translation, so go ahead. This change creates an unnecessary maintenance burden for people who works on Python documentation. --Berker From songofacandy at gmail.com Fri Feb 24 08:05:12 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 24 Feb 2017 22:05:12 +0900 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On Fri, Feb 24, 2017 at 9:20 PM, Berker Peksa? wrote: > On Thu, Feb 23, 2017 at 12:01 AM, Victor Stinner > wrote: >> 2017-02-22 19:04 GMT+01:00 Serhiy Storchaka : >>> What percent of lines is changed between bugfix releases? Feature releases? >>> >>> My largest apprehension is that the documentation can be outdated and >>> quickly become degraded if main contributors left it. >> >> If the original text (english) changes, untranslated text is >> displayed, not outdated text. > > I think that's much worse than showing the outdated text. I don't see > any point on showing half English and half French text if the reader > can't understand the other half of it. In Japan, everyone learn English at school, but most people is not good at English. So 80% translated text helps us very much. Especially, Sphinx split document by paragraph. It's best granularity for translation. For example, there are some English paragraph remains in pathlib document . But it's very useful for Japanese not good at English. > > As someone who have spent a lot of time reviewing and committing > documentation patches, I'm strongly against on marking documentation > translations as official. I totally agree with you. Our QA is not good as commit review of CPython. So what I want is (un|semi) official place for we share our efforts with other language translators. (e.g. automated build, hosting translated documentation, and downstream customizations like adding link to official English document). > The Python documentation updates frequently > and it's simply not possible to keep them sync with the official > documentation. See > https://github.com/python/cpython/commits/master/Doc for the commit > history of the official documentation. You can easily compare it with > the translations by looking their GitHub repositories. While we don't translate mastar branch, I admit we behind several weeks or months from upstream. But since we moved most work on Travis, we can make it weekly or even daily. And we can share the automation with other Languages if we have a team. I can do it without official agreement, but I don't want to do it until I get consensus about domain name or github organization name. > > Also, there are a lot of better educational materials (e.g. Django > Girls Tutorial) for people who don't speak English and have no > previous programming experience. Even the tutorial contains several > references to different programming concepts and programming languages > such as C++. > Sadly, there are many medium level programmers who are not good at English in Japan. So translated library reference is very helpful too. Regards, From ncoghlan at gmail.com Fri Feb 24 09:33:54 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Feb 2017 00:33:54 +1000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On 24 February 2017 at 23:05, INADA Naoki wrote: > On Fri, Feb 24, 2017 at 9:20 PM, Berker Peksa? > wrote: > > > > As someone who have spent a lot of time reviewing and committing > > documentation patches, I'm strongly against on marking documentation > > translations as official. > > I totally agree with you. Our QA is not good as commit review of CPython. > So what I want is (un|semi) official place for we share our efforts with > other > language translators. (e.g. automated build, hosting translated > documentation, > and downstream customizations like adding link to official English > document). > > > > The Python documentation updates frequently > > and it's simply not possible to keep them sync with the official > > documentation. See > > https://github.com/python/cpython/commits/master/Doc for the commit > > history of the official documentation. You can easily compare it with > > the translations by looking their GitHub repositories. > > While we don't translate mastar branch, I admit we behind several weeks or > months from upstream. > But since we moved most work on Travis, we can make it weekly or even > daily. > And we can share the automation with other Languages if we have a team. > Right, I think this is the key: helping the language translation communities set up common flows so they can collaborate on the backend automation, request process or tech changes in the main docs to simplify translation (such as resolving the issue with "implementation detail" notes disappearing when translated), and generally improving discoverability of the translated versions. In addition to the case of folks that struggle to read the English documentation at all, I'd assume that there are also folks that would appreciate the chance to check their own understanding against someone else's translation of various topics. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Feb 24 10:10:02 2017 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 25 Feb 2017 02:10:02 +1100 Subject: [Python-Dev] Translated Python documentation In-Reply-To: <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> Message-ID: <20170224151002.GG5689@ando.pearwood.info> On Fri, Feb 24, 2017 at 06:01:59AM -0500, tritium-list at sdamon.com wrote: > My gut splits the difference on this issue; I suggest an approach to > meet in the middle ? a version of the docs written in simplified > English (Not quite Up Goer Five simplified, but simplified.) As an English speaker, my gut tells me that it would be much harder to write *accurate* simplified English technical documentation than to translate it into another language. You have all the difficulties of translation, plus you're working under a handicap of only using some (ill-defined?) subset of English. Wikipedia offers some evidence supporting my view: - the main English Wikipedia has 5 million articles, written by nearly 140K active users; - the Swedish Wikipedia is almost as big, 3M articles from only 3K active users; - but the Simple English Wikipedia has just 123K articles and 871 active users. That's fewer articles than Esperanto! https://meta.wikimedia.org/wiki/List_of_Wikipedias Nevertheless, I certainly wouldn't object if people wanted to try writing Simple English translations of the docs. But I don't think they would be as useful as translations into non-English. [...] > For any language you want to support other than English, you need a > translator who is A: a python expert, B: fluent in English, and C: > fluent in the target language. I disagree. You need a translator who is A: skilled at technical documentation, with B and C as above. They don't need to be a Python expert. We have plenty of Python experts that they can consult with and ask questions. But they need to know the right questions to ask: "Python attributes, they're kind of like C members, right? I would translate 'member' into Klingon as 'gham', which means 'arm or leg', so I can use the same word for attribute." > ?And then you need another one to > check what was written. These are practical problems. There are > extant services to support this, they are expensive in either money or > time, and the docs produced usually lag behind English quite a bit. Is this a good use for some PSF funding? Would companies be willing to invest money in translating Python documentation? Just because we're Open Source, doesn't mean that everything we do has to be purely volunteer. -- Steve From status at bugs.python.org Fri Feb 24 12:09:14 2017 From: status at bugs.python.org (Python tracker) Date: Fri, 24 Feb 2017 18:09:14 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20170224170914.3DFB156758@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2017-02-17 - 2017-02-24) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5833 ( -4) closed 35565 (+54) total 41398 (+50) Open issues with patches: 2478 Issues opened (32) ================== #27840: functools.partial: don't copy keywoard arguments in partial_ca http://bugs.python.org/issue27840 reopened by haypo #29593: Improve UnboundLocalError message for deleted names http://bugs.python.org/issue29593 opened by mbussonn #29594: implementation of __or__ in enum.auto http://bugs.python.org/issue29594 opened by magu #29595: Expose max_queue_size in ThreadPoolExecutor http://bugs.python.org/issue29595 opened by prayerslayer #29596: Unfinished sentense in howto/clinic.rst http://bugs.python.org/issue29596 opened by mbussonn #29598: Write unit tests for pdb module http://bugs.python.org/issue29598 opened by Artem Muterko #29600: Returning an exception object from a coroutine triggers implic http://bugs.python.org/issue29600 opened by njs #29601: Need reST markup for enum types http://bugs.python.org/issue29601 opened by barry #29606: urllib FTP protocol stream injection http://bugs.python.org/issue29606 opened by ecbftw #29611: TextIOWrapper's write_through option behave differently betwee http://bugs.python.org/issue29611 opened by inada.naoki #29612: TarFile.extract() suffers from hard links inside tarball http://bugs.python.org/issue29612 opened by Jussi Judin #29613: Support for SameSite Cookies http://bugs.python.org/issue29613 opened by akash0x53 #29615: SimpleXMLRPCDispatcher._dispatch mangles tracebacks when invok http://bugs.python.org/issue29615 opened by petr at motejlek.net #29616: input() after restarting causes bug http://bugs.python.org/issue29616 opened by Juan #29617: Drop Python 3.4 support from asyncio http://bugs.python.org/issue29617 opened by inada.naoki #29619: st_ino (unsigned long long) is casted to long long in posixmod http://bugs.python.org/issue29619 opened by mba #29620: unittest.TestCase.assertWarns raises RuntimeEror if sys.module http://bugs.python.org/issue29620 opened by kernc #29621: telnetlib.Telnet.write gives confusing error message when a st http://bugs.python.org/issue29621 opened by John Hagen #29623: configparser.ConfigParser.read() does not accept Pathlib path http://bugs.python.org/issue29623 opened by David Ellis #29624: Python 3.5.3 x86 web installer cannot install launcher http://bugs.python.org/issue29624 opened by steve.dower #29626: Issue with spacing in argparse module while using help http://bugs.python.org/issue29626 opened by falu2010 #29627: configparser.ConfigParser.read() has undocumented/unexpected b http://bugs.python.org/issue29627 opened by David Ellis #29631: Error ???importlib.h, importlib_external.h updated. You will n http://bugs.python.org/issue29631 opened by Karen #29632: argparse values for action in add_argument() should be flags i http://bugs.python.org/issue29632 opened by pgacv2 #29633: MSI installer does not pass values as SecureProperty from UI http://bugs.python.org/issue29633 opened by aldehoff #29636: Specifying indent in the json.tool command http://bugs.python.org/issue29636 opened by dhimmel #29637: ast.get_docstring(): AttributeError: 'NoneType' object has no http://bugs.python.org/issue29637 opened by jwilk #29638: Spurious failures in test_collections in releak hunting mode a http://bugs.python.org/issue29638 opened by levkivskyi #29639: test suite intentionally avoids referring to localhost, destro http://bugs.python.org/issue29639 opened by gregory.p.smith #29640: _PyThreadState_Init and fork race leads to inconsistent key li http://bugs.python.org/issue29640 opened by J??n Stan??ek #29641: make install failure when using ./configure --enable-optimizat http://bugs.python.org/issue29641 opened by haypo #29642: Why does unittest.TestLoader.discover still rely on existence http://bugs.python.org/issue29642 opened by Andrei Fokau Most recent 15 issues with no replies (15) ========================================== #29642: Why does unittest.TestLoader.discover still rely on existence http://bugs.python.org/issue29642 #29640: _PyThreadState_Init and fork race leads to inconsistent key li http://bugs.python.org/issue29640 #29639: test suite intentionally avoids referring to localhost, destro http://bugs.python.org/issue29639 #29638: Spurious failures in test_collections in releak hunting mode a http://bugs.python.org/issue29638 #29637: ast.get_docstring(): AttributeError: 'NoneType' object has no http://bugs.python.org/issue29637 #29636: Specifying indent in the json.tool command http://bugs.python.org/issue29636 #29633: MSI installer does not pass values as SecureProperty from UI http://bugs.python.org/issue29633 #29631: Error ???importlib.h, importlib_external.h updated. You will n http://bugs.python.org/issue29631 #29627: configparser.ConfigParser.read() has undocumented/unexpected b http://bugs.python.org/issue29627 #29623: configparser.ConfigParser.read() does not accept Pathlib path http://bugs.python.org/issue29623 #29621: telnetlib.Telnet.write gives confusing error message when a st http://bugs.python.org/issue29621 #29620: unittest.TestCase.assertWarns raises RuntimeEror if sys.module http://bugs.python.org/issue29620 #29619: st_ino (unsigned long long) is casted to long long in posixmod http://bugs.python.org/issue29619 #29617: Drop Python 3.4 support from asyncio http://bugs.python.org/issue29617 #29615: SimpleXMLRPCDispatcher._dispatch mangles tracebacks when invok http://bugs.python.org/issue29615 Most recent 15 issues waiting for review (15) ============================================= #29623: configparser.ConfigParser.read() does not accept Pathlib path http://bugs.python.org/issue29623 #29615: SimpleXMLRPCDispatcher._dispatch mangles tracebacks when invok http://bugs.python.org/issue29615 #29613: Support for SameSite Cookies http://bugs.python.org/issue29613 #29568: undefined parsing behavior with the old style string formattin http://bugs.python.org/issue29568 #29557: binhex documentation claims unknown bug http://bugs.python.org/issue29557 #29555: Update Python Software Foundation Copyright Year http://bugs.python.org/issue29555 #29553: Argparser does not display closing parentheses in nested mutex http://bugs.python.org/issue29553 #29549: Improve docstring for str.index http://bugs.python.org/issue29549 #29540: Add compact=True flag to json.dump/dumps http://bugs.python.org/issue29540 #29537: Alternative fix for BUILD_MAP_UNPACK_WITH_CALL opcode in 3.5 http://bugs.python.org/issue29537 #29533: urllib2 works slowly with proxy on windows http://bugs.python.org/issue29533 #29523: latest setuptools breaks virtualenvs due to unbundling depende http://bugs.python.org/issue29523 #29507: Use FASTCALL in call_method() to avoid temporary tuple http://bugs.python.org/issue29507 #29476: Simplify set_add_entry() http://bugs.python.org/issue29476 #29475: option to not follow symlinks when globbing http://bugs.python.org/issue29475 Top 10 most discussed issues (10) ================================= #22273: abort when passing certain structs by value using ctypes http://bugs.python.org/issue22273 20 msgs #29549: Improve docstring for str.index http://bugs.python.org/issue29549 13 msgs #26789: Please do not log during shutdown http://bugs.python.org/issue26789 9 msgs #29176: /tmp does not exist on Android and is used by curses.window.pu http://bugs.python.org/issue29176 9 msgs #29592: abs_paths() in site.py is slow http://bugs.python.org/issue29592 7 msgs #29616: input() after restarting causes bug http://bugs.python.org/issue29616 7 msgs #29533: urllib2 works slowly with proxy on windows http://bugs.python.org/issue29533 6 msgs #28909: Adding LTTng-UST tracing support http://bugs.python.org/issue28909 5 msgs #29355: sqlite3: remove sqlite3_stmt_readonly() http://bugs.python.org/issue29355 5 msgs #29581: __init_subclass__ causes TypeError when used with standard lib http://bugs.python.org/issue29581 5 msgs Issues closed (53) ================== #9592: Limitations in objects returned by multiprocessing Pool http://bugs.python.org/issue9592 closed by serhiy.storchaka #10701: Error pickling objects with mutating __getstate__ http://bugs.python.org/issue10701 closed by serhiy.storchaka #11299: Allow deepcopying paused generators http://bugs.python.org/issue11299 closed by serhiy.storchaka #15657: Error in Python 3 docs for PyMethodDef http://bugs.python.org/issue15657 closed by barry #18195: error when deep copying module is confusing http://bugs.python.org/issue18195 closed by serhiy.storchaka #18423: Document limitations on -m http://bugs.python.org/issue18423 closed by ncoghlan #22807: uuid.uuid1() should use uuid_generate_time_safe() if available http://bugs.python.org/issue22807 closed by barry #24274: erroneous comments in dictobject.c http://bugs.python.org/issue24274 closed by inada.naoki #25617: Installing local installation of Python http://bugs.python.org/issue25617 closed by inada.naoki #26382: List object memory allocator http://bugs.python.org/issue26382 closed by inada.naoki #26970: Replace OpenSSL's CPRNG with system entropy source http://bugs.python.org/issue26970 closed by christian.heimes #27660: Replace size_t with Py_ssize_t as the type of local variable i http://bugs.python.org/issue27660 closed by xiang.zhang #27788: platform module's version number doesn't match its docstring http://bugs.python.org/issue27788 closed by lemburg #28121: If module starts with comment or empty line then frame.f_code. http://bugs.python.org/issue28121 closed by Aivar.Annamaa #28814: Deprecation notice on inspect.getargvalues() is incorrect http://bugs.python.org/issue28814 closed by ncoghlan #29165: Use forward compatible macro in example code for creating new http://bugs.python.org/issue29165 closed by inada.naoki #29347: Python could crash while creating weakref for a given object http://bugs.python.org/issue29347 closed by xiang.zhang #29436: Compilation failure against Android NDK r14 beta 2 http://bugs.python.org/issue29436 closed by xdegaye #29453: Remove reference to undefined dictionary ordering in Tutorial http://bugs.python.org/issue29453 closed by Mariatta #29463: Add `docstring` field to AST nodes http://bugs.python.org/issue29463 closed by inada.naoki #29503: Make embedded-Python detectable http://bugs.python.org/issue29503 closed by steve.dower #29509: redundant interning in PyObject_GetAttrString http://bugs.python.org/issue29509 closed by inada.naoki #29520: Documentation uses deprecated "defindex.html" Sphinx template http://bugs.python.org/issue29520 closed by inada.naoki #29525: Python 2.7.13 for Windows broken (from prompt) http://bugs.python.org/issue29525 closed by steve.dower #29529: Backport Travis configuration http://bugs.python.org/issue29529 closed by inada.naoki #29532: functools.partial is not compatible between 2.7 and 3.5 http://bugs.python.org/issue29532 closed by serhiy.storchaka #29546: A more helpful ImportError message http://bugs.python.org/issue29546 closed by barry #29554: profile/pstat doc clariification http://bugs.python.org/issue29554 closed by berker.peksag #29565: Still broken ctypes calling convention on MSVC / 64-bit Window http://bugs.python.org/issue29565 closed by vinay.sajip #29571: test_re is failing when local is set for `en_IN` http://bugs.python.org/issue29571 closed by ncoghlan #29575: doc 17.2.1: basic Pool example is too basic http://bugs.python.org/issue29575 closed by davin #29579: Windows Python 3.7 installer broken by README.txt renamed to R http://bugs.python.org/issue29579 closed by steve.dower #29580: "Built-in Functions" not being functions http://bugs.python.org/issue29580 closed by rhettinger #29589: test_asyncio & test_multiprocessing_forkserver failed http://bugs.python.org/issue29589 closed by xiang.zhang #29597: __new__ / __init__ calls during unpickling not documented corr http://bugs.python.org/issue29597 closed by serhiy.storchaka #29599: Github organization link is secret and not accessible from cor http://bugs.python.org/issue29599 closed by berker.peksag #29602: complex() on object with __complex__ function loses sign of ze http://bugs.python.org/issue29602 closed by mark.dickinson #29603: More informative Queue class: new method that returns number o http://bugs.python.org/issue29603 closed by slytomcat #29604: ImportError when importing the package `queue` http://bugs.python.org/issue29604 closed by christian.heimes #29605: platform.architecture() with Python2.7-32 misreports architect http://bugs.python.org/issue29605 closed by lemburg #29607: Broken stack_effect for CALL_FUNCTION_EX http://bugs.python.org/issue29607 closed by inada.naoki #29608: pip_gui http://bugs.python.org/issue29608 closed by berker.peksag #29609: Receiving messages from subprocess does not work on Mac OS X http://bugs.python.org/issue29609 closed by ned.deily #29610: ssl do_handshake fails on https-proxy (aka. https over https-p http://bugs.python.org/issue29610 closed by martin.panter #29614: Additional implementation alternative to DictReader http://bugs.python.org/issue29614 closed by rhettinger #29618: Making a Tuple or Dict with Callable[ ..., xxx??] gives error http://bugs.python.org/issue29618 closed by Pauli Salmenrinne #29622: ast module doesn't support optional fields of Parser/Python.as http://bugs.python.org/issue29622 closed by inada.naoki #29625: colorsys.rgb_to_hsv always returns saturation as 0 (python2.7 http://bugs.python.org/issue29625 closed by Will Pittman #29628: struct objects can no longer be pickled http://bugs.python.org/issue29628 closed by serhiy.storchaka #29629: rgb_to_hls in colorsys.py http://bugs.python.org/issue29629 closed by madhavendra.sharma #29630: REPL tab-completion triggers code execution http://bugs.python.org/issue29630 closed by christian.heimes #29634: Reduce deque repeat execution when maxlen exist and size is no http://bugs.python.org/issue29634 closed by rhettinger #29635: os.chdir() acts like `cd -P` by default, it should offer an op http://bugs.python.org/issue29635 closed by serhiy.storchaka From python at mrabarnett.plus.com Fri Feb 24 13:54:46 2017 From: python at mrabarnett.plus.com (MRAB) Date: Fri, 24 Feb 2017 18:54:46 +0000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: <20170224151002.GG5689@ando.pearwood.info> References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> <20170224151002.GG5689@ando.pearwood.info> Message-ID: <6fad474b-8fe7-2017-fe93-ad22b600df1b@mrabarnett.plus.com> On 2017-02-24 15:10, Steven D'Aprano wrote: > On Fri, Feb 24, 2017 at 06:01:59AM -0500, tritium-list at sdamon.com wrote: > >> My gut splits the difference on this issue; I suggest an approach to >> meet in the middle ? a version of the docs written in simplified >> English (Not quite Up Goer Five simplified, but simplified.) > > As an English speaker, my gut tells me that it would be much harder to > write *accurate* simplified English technical documentation than to > translate it into another language. > > You have all the difficulties of translation, plus you're working under > a handicap of only using some (ill-defined?) subset of English. > > Wikipedia offers some evidence supporting my view: > > - the main English Wikipedia has 5 million articles, written by nearly > 140K active users; > > - the Swedish Wikipedia is almost as big, 3M articles from only 3K > active users; > > - but the Simple English Wikipedia has just 123K articles and 871 > active users. That's fewer articles than Esperanto! > > https://meta.wikimedia.org/wiki/List_of_Wikipedias > > > Nevertheless, I certainly wouldn't object if people wanted to try > writing Simple English translations of the docs. But I don't think they > would be as useful as translations into non-English. > [snip] Would it be easier to make a translation into Esperanto, which is meant to be easier to learn than English? From victor.stinner at gmail.com Fri Feb 24 18:50:57 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Sat, 25 Feb 2017 00:50:57 +0100 Subject: [Python-Dev] Reports on my CPython contributions Message-ID: Hi, Recently, I wrote reports of my CPython contributions since 1 year 1/2. Some people on this list might be interested, so here is the list. I was also asked for more documentation about my work on FASTCALL and its effect on performance. You can expect more article on FASTCALL later, especially on its API and its implementaiton. == CPython contrib 2016 == My contributions to CPython during 2016 Q1: https://haypo.github.io/contrib-cpython-2016q1.html My contributions to CPython during 2016 Q2: https://haypo.github.io/contrib-cpython-2016q2.html PEP 524: os.urandom() now blocks on Linux in Python 3.6: https://haypo.github.io/pep-524-os-urandom-blocking.html CPython sprint, september 2016: https://haypo.github.io/cpython-sprint-2016.html My contributions to CPython during 2016 Q3: https://haypo.github.io/contrib-cpython-2016q3.html My contributions to CPython during 2016 Q4: https://haypo.github.io/contrib-cpython-2016q4.html == FASTCALL project == The start of the FASTCALL project: https://haypo.github.io/start-fastcall-project.html FASTCALL microbenchmarks: https://haypo.github.io/fastcall-microbenchmarks.html FASTCALL issues: https://haypo.github.io/fastcall-issues.html == CPython contrib 2015 == My contributions to CPython during 2015 Q3: https://haypo.github.io/contrib-cpython-2015q3.html My contributions to CPython during 2015 Q4: https://haypo.github.io/contrib-cpython-2015q4.html Victor From ncoghlan at gmail.com Sat Feb 25 01:32:18 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Feb 2017 16:32:18 +1000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: <20170224151002.GG5689@ando.pearwood.info> References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> <20170224151002.GG5689@ando.pearwood.info> Message-ID: On 25 February 2017 at 01:10, Steven D'Aprano wrote: > On Fri, Feb 24, 2017 at 06:01:59AM -0500, tritium-list at sdamon.com wrote: > > ?And then you need another one to > > check what was written. These are practical problems. There are > > extant services to support this, they are expensive in either money or > > time, and the docs produced usually lag behind English quite a bit. > > Is this a good use for some PSF funding? Would companies be willing to > invest money in translating Python documentation? > Translated documentation is certainly one of the things commercial open source users appreciate, and hence redistributors are willing to fund in the general case (see https://access.redhat.com/documentation/ja/red-hat-enterprise-linux/ or https://docs.openstack.org/ja/ for example) For the specific case of the PSF, credible development grant ideas are always worth considering (as those are an excellent way for the PSF to help enable community activities). > Just because we're Open Source, doesn't mean that everything we do has > to be purely volunteer. > Right, but this kind of problem is also one of the major reasons I tend to harp on about the fact that the commercial redistributors active in the Python community aren't contributing as effectively as they could be, and their existing customers aren't holding them accountable for the failure. Python's origins as a "scripting language for Linux" means it is often still perceived that way in the commercial sector and treated accordingly, even though it has subsequently matured into a full-fledged cross-platform application development and data analysis platform. Even those of us already working for redistributors can't readily provide that missing accountability, as any reasonable business is going to weigh the costs of additional investment in any given area against the potential for increased future revenue. That means the most effective pressure comes from industry partners, governments, existing customers, and credible potential customers (i.e. folks that have the ability to affect redistributor revenue directly) rather than from redistributor staff or community volunteers (as we're pretty much limited to using risk management and hiring pipeline based lines of argument, rather than being able to push the "future revenue potential" line directly). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 25 01:46:06 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 25 Feb 2017 16:46:06 +1000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: <6fad474b-8fe7-2017-fe93-ad22b600df1b@mrabarnett.plus.com> References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> <20170224151002.GG5689@ando.pearwood.info> <6fad474b-8fe7-2017-fe93-ad22b600df1b@mrabarnett.plus.com> Message-ID: On 25 February 2017 at 04:54, MRAB wrote: > On 2017-02-24 15:10, Steven D'Aprano wrote: > >> Nevertheless, I certainly wouldn't object if people wanted to try >> writing Simple English translations of the docs. But I don't think they >> would be as useful as translations into non-English. >> >> [snip] > > Would it be easier to make a translation into Esperanto, which is meant to > be easier to learn than English? Not really, as there are two main criteria for building a successful translation community: - potential audience for the translation (i.e. native speakers of the target language) - an interested pool of bilingual speakers to do the translation Those numbers are far more favourable for native language translations to widely spoken languages like Japanese, Mandarin Chinese, Spanish, and French than they are for Esperanto. When it comes to the idea of creating a Simple English translation of the standard tutorial, my experience is that what folks are more inclined to do on that front is to just write a new tutorial of their own specifically targeting the audience they care about, rather than translating the standard one. That provides a lot more freedom to not only adjust the specific words used, but also decide when and how to introduce different concepts based on their particular target audience (e.g. teaching Python to a group of mid-career scientific researchers or a group of professional engineers is very different from teaching it to a classroom full of 6 year olds, or students pursuing a computing degree at university, which in turn are very different from teaching groups of interested people that have designed to sacrifice their own time to attend a community run introduction to programming workshop). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sat Feb 25 02:40:00 2017 From: larry at hastings.org (Larry Hastings) Date: Fri, 24 Feb 2017 23:40:00 -0800 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: <09c0a605-41a9-b045-82e6-0b2be170f928@trueblade.com> References: <09c0a605-41a9-b045-82e6-0b2be170f928@trueblade.com> Message-ID: <2eb15163-ecab-1d12-768e-1012bbab8211@hastings.org> On 02/22/2017 08:26 AM, Eric V. Smith wrote: > On 2/22/17 7:27 AM, Victor Stinner wrote: >> OpenStack happily fixed this issue one or two years ago with "reno": >> https://pypi.python.org/pypi/reno >> [...] >> What do you think of the overall idea? > I'm for it. I'd be interested in hearing what current and former > release managers say. I'm for it too. Fixing up Misc/NEWS is probably the trickiest and most error-prone part of cutting a release. We've discussed this before, somewhere on bpo though I don't have the issue no. handy. I actually wrote a tool specifically to solve this problem at the time: https://bitbucket.org/larry/mergenews Obviously it's out of date as it's designed to work with Mercurial. I don't know how such a tool would be integrated into a Git-based workflow. While I'm sure we can learn a lot by examining "reno", my suspicion is we'll want our own custom tool. I don't think the hard part is writing the tool--the hard part is figuring out what we want and how it should be integrated into our workflow. Actually writing such a tool is easy and fun, and I'm sure our community would fall all over themselves to write it if asked. In fact, I'll kind of pre-volunteer to write it right now, as long as I can write it in Python 3 ;-) //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sat Feb 25 07:26:55 2017 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 25 Feb 2017 14:26:55 +0200 Subject: [Python-Dev] Statistics of documentation changes Message-ID: For the discussion about translating Python documentation I have collected some statistics. See the attachment. The difference of the documentation between recent feature releases is only 5-7%. Early bugfix releases change about 1-2% of the documentation. The tutorial is more stable. There is only 2% difference between recent releases. Early bugfix releases change about 1% of the tutorial. -------------- next part -------------- Doc/ changes: total: 557 files changed, 238295 insertions(+), 43 deletions(-) 3.2 -> 3.3: 324 files changed, 30040 insertions(+), 10043 deletions(-) 3.3 -> 3.4: 390 files changed, 29534 insertions(+), 10609 deletions(-) 3.4 -> 3.5: 407 files changed, 16543 insertions(+), 7020 deletions(-) 3.5 -> 3.6: 186 files changed, 10966 insertions(+), 3564 deletions(-) v2.7 -> v2.7.1: 165 files changed, 3295 insertions(+), 1578 deletions(-) v2.7.1 -> v2.7.2: 113 files changed, 6418 insertions(+), 5082 deletions(-) v2.7.2 -> v2.7.3: 312 files changed, 7191 insertions(+), 6729 deletions(-) v2.7.3 -> v2.7.4: 263 files changed, 5128 insertions(+), 2979 deletions(-) v2.7.4 -> v2.7.5: 59 files changed, 1039 insertions(+), 1144 deletions(-) v2.7.5 -> v2.7.6: 144 files changed, 2183 insertions(+), 1556 deletions(-) v2.7.6 -> v2.7.7: 108 files changed, 1348 insertions(+), 856 deletions(-) v2.7.7 -> v2.7.8: 33 files changed, 413 insertions(+), 115 deletions(-) v2.7.8 -> v2.7.9: 168 files changed, 5464 insertions(+), 3623 deletions(-) v2.7.9 -> v2.7.10: 59 files changed, 1591 insertions(+), 1199 deletions(-) v2.7.10 -> v2.7.11: 88 files changed, 810 insertions(+), 480 deletions(-) v2.7.11 -> v2.7.12: 198 files changed, 2420 insertions(+), 1744 deletions(-) v2.7.12 -> v2.7.13: 168 files changed, 1199 insertions(+), 778 deletions(-) v3.2 -> v3.2.1: 114 files changed, 1987 insertions(+), 1500 deletions(-) v3.2.1 -> v3.2.2: 81 files changed, 871 insertions(+), 562 deletions(-) v3.2.2 -> v3.2.3: 155 files changed, 2473 insertions(+), 3859 deletions(-) v3.2.3 -> v3.2.4: 219 files changed, 6427 insertions(+), 3319 deletions(-) v3.2.4 -> v3.2.5: v3.2.5 -> v3.2.6: 7 files changed, 89 insertions(+), 27 deletions(-) v3.3.0 -> v3.3.1: 204 files changed, 5196 insertions(+), 3645 deletions(-) v3.3.1 -> v3.3.2: 93 files changed, 1632 insertions(+), 1514 deletions(-) v3.3.2 -> v3.3.3: 184 files changed, 3339 insertions(+), 2507 deletions(-) v3.3.3 -> v3.3.4: 105 files changed, 1249 insertions(+), 583 deletions(-) v3.3.4 -> v3.3.5: 34 files changed, 323 insertions(+), 240 deletions(-) v3.3.5 -> v3.3.6: 34 files changed, 477 insertions(+), 502 deletions(-) v3.4.0 -> v3.4.1: 81 files changed, 1039 insertions(+), 772 deletions(-) v3.4.1 -> v3.4.2: 94 files changed, 2723 insertions(+), 1130 deletions(-) v3.4.2 -> v3.4.3: 220 files changed, 6152 insertions(+), 5526 deletions(-) v3.4.3 -> v3.4.4: 208 files changed, 3091 insertions(+), 1833 deletions(-) v3.4.4 -> v3.4.5: 16 files changed, 108 insertions(+), 114 deletions(-) v3.4.5 -> v3.4.6: 4 files changed, 28 insertions(+), 4 deletions(-) v3.5.0 -> v3.5.1: 143 files changed, 2337 insertions(+), 1196 deletions(-) v3.5.1 -> v3.5.2: 348 files changed, 4170 insertions(+), 3548 deletions(-) v3.5.2 -> v3.5.3: 223 files changed, 2838 insertions(+), 1761 deletions(-) Doc/tutorial/ changes: total: 17 files changed, 6145 insertions(+) 3.2 -> 3.3: 10 files changed, 516 insertions(+), 565 deletions(-) 3.3 -> 3.4: 15 files changed, 549 insertions(+), 348 deletions(-) 3.4 -> 3.5: 14 files changed, 150 insertions(+), 170 deletions(-) 3.5 -> 3.6: 9 files changed, 122 insertions(+), 101 deletions(-) v2.7 -> v2.7.1: 6 files changed, 123 insertions(+), 103 deletions(-) v2.7.1 -> v2.7.2: 5 files changed, 34 insertions(+), 40 deletions(-) v2.7.2 -> v2.7.3: 8 files changed, 218 insertions(+), 122 deletions(-) v2.7.3 -> v2.7.4: 11 files changed, 169 insertions(+), 98 deletions(-) v2.7.4 -> v2.7.5: 2 files changed, 8 insertions(+), 18 deletions(-) v2.7.5 -> v2.7.6: 7 files changed, 64 insertions(+), 54 deletions(-) v2.7.6 -> v2.7.7: 2 files changed, 52 insertions(+), 35 deletions(-) v2.7.7 -> v2.7.8: 2 files changed, 83 insertions(+), 2 deletions(-) v2.7.8 -> v2.7.9: 8 files changed, 413 insertions(+), 418 deletions(-) v2.7.9 -> v2.7.10: 3 files changed, 8 insertions(+), 6 deletions(-) v2.7.10 -> v2.7.11: 3 files changed, 13 insertions(+), 13 deletions(-) v2.7.11 -> v2.7.12: 12 files changed, 32 insertions(+), 25 deletions(-) v2.7.12 -> v2.7.13: 5 files changed, 22 insertions(+), 16 deletions(-) v3.2 -> v3.2.1: 5 files changed, 61 insertions(+), 39 deletions(-) v3.2.1 -> v3.2.2: 4 files changed, 44 insertions(+), 18 deletions(-) v3.2.2 -> v3.2.3: 7 files changed, 152 insertions(+), 120 deletions(-) v3.2.3 -> v3.2.4: 12 files changed, 195 insertions(+), 105 deletions(-) v3.2.4 -> v3.2.5: v3.2.5 -> v3.2.6: v3.3.0 -> v3.3.1: 12 files changed, 165 insertions(+), 100 deletions(-) v3.3.1 -> v3.3.2: 2 files changed, 8 insertions(+), 18 deletions(-) v3.3.2 -> v3.3.3: 8 files changed, 371 insertions(+), 445 deletions(-) v3.3.3 -> v3.3.4: 3 files changed, 54 insertions(+), 37 deletions(-) v3.3.4 -> v3.3.5: 2 files changed, 5 insertions(+), 6 deletions(-) v3.3.5 -> v3.3.6: v3.4.0 -> v3.4.1: 3 files changed, 14 insertions(+), 15 deletions(-) v3.4.1 -> v3.4.2: 6 files changed, 209 insertions(+), 113 deletions(-) v3.4.2 -> v3.4.3: 7 files changed, 24 insertions(+), 17 deletions(-) v3.4.3 -> v3.4.4: 10 files changed, 253 insertions(+), 38 deletions(-) v3.4.4 -> v3.4.5: v3.4.5 -> v3.4.6: v3.5.0 -> v3.5.1: 5 files changed, 15 insertions(+), 15 deletions(-) v3.5.1 -> v3.5.2: 13 files changed, 59 insertions(+), 47 deletions(-) v3.5.2 -> v3.5.3: 7 files changed, 62 insertions(+), 94 deletions(-) From brett at python.org Sat Feb 25 13:19:45 2017 From: brett at python.org (Brett Cannon) Date: Sat, 25 Feb 2017 18:19:45 +0000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> Message-ID: On Fri, 24 Feb 2017 at 03:50 INADA Naoki wrote: > > Where should these translated docs live and how does one make it clear to > > users reading them that doc bugs shouldn't be submitted to the main bug > > tracker/github repo? > > > > We setup github page. See https://python-doc-ja.github.io/py36/ (note > that > version switcher won't work because this html is build for docs.python.jp > .) > > I reserved "python-docs" organization. So we can use URL like > https://python-doc.github.io//3.6/ > (Because the organization name looks "something official", I won't use it > until we get consensus about it.) > > For issue tracker, bugs.html must mention about translated document, before > mention to normal issue tracker. > It's getting a little hard to tease out what exactly is being asked at this point. Perhaps it's time to move the discussion over to a translation SIG (which probably needs to be created unless the old https://mail.python.org/mailman/listinfo/i18n-sig makes sense)? That way active translators can figure out exactly what they want to ask of python-dev in terms of support and we can have a more focused discussion. -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Sat Feb 25 14:08:47 2017 From: nad at python.org (Ned Deily) Date: Sat, 25 Feb 2017 14:08:47 -0500 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> Message-ID: <41A68EC6-6987-41F6-BB31-D1D8F1DCB4BD@python.org> On Feb 25, 2017, at 13:19, Brett Cannon wrote: > It's getting a little hard to tease out what exactly is being asked at this point. Perhaps it's time to move the discussion over to a translation SIG (which probably needs to be created unless the old https://mail.python.org/mailman/listinfo/i18n-sig makes sense)? That way active translators can figure out exactly what they want to ask of python-dev in terms of support and we can have a more focused discussion. I agree. We would need a more concrete and detailed proposal to really make any thoughtful decision. That may also include getting approval from the PSF Board as they are ultimately responsible for the contents of python.org. Without more detail, I don't have an opinion myself other than to note that, if we do something, it needs to be careful to not complicate or restrict the Python release process. -- Ned Deily nad at python.org -- [] From songofacandy at gmail.com Sat Feb 25 14:57:45 2017 From: songofacandy at gmail.com (INADA Naoki) Date: Sun, 26 Feb 2017 04:57:45 +0900 Subject: [Python-Dev] Translated Python documentation In-Reply-To: <41A68EC6-6987-41F6-BB31-D1D8F1DCB4BD@python.org> References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> <41A68EC6-6987-41F6-BB31-D1D8F1DCB4BD@python.org> Message-ID: Yes, right place to discussion is one of important things what I want. I haven't know about i18n-sig. Is it better than docs-sig? Another thing I want is agreement to use project name looks like (semi)official project. We used "python-doc-jp" name on Transifex at first. But since some people other than Japanese ask me to allow other languages, I renamed it to "python-doc". And I reserved "python-docs" organization at Github. While it's better name for working together with other Language translators, I don't like that unofficial project use such name. Hosting at docs.python.org is desirable too, but it can be discussed later. Regards, On Sun, Feb 26, 2017 at 4:08 AM, Ned Deily wrote: > On Feb 25, 2017, at 13:19, Brett Cannon wrote: >> It's getting a little hard to tease out what exactly is being asked at this point. Perhaps it's time to move the discussion over to a translation SIG (which probably needs to be created unless the old https://mail.python.org/mailman/listinfo/i18n-sig makes sense)? That way active translators can figure out exactly what they want to ask of python-dev in terms of support and we can have a more focused discussion. > > I agree. We would need a more concrete and detailed proposal to really make any thoughtful decision. That may also include getting approval from the PSF Board as they are ultimately responsible for the contents of python.org. Without more detail, I don't have an opinion myself other than to note that, if we do something, it needs to be careful to not complicate or restrict the Python release process. > > -- > Ned Deily > nad at python.org -- [] > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com From barry at python.org Sat Feb 25 17:40:18 2017 From: barry at python.org (Barry Warsaw) Date: Sat, 25 Feb 2017 17:40:18 -0500 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: <2eb15163-ecab-1d12-768e-1012bbab8211@hastings.org> References: <09c0a605-41a9-b045-82e6-0b2be170f928@trueblade.com> <2eb15163-ecab-1d12-768e-1012bbab8211@hastings.org> Message-ID: <20170225174018.629294f6@subdivisions.wooz.org> On Feb 24, 2017, at 11:40 PM, Larry Hastings wrote: >I'm for it too. Fixing up Misc/NEWS is probably the trickiest and most >error-prone part of cutting a release. Not to mention cherry picking for older releases. >While I'm sure we can learn a lot by examining "reno", my suspicion is we'll >want our own custom tool. I don't think the hard part is writing the >tool--the hard part is figuring out what we want and how it should be >integrated into our workflow. Actually writing such a tool is easy and fun, >and I'm sure our community would fall all over themselves to write it if >asked. In fact, I'll kind of pre-volunteer to write it right now, as long as >I can write it in Python 3 ;-) What else would you write it in?! :) Let's start with a simple question: do we like the general layout of the existing Misc/NEWS file? I do, and I *really* like it containing issue numbers, because that makes it relatively easy to search to see if a particular bug was fixed in a particular version (which is also why I wish all pull requests were tied to a bug). Other things I like about the file: * It's easy to search for a particular Python release, just by searching for "What's New". * I like having a release date easily visible with the Python version heading. * I like the bullet list format of the file, and I like the alphabetized category headings. What I don't like: * Lines that are too long ;) * All the inherent conflicts in editing a single file. * Misc/NEWS (on master) doesn't go back farther than 2 Python versions and 2 years, so it's harder to search for earlier changes. * It's in Misc/ not Doc/ Can we agree then that Misc/NEWS is the target output format for this tool? Maybe it's not the *only* target output format? Now, what's the input? The other thing is that I'm not sure design-by-committee is going to work so well here. There are several competing interests here. Release managers, non-core devs (drive-by contributors needing sponsorship), core devs (back porting/cherry picking changes), downstream consumers. Cheers, -Barry From brett at python.org Sat Feb 25 14:10:14 2017 From: brett at python.org (Brett Cannon) Date: Sat, 25 Feb 2017 19:10:14 +0000 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: <9868CB93-F573-4CA9-9C17-7CA1883891CF@python.org> References: <9868CB93-F573-4CA9-9C17-7CA1883891CF@python.org> Message-ID: On Wed, 22 Feb 2017 at 13:30 Ned Deily wrote: > On Feb 22, 2017, at 07:27, Victor Stinner > wrote: > > > [...]My idea is not new, it's already discussed in the Python > core-workflow: > > https://github.com/python/core-workflow/issues/6 > > > > I'm opening a thread on python-dev because Misc/NEWS quickly became a > > blocker issue with the Python new workflow. > > > > What do you think of the overall idea? > > I think we clearly need to do something about Misc/NEWS but this issue is > already under discussion in the core-workflow working group and I think we > shouldn't be moving it to python-dev until there is a final concrete > proposal. I know it is high on Brett's list but I believe he's taking a > few days of holiday right so he may not be able to respond immediately. > Let's move this discussion over to the core-workflow mailing list. It can > wait a few days. > > https://mail.python.org/mailman/listinfo/core-workflow Ned is right that I was on holiday for the past week and am slowly digging myself out from underneath my pile of email. Ned is also right that I view the Misc/NEWS issue the most important thing to solve next for our workflow as we have been putting up with merge conflicts on this file for long enough. Since this is going to be a somewhat protracted discussion as there are multiple tools consider (including writing our own), I think we should keep the discussion at https://github.com/python/core-workflow/issues/6. -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Sat Feb 25 22:18:16 2017 From: wes.turner at gmail.com (Wes Turner) Date: Sat, 25 Feb 2017 21:18:16 -0600 Subject: [Python-Dev] Split Misc/NEWS into individual files In-Reply-To: References: <9868CB93-F573-4CA9-9C17-7CA1883891CF@python.org> Message-ID: tools, projects, changelog filenames x-posting from https://github.com/python/core-workflow/issues/6#issuecomment-282529194 : ... changelog filenames - CHANGELOG.rst - HISTORY.rst - whatsnew.rst - Misc/NEWS ### ``Escaping M``arkup NOTE: commit logs may contain (executable) markup - **https://wiki.python.org/moin/EscapingHtml** - https://github.com/westurner/dotfiles/blob/develop/scripts/git-changelog.py - ``rst_escape()`` (!?) Tools - mergenews - Src: https://bitbucket.org/larry/mergenews - splitnews - mergenews - pyci - hg-specific - Reno - Src: https://github.com/openstack/reno - Src: https://github.com/openstack/reno/blob/master/reno/formatter.py - Src: https://github.com/openstack/reno/blob/master/reno/sphinxext.py - PyPI: https://pypi.org/project/reno - Docs: https://docs.openstack.org/developer/reno/usage.html#editing-a-release-note - [x] ENH: "make sections configurable" https://github.com/openstack/reno/commit/081a4145e18c82acba877ee22c180b3428c773f6 - Towncrier - Src: https://github.com/hawkowl/towncrier - PyPI: https://pypi.org/project/towncrier Projects: - CPython - https://github.com/python/cpython/blob/master/Misc/NEWS - IPython - https://github.com/ipython/ipython/tree/master/docs/source/whatsnew/pr#documenting-whats-new - https://github.com/ipython/ipython/tree/master/docs/source/whatsnew - Src: https://github.com/ipython/ipython/blob/master/tools/update_whatsnew.py - Matplotlib - https://github.com/matplotlib/matplotlib/blob/master/doc/users/whats_new.rst - https://github.com/matplotlib/matplotlib/tree/master/doc/users/prev_whats_new - https://github.com/matplotlib/matplotlib/tree/master/doc/users/whats_new README.rst - scikit-learn - ENH, BUG, DOC, MRG - https://github.com/scikit-learn/scikit-learn/blob/master/doc/whats_new.rst - statsmodels - ENH, BUG, DOC, MRG - https://github.com/statsmodels/statsmodels/tree/master/docs/source/release - OpenStack Nova - reno - https://git.openstack.org/cgit/openstack/nova/tree/releasenotes/notes - https://github.com/openstack/nova/tree/master/releasenotes/notes - http://lists.openstack.org/pipermail/release-announce/2017-February/000784.html On Sat, Feb 25, 2017 at 1:10 PM, Brett Cannon wrote: > > > On Wed, 22 Feb 2017 at 13:30 Ned Deily wrote: > >> On Feb 22, 2017, at 07:27, Victor Stinner >> wrote: >> >> > [...]My idea is not new, it's already discussed in the Python >> core-workflow: >> > https://github.com/python/core-workflow/issues/6 >> > >> > I'm opening a thread on python-dev because Misc/NEWS quickly became a >> > blocker issue with the Python new workflow. >> > >> > What do you think of the overall idea? >> >> I think we clearly need to do something about Misc/NEWS but this issue is >> already under discussion in the core-workflow working group and I think we >> shouldn't be moving it to python-dev until there is a final concrete >> proposal. I know it is high on Brett's list but I believe he's taking a >> few days of holiday right so he may not be able to respond immediately. >> Let's move this discussion over to the core-workflow mailing list. It can >> wait a few days. >> >> https://mail.python.org/mailman/listinfo/core-workflow > > > Ned is right that I was on holiday for the past week and am slowly digging > myself out from underneath my pile of email. > > Ned is also right that I view the Misc/NEWS issue the most important thing > to solve next for our workflow as we have been putting up with merge > conflicts on this file for long enough. > > Since this is going to be a somewhat protracted discussion as there are > multiple tools consider (including writing our own), I think we should keep > the discussion at https://github.com/python/core-workflow/issues/6. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > wes.turner%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Feb 25 23:22:28 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Feb 2017 14:22:28 +1000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> <41A68EC6-6987-41F6-BB31-D1D8F1DCB4BD@python.org> Message-ID: On 26 February 2017 at 05:57, INADA Naoki wrote: > Yes, right place to discussion is one of important things what I want. > I haven't know about i18n-sig. Is it better than docs-sig? > > > Another thing I want is agreement to use project name looks like > (semi)official project. > > We used "python-doc-jp" name on Transifex at first. But since some people > other than Japanese ask me to allow other languages, I renamed it to > "python-doc". > And I reserved "python-docs" organization at Github. > +1 from me for continuing to use those names (although you may want to standardise on "python-docs", since the mailing list is docs-sig, and the site is docs.python.org). While it's not translation related, I'll also note an idea that came up at the language summit a couple of years ago: moving the tutorial and the howto guides *out* of the version-specific documentation and into a separate version independent docs repository. As the stats Serhiy posted suggest, these typically *don't* change a great deal between versions, and when they do, the version differences are often going to be better handled inline (e.g. "On versions older than 3.x, ....") rather than by maintaining multiple distinct versions of the document. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Feb 26 01:00:56 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 26 Feb 2017 16:00:56 +1000 Subject: [Python-Dev] Handling PR branches in "make patchcheck" Message-ID: Hi folks, With the move to a PR based workflow, I've been finding "make patchcheck" less helpful than it used to be, as it was only checking the changes that had yet to be committed at all, and ignoring those that had already been checked in on the PR branch. I've put together a proposed update to Tools/scripts/patchcheck.py that makes it aware of git branches, and automatically diffs against the base branch when it detects something that "looks like" a PR branch: http://bugs.python.org/issue29656 I can vouch for it working in my particular setup (where "origin" refers to the upstream repo), but it would be good if folks could try it out with other configurations before we merge it (in particular, it *should* handle the case where the upstream remote is called "upstream", but I don't use that arrangement myself). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From xdegaye at gmail.com Sun Feb 26 05:19:36 2017 From: xdegaye at gmail.com (Xavier de Gaye) Date: Sun, 26 Feb 2017 11:19:36 +0100 Subject: [Python-Dev] attributes of Python code objects Message-ID: <7d2f8077-6f1f-a888-9631-2fa0c4e44fb0@gmail.com> In PR 218 [1], INADA Naoki wrote: > I think attributes of Python's code object is implementation detail, even > though PyPy follows. > But it's not big problem until there are some Python implementation > having different code implementation. Python's code object attributes are described in the Data Model [2] section of the documentation. 1) Is this description normative (to be followed by all the implementations of the Python language)? 2) Is the description of co_lnotab in [3] normative ? Xavier [1] https://github.com/python/cpython/pull/218 [2] https://docs.python.org/3/reference/datamodel.html [3] https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt From brett at python.org Sun Feb 26 15:44:16 2017 From: brett at python.org (Brett Cannon) Date: Sun, 26 Feb 2017 20:44:16 +0000 Subject: [Python-Dev] attributes of Python code objects In-Reply-To: <7d2f8077-6f1f-a888-9631-2fa0c4e44fb0@gmail.com> References: <7d2f8077-6f1f-a888-9631-2fa0c4e44fb0@gmail.com> Message-ID: On Sun, 26 Feb 2017 at 02:20 Xavier de Gaye wrote: > In PR 218 [1], INADA Naoki wrote: > > I think attributes of Python's code object is implementation detail, > even > > though PyPy follows. > > But it's not big problem until there are some Python implementation > > having different code implementation. > > Python's code object attributes are described in the Data Model [2] > section of > the documentation. 1) Is this description normative (to be followed by > all the > implementations of the Python language)? 2) Is the description of > co_lnotab > in [3] normative ? > That's a good question. "Code objects represent byte-compiled executable Python code, or bytecode" which would suggest that if an implementation didn't have a compiled code then code objects wouldn't make sense. Then again the object contains details about the function that can be useful and are relied upon by parts of the stdlib (e.g. inspect.signature() can't function without the code object). Due to that last point I would say the attributes on code objects are not an implementation detail, but it's acceptable for things like co_code to be set to None when it doesn't make sense. -Brett > > Xavier > > [1] https://github.com/python/cpython/pull/218 > [2] https://docs.python.org/3/reference/datamodel.html > [3] https://github.com/python/cpython/blob/master/Objects/lnotab_notes.txt > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Feb 26 15:32:28 2017 From: brett at python.org (Brett Cannon) Date: Sun, 26 Feb 2017 20:32:28 +0000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> <41A68EC6-6987-41F6-BB31-D1D8F1DCB4BD@python.org> Message-ID: On Sat, 25 Feb 2017 at 11:59 INADA Naoki wrote: > Yes, right place to discussion is one of important things what I want. > I haven't know about i18n-sig. Is it better than docs-sig? > Probably not (I honestly forgot about docs-sig). > > > Another thing I want is agreement to use project name looks like > (semi)official project. > > We used "python-doc-jp" name on Transifex at first. But since some people > other than Japanese ask me to allow other languages, I renamed it to > "python-doc". > As Nick suggested, "python-docs" is probably a better name to be consistent. -Brett > And I reserved "python-docs" organization at Github. > > While it's better name for working together with other Language > translators, > I don't like that unofficial project use such name. > > > Hosting at docs.python.org is desirable too, but it can be discussed > later. > > Regards, > > On Sun, Feb 26, 2017 at 4:08 AM, Ned Deily wrote: > > On Feb 25, 2017, at 13:19, Brett Cannon wrote: > >> It's getting a little hard to tease out what exactly is being asked at > this point. Perhaps it's time to move the discussion over to a translation > SIG (which probably needs to be created unless the old > https://mail.python.org/mailman/listinfo/i18n-sig makes sense)? That way > active translators can figure out exactly what they want to ask of > python-dev in terms of support and we can have a more focused discussion. > > > > I agree. We would need a more concrete and detailed proposal to really > make any thoughtful decision. That may also include getting approval from > the PSF Board as they are ultimately responsible for the contents of > python.org. Without more detail, I don't have an opinion myself other > than to note that, if we do something, it needs to be careful to not > complicate or restrict the Python release process. > > > > -- > > Ned Deily > > nad at python.org -- [] > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Sun Feb 26 17:19:00 2017 From: nad at python.org (Ned Deily) Date: Sun, 26 Feb 2017 17:19:00 -0500 Subject: [Python-Dev] IMPORTANT: Python 3.6.1 RC delayed 4 days, now 2017-03-03 References: <9DEC04ED-5B6E-45CE-AC83-4A9CD8EBF829@python.org> Message-ID: I had previously announced tomorrow as the date for the release candidate of our first 3.6 maintenance release. But, with the switchover to the new GitHub-based development process, there are a number of changes needed behind the scenes to our release production process and I think the release team (read "Ned") needs a few more days to get ready to produce the release. So, reluctantly, I'm delaying the cutoff and production of 3.6.1rc1 for 4 days, that is, until Friday 2017-03-03 UTC. As a plus, it gives us all a few more days to tackle the release-blocker, deferred-blocker, and critical issues still open against 3.6 and also to keep working though the new development process. For now, I'm not changing the date for 3.6.1 final (2017-03-13) until we see how rc1 goes. My apologies for the delay! --Ned Begin forwarded message: > From: Ned Deily > Subject: IMPORTANT: Python 3.6.1 Maintenance Release Release Candidate in 7 days (2017-02-27) > Date: February 20, 2017 at 15:12:47 EST > To: python-committers > Cc: Python Dev > > It seems like last year already since the release of 3.6.0. I guess that's because it was last year, 2016-12-22 to be exact! Now we're approaching the end of the first quarter and, according to PEP 494, it's time to start producing the first maintenance release for the 3.6 series. The schedule calls for the release candidate to be produced on Monday 2017-02-27 UTC. As was the case with the 3.6.0 release cycle, the plan is for the release candidate to be the same as the final release, that is, no additional changes go in after the release candidate except for any showstopper critical problems that might be discovered with rc1. So please plan to get any security fixes, bug fixes, and documentation changes you think should be in 3.6.1 merged in before 2017-02-27. I will send out another reminder a couple of days before. The 3.6.1 final is planned for two weeks following rc1, that is, on 2017-03-13. I expect the next 3.6 maintenance release (3.6.2) will follow about 3 months later, so most likely in 2017-06 after PyCon US. > > 3.6.1 will be the first release using our new GitHub-based development process (thanks, Brett and team!). If you are planning to push something for 3.6.1 and haven't yet tried out the new workflow or are not yet familiar with GitHub pull requests, you should probably give yourself some extra time. As always, the Developer's Guide is the primary reference for the development workflow; not surprisingly, with such a major change, there are likely still some parts of the guide that could use further changes and clarifications. You can help by reviewing the devguide's open issues and pull requests in its repository and adding to them as you work through issues. If you have comments on or improvement suggestions for the new workflow, the place to discuss them is on the core-workflow mailing list. > > Thanks again for all of your efforts in bringing 3.6.0 into the world and for helping now to make it even better! > > https://www.python.org/dev/peps/pep-0494/ > http://cpython-devguide.readthedocs.io > https://mail.python.org/mailman/listinfo/core-workflow -- Ned Deily nad at python.org -- [] From rob.cliffe at btinternet.com Sun Feb 26 21:18:27 2017 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 27 Feb 2017 02:18:27 +0000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On 24/02/2017 12:20, Berker Peksa? wrote: > On Thu, Feb 23, 2017 at 12:01 AM, Victor Stinner > wrote: >> 2017-02-22 19:04 GMT+01:00 Serhiy Storchaka : >>> What percent of lines is changed between bugfix releases? Feature releases? >>> >>> My largest apprehension is that the documentation can be outdated and >>> quickly become degraded if main contributors left it. >> If the original text (english) changes, untranslated text is >> displayed, not outdated text. > I think that's much worse than showing the outdated text. I don't see > any point on showing half English and half French text if the reader > can't understand the other half of it. > Fair point (especially if you substitute a language totally unrelated to English (e.g. Turkish, Finnish, Urdu, Japanese) for "French"). But it can be turned around: Popular demand would encourage whoever is maintaining the translated versions to catch up within a reasonable time when changes are made. It would nudge non-English readers to look at the untranslated English text and begin/develop their knowledge of English. Best wishes, Rob Cliffe From mertz at gnosis.cx Sun Feb 26 23:03:02 2017 From: mertz at gnosis.cx (David Mertz) Date: Sun, 26 Feb 2017 20:03:02 -0800 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: Could we have side-by-side English and whatever translated language? Then also use some sort of typographic indicator like color to show when the translation is out of date? On Feb 26, 2017 6:39 PM, "Rob Cliffe" wrote: > > > On 24/02/2017 12:20, Berker Peksa? wrote: > >> On Thu, Feb 23, 2017 at 12:01 AM, Victor Stinner >> wrote: >> >>> 2017-02-22 19:04 GMT+01:00 Serhiy Storchaka : >>> >>>> What percent of lines is changed between bugfix releases? Feature >>>> releases? >>>> >>>> My largest apprehension is that the documentation can be outdated and >>>> quickly become degraded if main contributors left it. >>>> >>> If the original text (english) changes, untranslated text is >>> displayed, not outdated text. >>> >> I think that's much worse than showing the outdated text. I don't see >> any point on showing half English and half French text if the reader >> can't understand the other half of it. >> >> Fair point (especially if you substitute a language totally unrelated to > English (e.g. Turkish, Finnish, Urdu, Japanese) for "French"). But it can > be turned around: > Popular demand would encourage whoever is maintaining the translated > versions to catch up within a reasonable time when changes are made. > It would nudge non-English readers to look at the untranslated English > text and begin/develop their knowledge of English. > Best wishes, > Rob Cliffe > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mertz% > 40gnosis.cx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 27 02:24:58 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Feb 2017 17:24:58 +1000 Subject: [Python-Dev] attributes of Python code objects In-Reply-To: References: <7d2f8077-6f1f-a888-9631-2fa0c4e44fb0@gmail.com> Message-ID: On 27 February 2017 at 06:44, Brett Cannon wrote: > > > On Sun, 26 Feb 2017 at 02:20 Xavier de Gaye wrote: > >> In PR 218 [1], INADA Naoki wrote: >> > I think attributes of Python's code object is implementation detail, >> even >> > though PyPy follows. >> > But it's not big problem until there are some Python implementation >> > having different code implementation. >> >> Python's code object attributes are described in the Data Model [2] >> section of >> the documentation. 1) Is this description normative (to be followed by >> all the >> implementations of the Python language)? 2) Is the description of >> co_lnotab >> in [3] normative ? >> > > That's a good question. "Code objects represent byte-compiled executable > Python code, or bytecode" which would suggest that if an implementation > didn't have a compiled code then code objects wouldn't make sense. Then > again the object contains details about the function that can be useful and > are relied upon by parts of the stdlib (e.g. inspect.signature() can't > function without the code object). > > Due to that last point I would say the attributes on code objects are not > an implementation detail, but it's acceptable for things like co_code to be > set to None when it doesn't make sense. > Right, we should probably try to be more explicit about this, as there are certainly aspects of the code object structure that other implementations will need to emulate if they want to re-use modules like inspect without modification. However, there are also aspects that we explicitly call out as implementation details, like the internal structure of co_code and co_lnotab, where they may change arbitrarily between CPython feature releases. The generator and coroutine ABCs were also specifically added so they could be replaced with objects that don't have __code__.co_flags attributes (e.g. by Cython) without breaking asyncio. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Feb 27 02:30:36 2017 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 27 Feb 2017 17:30:36 +1000 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On 27 February 2017 at 14:03, David Mertz wrote: > Could we have side-by-side English and whatever translated language? Then > also use some sort of typographic indicator like color to show when the > translation is out of date? > This kind of interface is what services like Transifex and Zanata offer translators (they also have things like phrase dictionaries, showing how particular terms have been translated elsewhere in the project). For the actual documentation, showing partial translations is the standard practice, as the assumption is that many readers will have *some* ability to read English, they just prefer to read their native language. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Mon Feb 27 03:20:03 2017 From: mertz at gnosis.cx (David Mertz) Date: Mon, 27 Feb 2017 00:20:03 -0800 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: On Sun, Feb 26, 2017 at 11:30 PM, Nick Coghlan wrote: > On 27 February 2017 at 14:03, David Mertz wrote: > >> Could we have side-by-side English and whatever translated language? Then >> also use some sort of typographic indicator like color to show when the >> translation is out of date? >> > > This kind of interface is what services like Transifex and Zanata offer > translators (they also have things like phrase dictionaries, showing how > particular terms have been translated elsewhere in the project). > > For the actual documentation, showing partial translations is the standard > practice, as the assumption is that many readers will have *some* ability > to read English, they just prefer to read their native language. > I think it would be at least as useful for readers as for the translators. As you mention, many readers will have *some* English. If they can look from the left half to the right half of the screen in synchronized texts (or perhaps top/bottom; whatever), they can read the English as well as they are able while simultaneously reading as much of their preferred language as is available. If their preferred language is available but possibly not current, they could decide whether to try to understand the difference in the canonical English version. I really liked this in books I've read. There are a fair number of languages other than English where I can make a little bit of sense of the text (but sadly only the one in which I'm proficient). Nonetheless, I like looking at the original text next to the English that I actually understand fully. Admittedly this is especially nice for something like poetry where you can read for meter on one side then content on the other... that's not the same concern as technical documentation, I realize. But even if only as an option, I think it would be a valuable interface for many readers. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Feb 27 05:31:57 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 27 Feb 2017 11:31:57 +0100 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> Message-ID: 2017-02-25 19:19 GMT+01:00 Brett Cannon : > It's getting a little hard to tease out what exactly is being asked at this > point. Perhaps it's time to move the discussion over to a translation SIG > (which probably needs to be created unless the old > https://mail.python.org/mailman/listinfo/i18n-sig makes sense)? That way > active translators can figure out exactly what they want to ask of > python-dev in terms of support and we can have a more focused discussion. Things are already happening in the background on other lists and other Python projects, but the problem is that the translation project seems "blocked" for some reasons. That's why I started the thread. Example of a recent CPython PR, blocked: https://github.com/python/cpython/pull/195 "bpo-28331: fix "CPython implementation detail:" label is removed when content is translated." opened 7 days ago by INADA Naoki (JP translation) Example of a docsbuild PR: https://github.com/python/docsbuild-scripts/pull/8 "[WIP] Add french, japanese, and chinese", opened at 12 Dec 2016 by Julien Palard (FR translation) See also Julien Palard's threads on python-ideas: no decision was taken, so the project is blocked. According to this thread, there is an official GO for official translations, so these PR should be merged, right? Victor From victor.stinner at gmail.com Mon Feb 27 05:58:34 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 27 Feb 2017 11:58:34 +0100 Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> Message-ID: 2017-02-24 13:20 GMT+01:00 Berker Peksa? : >> If the original text (english) changes, untranslated text is >> displayed, not outdated text. > > I think that's much worse than showing the outdated text. I don't see > any point on showing half English and half French text if the reader > can't understand the other half of it. Sorry, it doesn't make sense to me. If I wouldn't be to read english at all, and I have the choice between a doc partially translated and a doc written fully in english (current docs.python.org), the obvious choice for me would be to pick the partially translated doc. It's better than nothing. No? Victor From victor.stinner at gmail.com Mon Feb 27 06:04:21 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 27 Feb 2017 12:04:21 +0100 Subject: [Python-Dev] Translated Python documentation In-Reply-To: <20170224151002.GG5689@ando.pearwood.info> References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> <20170224151002.GG5689@ando.pearwood.info> Message-ID: 2017-02-24 16:10 GMT+01:00 Steven D'Aprano : >> ?And then you need another one to >> check what was written. These are practical problems. There are >> extant services to support this, they are expensive in either money or >> time, and the docs produced usually lag behind English quite a bit. > > Is this a good use for some PSF funding? Would companies be willing to > invest money in translating Python documentation? > > Just because we're Open Source, doesn't mean that everything we do has > to be purely volunteer. IHMO translating the *whole* Python documentation at once by a professional translator can be very expensive, no somthing that the PSF would affort. Which language would you pick? Depending on what? We already have motivated translators for free who only ask us for the permission to make tiny changes to make their life simpler and make the doc more visible. I'm in favor of allowing them to translate and make the translated doc official ;-) IMHO a better usage of the PSF funding would be to organize some local sprints to translate the Python documentation. Such sprints are fun, cheap, and can be a nice opportunity to recruit free and motivated translators. We are looking for people involved to translate the doc the doc is updated, not only translate the doc once and go away. Right? Victor From alexander.belopolsky at gmail.com Mon Feb 27 19:25:22 2017 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 27 Feb 2017 19:25:22 -0500 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken Message-ID: I have opened an issue on b.p.o., [1] but I wonder whether Misc/gdbinit is still supported in 3.6. [1]: http://bugs.python.org/issue29673 -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Mon Feb 27 19:34:37 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 28 Feb 2017 01:34:37 +0100 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: I suggest to remove them. I didn't use them since python-gdb.py exist. Does someone still need gdbinit macros for gdb without python binding? Victor Le 28 f?vr. 2017 1:27 AM, "Alexander Belopolsky" < alexander.belopolsky at gmail.com> a ?crit : > I have opened an issue on b.p.o., [1] but I wonder whether Misc/gdbinit is > still supported in 3.6. > > [1]: http://bugs.python.org/issue29673 > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > victor.stinner%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Mon Feb 27 19:37:57 2017 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Mon, 27 Feb 2017 19:37:57 -0500 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: On Mon, Feb 27, 2017 at 7:34 PM, Victor Stinner wrote: > Does someone still need gdbinit macros for gdb without python binding? > I find them useful. I've never had success with python-gdb.py. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Mon Feb 27 21:00:52 2017 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Mon, 27 Feb 2017 20:00:52 -0600 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: Alexander> I find them useful. I've never had success with python-gdb.py. As the original author, and occasional user (just in the last week or two) I still find the current crude hack useful. I tried to get the Python support in GDB working a couple years ago, but gave up in frustration. I hope it's better now. I added a comment to Alexander's bug report. Skip From victor.stinner at gmail.com Tue Feb 28 03:33:18 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 28 Feb 2017 09:33:18 +0100 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: 2017-02-28 3:00 GMT+01:00 Skip Montanaro : > Alexander> I find them useful. I've never had success with python-gdb.py. Alexander, Skip: Oh, which kind of issues do you have with python-gdb.py? It doesn't work? You are unable to dump some data? python-gdb.py doesn't run code to analyze memory, it prevents crashes when analysing a running process, and it works on core dumps. It has many features, it works well. I never had major issues with it, and I enhance it regulary. Recently, I added support for "method-wrapper". Victor From skip.montanaro at gmail.com Tue Feb 28 10:09:49 2017 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 28 Feb 2017 09:09:49 -0600 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: I haven't tried using Python support in gdb in a long time. I don't know what python-gdb.py is. How is that different than Tools/gdb/libpython.py? Is it distributed with Python? (I'm kind of blind at work without a source distribution I can grep through, and didn't see it in Tools/gdb or Misc.) I'm a simple man. I need the kind of stuff Misc/gdbinit provides (smart stack traces and variable printing), but I most frequently use the pyo user-defined command to print the contents of PyObject pointers. When they are working () pystack and pyframe are also useful. Skip On Tue, Feb 28, 2017 at 2:33 AM, Victor Stinner wrote: > 2017-02-28 3:00 GMT+01:00 Skip Montanaro : >> Alexander> I find them useful. I've never had success with python-gdb.py. > > Alexander, Skip: Oh, which kind of issues do you have with > python-gdb.py? It doesn't work? You are unable to dump some data? > > python-gdb.py doesn't run code to analyze memory, it prevents crashes > when analysing a running process, and it works on core dumps. It has > many features, it works well. I never had major issues with it, and I > enhance it regulary. Recently, I added support for "method-wrapper". > > Victor From barry at python.org Tue Feb 28 10:39:59 2017 From: barry at python.org (Barry Warsaw) Date: Tue, 28 Feb 2017 10:39:59 -0500 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: <20170228103959.66db69e7@subdivisions.wooz.org> On Feb 28, 2017, at 09:09 AM, Skip Montanaro wrote: >I haven't tried using Python support in gdb in a long time. I don't >know what python-gdb.py is. How is that different than >Tools/gdb/libpython.py? Is it distributed with Python? (I'm kind of >blind at work without a source distribution I can grep through, and >didn't see it in Tools/gdb or Misc.) > >I'm a simple man. I need the kind of stuff Misc/gdbinit provides >(smart stack traces and variable printing), but I most frequently use >the pyo user-defined command to print the contents of PyObject >pointers. When they are working () pystack and pyframe are also >useful. I think I'm largely in the same boat as Skip although it's been a while since I stepped through the C code. I also had trouble (ages ago) getting the gdb integration working, so I relied on the simple stuff in gdbinit. I think there's still utility in that if we can keep it working. -Barry From angwerzx at 126.com Tue Feb 28 05:00:07 2017 From: angwerzx at 126.com (Xiang Zhang) Date: Tue, 28 Feb 2017 18:00:07 +0800 (CST) Subject: [Python-Dev] Translated Python documentation In-Reply-To: References: <20170222164058.7b0333ef@fsol> <1b5001d28e8d$6c1e6570$445b3050$@hotmail.com> <20170224151002.GG5689@ando.pearwood.info> Message-ID: <308218bd.a0a4.15a8429aaa6.Coremail.angwerzx@126.com> I support having an official translated doc. I have seen several groups trying to translate part of our official doc. But their efforts are disperse and quickly become lost because they are not organized to work towards a single common result and their results are hold anywhere on the Web and hard to find. An official one could help ease the pain. But I agree we may need more details on the workflow. At 2017-02-27 19:04:21, "Victor Stinner" wrote: >2017-02-24 16:10 GMT+01:00 Steven D'Aprano : >>> ?And then you need another one to >>> check what was written. These are practical problems. There are >>> extant services to support this, they are expensive in either money or >>> time, and the docs produced usually lag behind English quite a bit. >> >> Is this a good use for some PSF funding? Would companies be willing to >> invest money in translating Python documentation? >> >> Just because we're Open Source, doesn't mean that everything we do has >> to be purely volunteer. > >IHMO translating the *whole* Python documentation at once by a >professional translator can be very expensive, no somthing that the >PSF would affort. Which language would you pick? Depending on what? > >We already have motivated translators for free who only ask us for the >permission to make tiny changes to make their life simpler and make >the doc more visible. I'm in favor of allowing them to translate and >make the translated doc official ;-) > >IMHO a better usage of the PSF funding would be to organize some local >sprints to translate the Python documentation. Such sprints are fun, >cheap, and can be a nice opportunity to recruit free and motivated >translators. We are looking for people involved to translate the doc >the doc is updated, not only translate the doc once and go away. >Right? > >Victor > -------------- next part -------------- An HTML attachment was scrubbed... URL: From edouard at bluefarmenvironment.com Tue Feb 28 09:56:35 2017 From: edouard at bluefarmenvironment.com (Edouard ROYER) Date: Tue, 28 Feb 2017 15:56:35 +0100 Subject: [Python-Dev] PyQt - DLL load failed Message-ID: Hi, I'm pretty new in Python but as I was looking for some new programmer's tool, it seemed to me that Python should be the right language for my needs. However, trying to run PyQt on my computer I have an issue that I didn't manage yet : trying to import objects from PyQt library leads to the following message : "ImportError : DLL load failed : Impossibile trovare la procedura specificata". Either with PyCharm or launching Jupyter QtConsole. I already read a lot of posts talking about similar issues nut none of them gave me a valid solution. Did i miss something ? Could it be a version problem (even if I'm pretty sure to have install in 32 bits compatibility) ? Many thanks. Edouard -- Edouard ROYER BLUEFARM s.r.l. Via delle Industrie 15 30175 Venezia Marghera (IT) Tel. +39 327 1989840 --- Questa e-mail ? stata controllata per individuare virus con Avast antivirus. https://www.avast.com/antivirus From b.dunphy.342 at gmail.com Tue Feb 28 13:25:24 2017 From: b.dunphy.342 at gmail.com (Bryan Dunphy) Date: Tue, 28 Feb 2017 12:25:24 -0600 Subject: [Python-Dev] Python 2.7.13 unmodified will not build from source on Mac OS El Capitan (10.11) Message-ID: <326AA2BF-261F-4BFD-91DD-4D071149362A@gmail.com> I have Xcode installed and other large projects build just fine. I ran ?./configure && make? in the ?Python-2.7.13? directory and ?make? generated two errors both referring to the same thing missing. The exact error messages are copied and pasted below. dyld: lazy symbol binding failed: Symbol not found: _getentropy Referenced from: /Users/bryandunphy/Downloads/Python-2.7.13/./python.exe Expected in: /usr/lib/libSystem.B.dylib dyld: Symbol not found: _getentropy Referenced from: /Users/bryandunphy/Downloads/Python-2.7.13/./python.exe Expected in: /usr/lib/libSystem.B.dylib /bin/sh: line 1: 14105 Trace/BPT trap: 5 CC='gcc' LDSHARED='gcc -bundle -undefined dynamic_lookup -L/usr/local/lib' OPT='-DNDEBUG -g -fwrapv -O3 -Wall -Wstrict-prototypes' _TCLTK_INCLUDES='' _TCLTK_LIBS='' ./python.exe -E ./setup.py $quiet build make: *** [sharedmods] Error 133 That was the final output from make. Any ideas on how to fix these errors and allow a build? From alexander.belopolsky at gmail.com Tue Feb 28 14:38:32 2017 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Tue, 28 Feb 2017 14:38:32 -0500 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: On Tue, Feb 28, 2017 at 3:33 AM, Victor Stinner wrote: > > Alexander, Skip: Oh, which kind of issues do you have with > python-gdb.py? It doesn't work? You are unable to dump some data? First, I had to rename python-gdb.py to python3.6-gdb.py to make it load. Then running backtrace gave me a bunch of error messages: (gdb) bt #0 builtin_hex (module=, number=0) at Python/bltinmodule.c:1436 #1 0x00007ffff7a9304c in _PyCFunction_FastCallDict (func_obj=, args=0x662cf8, nargs=1, kwargs=0x0) at Objects/methodobject.c:209 #2 0x00007ffff7a9344b in _PyCFunction_FastCallKeywords (func=, stack=0x662cf8, nargs=1, kwnames=0x0) at Objects/methodobject.c:295 #3 0x00007ffff7bac6d9 in call_function (pp_stack=0x7fffffffb6d8, oparg=1, kwnames=0x0) at Python/ceval.c:4788 #4 0x00007ffff7ba3e6c in _PyEval_EvalFrameDefault (f=Frame 0x662b68, for file r.py, line 2, in (), throwflag=0) at Python/ceval.c:3275 #5 0x00007ffff7b8cc67 in PyEval_EvalFrameEx (f=Frame 0x662b68, for file r.py, line 2, in (), throwflag=0) at Python/ceval.c:718 #6 0x00007ffff7ba9745 in _PyEval_EvalCodeWithName (_co=, globals=Traceback (most recent call last): File "/home/a/.virtualenvs/3.6g/bin/python3.6-gdb.py", line 1358, in to_string return pyop.get_truncated_repr(MAX_OUTPUT_LEN) File "/home/a/.virtualenvs/3.6g/bin/python3.6-gdb.py", line 243, in get_truncated_repr self.write_repr(out, set()) File "/home/a/.virtualenvs/3.6g/bin/python3.6-gdb.py", line 702, in write_repr for pyop_key, pyop_value in self.iteritems(): File "/home/a/.virtualenvs/3.6g/bin/python3.6-gdb.py", line 669, in iteritems entries, nentries = self._get_entries(keys) File "/home/a/.virtualenvs/3.6g/bin/python3.6-gdb.py", line 717, in _get_entries except gdb.error: AttributeError: 'module' object has no attribute 'error' ... It looks like there is a mismatch between python-gdb.py and the gdb module that I assume comes with gdb. $ gdb --version GNU gdb (GDB) Red Hat Enterprise Linux (7.2-90.el6) .. -------------- next part -------------- An HTML attachment was scrubbed... URL: From skip.montanaro at gmail.com Tue Feb 28 15:19:13 2017 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 28 Feb 2017 14:19:13 -0600 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: > First, I had to rename python-gdb.py ... Okay, I found a copy of python-gdb.py at the top level in a 2.7.13 snapshot I have at-hand. I have no idea where it came from. A file search on GitHub in the python/cpython repo on both master and 2.7 branches yielded nothing. It does seem to be working for me on a debug build of Python 2.7.13 (I ran strings over my GDB executable first to see if it knew anything about Python, and it did). Thx, Skip From victor.stinner at gmail.com Tue Feb 28 15:40:47 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 28 Feb 2017 21:40:47 +0100 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: Ok, it seems like very few people know how to use python-gdb.py :-/ Sorry, I expect that everybody was using it! python-gdb.py was written by Dave Malcolm, it includes: * a pretty printer for many builtin Python types: str, tuple, list, dict, frame, etc. It means that a regular gdb "print obj" command displays the Python object content instead of a raw pointer. This feature is super useful, it avoids to call _PyObject_Dump(obj) which is likely to crash! * commands to navigate between Python frames: py-up, py-down * command to dump the Python traceback: py-bt, py-bt-full (also include C frames) * a few more commands For me, these commands are really amazing! It's impressive to be able to that in a debugger which doesn't know Python internals at all! It's cool to be able to "program" (extend) a debugger! I never tried to install it, I always use it with a ./python binary compiled in the source code tree: --- $ make $ gdb ./python # load ./python-gdb.py ... --- Note: ./python-gdb.py is simply a copy of Tools/gdb/libpython.py, copied by Makefile. On Fedora, gdb doesn't load python-gdb.py because it doesn't trust my $HOME/prog directory (root of all my development directories). I had to trust it using this ~/.gdbinit config: --- add-auto-load-safe-path ~/prog/ --- More generally, when gdb loads a "program", it tries to load "program-gdb.py" in the same directory. Maybe it can load "program-gdb.py" from other directories, but I never understood this part, and hopefully I never had to understand it :-D Maybe the debug package of Python on Debian and Fedora installs pythonX.Y-gdb.py in the right directory, I didn't check, but I always debug using a freshly compiled Python, so I never tried to understand how these things work. Example: ---- haypo at selma$ gdb ./python (gdb) b _PyEval_EvalFrameDefault (gdb) run Breakpoint 1, _PyEval_EvalFrameDefault ( f=Frame 0x7ffff7f22058, for file , line 25, in (), throwflag=0) at Python/ceval.c:678 678 PyObject *retval = NULL; /* Return value */ => gdb displays the content of the frame "f", you can see immediately the filename and the line number (well, this frame is a little bit special, it's the frozen module importlib) (gdb) py-bt Traceback (most recent call first): File "", line 25, in --- Example of Python traceback: ---- haypo at selma$ gdb -args ./python -m test -r (gdb) run ... ^C (gdb) py-bt Traceback (most recent call first): File "/home/haypo/prog/python/master/Lib/test/test_long.py", line 947, in test_bit_length self.assertEqual(k, len(bin(x).lstrip('-0b'))) File "/home/haypo/prog/python/master/Lib/unittest/case.py", line 601, in run testMethod() File "/home/haypo/prog/python/master/Lib/unittest/case.py", line 649, in __call__ return self.run(*args, **kwds) File "/home/haypo/prog/python/master/Lib/unittest/suite.py", line 122, in run test(result) File "/home/haypo/prog/python/master/Lib/unittest/suite.py", line 84, in __call__ return self.run(*args, **kwds) ... => you get filename, line number, function name and even the Python line! It seems obvious to get such information, but remind that you are in gdb, not in Python (gdb) py-list 942 def test_bit_length(self): 943 tiny = 1e-10 944 for x in range(-65000, 65000): 945 k = x.bit_length() 946 # Check equivalence with Python version >947 self.assertEqual(k, len(bin(x).lstrip('-0b'))) 948 # Behaviour as specified in the docs 949 if x != 0: 950 self.assertTrue(2**(k-1) <= abs(x) < 2**k) 951 else: 952 self.assertEqual(k, 0) # move to the parent Python frame (= skip mutiple C frames until the next Python frame) (gdb) py-up (...) (gdb) py-list 596 with outcome.testPartExecutor(self): 597 self.setUp() 598 if outcome.success: 599 outcome.expecting_failure = expecting_failure 600 with outcome.testPartExecutor(self, isTest=True): >601 testMethod() 602 outcome.expecting_failure = False 603 with outcome.testPartExecutor(self): 604 self.tearDown() 605 606 self.doCleanups() (gdb) py-locals self = expecting_failure_method = False expecting_failure_class = False expecting_failure = False ... => dump all variables of the current Python frame! -------- Victor From victor.stinner at gmail.com Tue Feb 28 15:46:20 2017 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 28 Feb 2017 21:46:20 +0100 Subject: [Python-Dev] GDB macros in Misc/gdbinit are broken In-Reply-To: References: Message-ID: 2017-02-28 20:38 GMT+01:00 Alexander Belopolsky : > First, I had to rename python-gdb.py to python3.6-gdb.py to make it load. There is nothing specific to Python here, the file command of gdb (or gdb file) tries to load "file-gdb.py". So if your program is called python3.6, gdb looks for python3.6-gdb.py, yeah. > Then running backtrace gave me a bunch of error messages: > (...) > except gdb.error: > AttributeError: 'module' object has no attribute 'error' Oh, I never tried python-gdb.py on RHEL. Sadly, I know that the Python binding of gdb changes depending on the gdb version, but I also suspect that it changes depending on the OS! I'm quite sure that Fedora backports gdb changes for its Python API. I'm only using python-gdb.py on Fedora, usually even the latest Fedora version. Fedora always has bleeding edge versions of development tools! It shouldn't be hard to catch this "except gdb.error:" line. We should find the right exception and patch libpython.py. > It looks like there is a mismatch between python-gdb.py and the gdb module > that I assume comes with gdb. python-gdb.py (libpython.py) should work on any gdb version. It shouldn't be hard to support multiple gdb versions in same code base, we already support Python 2 and Python 3 in the same code base! (Depending on the OS and OS version, gdb may use Python 2 or Python 3 to run python-gdb.py, this script is run by gdb, not by test debugged Python binary! ;-)) Victor From tjreedy at udel.edu Tue Feb 28 16:18:49 2017 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 28 Feb 2017 16:18:49 -0500 Subject: [Python-Dev] PyQt - DLL load failed In-Reply-To: References: Message-ID: On 2/28/2017 9:56 AM, Edouard ROYER wrote: > Hi, > > I'm pretty new in Python but as I was looking for some new programmer's > tool, it seemed to me that Python should be the right language for my > needs. > > However, trying to run PyQt on my computer I have an issue that I didn't > manage yet : trying to import objects from PyQt library leads to the > following message : "ImportError : DLL load failed : Impossibile > trovare la procedura specificata". Either with PyCharm or launching > Jupyter QtConsole. Pydev is for developing future versions of Python and CPython. Questions about using current versions should be directed to pythohn-list or other forums. -- Terry Jan Reedy