From nad at python.org Tue Nov 1 00:55:41 2016 From: nad at python.org (Ned Deily) Date: Tue, 1 Nov 2016 00:55:41 -0400 Subject: [Python-Dev] [RELEASE] Python 3.6.0b3 is now available Message-ID: On behalf of the Python development community and the Python 3.6 release team, I'm pleased to announce the availability of Python 3.6.0b3. 3.6.0b3 is the third of four planned beta releases of Python 3.6, the next major release of Python. Among the new major new features in Python 3.6 are: * PEP 468 - Preserving the order of **kwargs in a function * PEP 487 - Simpler customization of class creation * PEP 495 - Local Time Disambiguation * PEP 498 - Literal String Formatting * PEP 506 - Adding A Secrets Module To The Standard Library * PEP 509 - Add a private version to dict * PEP 515 - Underscores in Numeric Literals * PEP 519 - Adding a file system path protocol * PEP 520 - Preserving Class Attribute Definition Order * PEP 523 - Adding a frame evaluation API to CPython * PEP 524 - Make os.urandom() blocking on Linux (during system startup) * PEP 525 - Asynchronous Generators (provisional) * PEP 526 - Syntax for Variable Annotations (provisional) * PEP 528 - Change Windows console encoding to UTF-8 (provisional) * PEP 529 - Change Windows filesystem encoding to UTF-8 (provisional) * PEP 530 - Asynchronous Comprehensions Please see "What?s New In Python 3.6" for more information: https://docs.python.org/3.6/whatsnew/3.6.html You can find Python 3.6.0b3 here: https://www.python.org/downloads/release/python-360b3/ Beta releases are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We strongly encourage maintainers of third-party Python projects to test with 3.6 during the beta phase and report issues found to bugs.python.org as soon as possible. While the release is feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (2016-12-05). Our goal is have no changes after rc1. To achieve that, it will be extremely important to get as much exposure for 3.6 as possible during the beta phase. Please keep in mind that this is a preview release and its use is not recommended for production environments The next pre-release of Python 3.6 will be 3.6.0b4, currently scheduled for 2016-11-21. More information about the release schedule can be found here: https://www.python.org/dev/peps/pep-0494/ -- Ned Deily nad at python.org -- [] From skip.montanaro at gmail.com Tue Nov 1 08:26:21 2016 From: skip.montanaro at gmail.com (Skip Montanaro) Date: Tue, 1 Nov 2016 07:26:21 -0500 Subject: [Python-Dev] 2.6 branch in Mercurial? In-Reply-To: References: Message-ID: Cool, thanks to Ned and Zach. Hg never gets allocated very many neurons in my brain. Then there's the whole brain-in-neutral aspect of things which makes me fail to consider there might be help and/or closed branches which aren't displayed... Sorry for the distraction. Skip -------------- next part -------------- An HTML attachment was scrubbed... URL: From rv at rasmusvillemoes.dk Tue Nov 1 17:56:05 2016 From: rv at rasmusvillemoes.dk (Rasmus Villemoes) Date: Tue, 01 Nov 2016 22:56:05 +0100 Subject: [Python-Dev] Implementing (parts of) copy module in C Message-ID: <878tt2x3t6.fsf@rasmusvillemoes.dk> First, apologies if this isn't the appropriate list; I trust I'll be nudged in the right direction. I'm using (and contributing to) an application which spends a significant part of its startup time calling copy.deepcopy, so I thought I'd try to write a C extension to speed this up. Aside from avoiding the interpreter overhead, we should also save some time by having access to pre-sizing the containers. My plan is to let the C code handle the [at least in my case] most common stuff - atomic types, dicts, lists, tuples etc., while falling back to the python implementation for the stuff that's inconvenient to do in C, or which just hasn't been done yet - this allows a simple piecemeal approach. Since my main use case runs on 2.7 only, I've started from the 2.7 branch, and have some working code (passes the test suite, speedup between 3x-30x depending on the data structure). Before spending time forward-porting this to Python 3, I'd like to hear if there's any interest in it? I would of course also be very interested in getting it into 2.7.x, but I assume that's impossible(?). Where/how should RFC patches be posted? Thanks, Rasmus From gmludo at gmail.com Tue Nov 1 19:34:13 2016 From: gmludo at gmail.com (Ludovic Gasc) Date: Wed, 2 Nov 2016 00:34:13 +0100 Subject: [Python-Dev] Benchmarking Python and micro-optimizations In-Reply-To: References: Message-ID: Hi, Thanks first for that, it's very interesting. About to enrich benchmark suite, I might have a suggestion: We might add REST/JSON scenarios, because a lot of people use Python for that. It isn't certainly not the best REST/JSON scenarios, because they have a small payload, but better than nothing: https://www.techempower.com/benchmarks/#section=code&hw=peak&test=fortune Moreover, we already have several implementations for the most populars Web frameworks: https://github.com/TechEmpower/FrameworkBenchmarks/tree/master/frameworks/Python The drawback is that a lot of tests need a database. I can help if you're interested in. Have a nice week. -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Nov 2 02:23:22 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 2 Nov 2016 15:23:22 +0900 Subject: [Python-Dev] Implementing (parts of) copy module in C In-Reply-To: <878tt2x3t6.fsf@rasmusvillemoes.dk> References: <878tt2x3t6.fsf@rasmusvillemoes.dk> Message-ID: <22553.34394.464326.178260@turnbull.sk.tsukuba.ac.jp> Rasmus Villemoes writes: > First, apologies if this isn't the appropriate list; I trust I'll be > nudged in the right direction. Given the relatively advanced state of patch, I doubt that this is the *wrong* list. However, you would probably benefit from posting to python-list at python.org to collect use cases. (We don't "vote" on such additions. Rather the senior devs consider whether the use cases seem general enough to justify on-going maintenance costs for new code.) > I would of course also be very interested in getting it into 2.7.x, > but I assume that's impossible(?). That is correct. This is clearly a feature, and 2.7 currently is accepting only security-related patches (broadly construed -- a sufficiently severe bug, such as a crash or infloop, is security- related because it could be used to implement a DoS attack). > Where/how should RFC patches be posted? To the issue tracker, bugs.python.org. A follow-up here (and if you choose to post to python-list, there) to mention the issue number and URL is a good idea. From breamoreboy at yahoo.co.uk Wed Nov 2 03:54:48 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 2 Nov 2016 07:54:48 +0000 Subject: [Python-Dev] Implementing (parts of) copy module in C In-Reply-To: <22553.34394.464326.178260@turnbull.sk.tsukuba.ac.jp> References: <878tt2x3t6.fsf@rasmusvillemoes.dk> <22553.34394.464326.178260@turnbull.sk.tsukuba.ac.jp> Message-ID: On 02/11/2016 06:23, Stephen J. Turnbull wrote: > Rasmus Villemoes writes: > > > First, apologies if this isn't the appropriate list; I trust I'll be > > nudged in the right direction. > > Given the relatively advanced state of patch, I doubt that this is the > *wrong* list. However, you would probably benefit from posting to > python-list at python.org to collect use cases. (We don't "vote" on such > additions. Rather the senior devs consider whether the use cases seem > general enough to justify on-going maintenance costs for new code.) > > > I would of course also be very interested in getting it into 2.7.x, > > but I assume that's impossible(?). > > That is correct. This is clearly a feature, and 2.7 currently is > accepting only security-related patches (broadly construed -- a > sufficiently severe bug, such as a crash or infloop, is security- > related because it could be used to implement a DoS attack). > Surely patches related to any bugs, not just security related ones, will be accepted until EOL in 2020? -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From victor.stinner at gmail.com Wed Nov 2 06:54:59 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 2 Nov 2016 11:54:59 +0100 Subject: [Python-Dev] Implementing (parts of) copy module in C In-Reply-To: <878tt2x3t6.fsf@rasmusvillemoes.dk> References: <878tt2x3t6.fsf@rasmusvillemoes.dk> Message-ID: Hi, 2016-11-01 22:56 GMT+01:00 Rasmus Villemoes : > I'm using (and contributing to) an application which spends a > significant part of its startup time calling copy.deepcopy, so I thought > I'd try to write a C extension to speed this up. Maybe you should consider another option: using copy.dpeecopy() is *bad* practice and should be avoided in almost all cases. By design, copy.deepcopy() cannot be efficient. For efficient code, you should redesign the application to use immutable structures for example. Victor From tjreedy at udel.edu Wed Nov 2 08:09:57 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 2 Nov 2016 08:09:57 -0400 Subject: [Python-Dev] Implementing (parts of) copy module in C In-Reply-To: References: <878tt2x3t6.fsf@rasmusvillemoes.dk> <22553.34394.464326.178260@turnbull.sk.tsukuba.ac.jp> Message-ID: On 11/2/2016 3:54 AM, Mark Lawrence via Python-Dev wrote: > On 02/11/2016 06:23, Stephen J. Turnbull wrote: >> That is correct. This is clearly a feature, and 2.7 currently is >> accepting only security-related patches (broadly construed -- a >> sufficiently severe bug, such as a crash or infloop, is security- >> related because it could be used to implement a DoS attack). >> > > Surely patches related to any bugs, not just security related ones, will > be accepted until EOL in 2020? That depends on the maintainers of a particular module. Some core developers have stopped patching 2.7. One should ask before writing and submitting non-security 2.7 code that does not clearly have a chance to be applied. -- Terry Jan Reedy From breamoreboy at yahoo.co.uk Wed Nov 2 08:54:56 2016 From: breamoreboy at yahoo.co.uk (Mark Lawrence) Date: Wed, 2 Nov 2016 12:54:56 +0000 Subject: [Python-Dev] Implementing (parts of) copy module in C In-Reply-To: References: <878tt2x3t6.fsf@rasmusvillemoes.dk> <22553.34394.464326.178260@turnbull.sk.tsukuba.ac.jp> Message-ID: On 02/11/2016 12:09, Terry Reedy wrote: > On 11/2/2016 3:54 AM, Mark Lawrence via Python-Dev wrote: >> On 02/11/2016 06:23, Stephen J. Turnbull wrote: > >>> That is correct. This is clearly a feature, and 2.7 currently is >>> accepting only security-related patches (broadly construed -- a >>> sufficiently severe bug, such as a crash or infloop, is security- >>> related because it could be used to implement a DoS attack). >>> >> >> Surely patches related to any bugs, not just security related ones, will >> be accepted until EOL in 2020? > > That depends on the maintainers of a particular module. Some core > developers have stopped patching 2.7. One should ask before writing and > submitting non-security 2.7 code that does not clearly have a chance to > be applied. > Okay, thanks for that :) -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Nov 2 10:26:42 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 2 Nov 2016 23:26:42 +0900 Subject: [Python-Dev] Implementing (parts of) copy module in C In-Reply-To: References: <878tt2x3t6.fsf@rasmusvillemoes.dk> <22553.34394.464326.178260@turnbull.sk.tsukuba.ac.jp> Message-ID: <22553.63394.855716.269606@turnbull.sk.tsukuba.ac.jp> Mark Lawrence via Python-Dev writes: > Surely patches related to any bugs, not just security related ones, will > be accepted until EOL in 2020? Maybe, since it is the last in the 2.x line -- ask the RM, not me. I shouldn't have said anything, my apologies. But that doesn't matter for this contribution, since it's obviously a feature. From franciscouzo at gmail.com Wed Nov 2 18:03:27 2016 From: franciscouzo at gmail.com (Francisco Couzo) Date: Wed, 2 Nov 2016 19:03:27 -0300 Subject: [Python-Dev] itertools predicates Message-ID: Some functions that take a predicate from itertools accept None as its predicate: >>> list(itertools.filterfalse(None, range(10))) [0] >>> list(itertools.groupby([0,0,1,1], None)) [(0, ), (1, )] While others don't: >>> list(itertools.dropwhile(None, range(10))) Traceback (most recent call last): File "", line 1, in TypeError: 'NoneType' object is not callable I'd be interested in writing a patch to make itertools more consistent if there's a consensus. I see two possible solutions: - Either remove None as a predicate parameter, which would not be backwards compatible. - Or make the functions that don't accept None start accepting it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Nov 2 19:59:40 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 2 Nov 2016 19:59:40 -0400 Subject: [Python-Dev] itertools predicates In-Reply-To: References: Message-ID: On 11/2/2016 6:03 PM, Francisco Couzo wrote: > Some functions that take a predicate from itertools accept None as its > predicate: > >>>> list(itertools.filterfalse(None, range(10))) > [0] >>>> list(itertools.groupby([0,0,1,1], None)) > [(0, ), (1, > 0471d0>)] > > While others don't: > >>>> list(itertools.dropwhile(None, range(10))) > Traceback (most recent call last): > File "", line 1, in > TypeError: 'NoneType' object is not callable > > I'd be interested in writing a patch to make itertools more consistent > if there's a consensus. > > I see two possible solutions: > > * Either remove None as a predicate parameter, which would not be > backwards compatible. > * Or make the functions that don't accept None start accepting it. I think you should post this as an enhancement proposal on the tracker. Put Raymond Hettinger, the itertools originator and maintainer, as nosy. -- Terry Jan Reedy From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Nov 3 06:07:34 2016 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 3 Nov 2016 19:07:34 +0900 Subject: [Python-Dev] itertools predicates In-Reply-To: References: Message-ID: <22555.3174.620662.967425@turnbull.sk.tsukuba.ac.jp> Francisco Couzo writes: > I'd be interested in writing a patch to make itertools more consistent if > there's a consensus. I don't understand what you mean by "consistent". I would argue that in Python, an argument of None means "use the TOOWTDI default". For "filterfalse", bool() is pretty obvious for the default predicate since most Python classes do have a boolean interpretation. For "groupby", I guess the identity is the "intuitive" default. But for dropwhile, bool() or the identity seem like obvious choices, but intuitively they're not very useful since the head of what's left is a false-y. When would you want that? Typically false-ies are "nothing to see here, people, move along" values. I guess that consideration suggest lambda x: not(x), but I don't see why one would want to spell that dropwhile(None, ...) rather than dropwhile(not, ...), especially since in many contexts what you might really want (and expect None to default to) is dropwhile(lambda x: x is None, ...). Bottom line for me is that this use (and non-use) of None is consistent with Python practice regarding defaults. So, "accepts None" vs "doesn't accept None" doesn't seem to me to be an important enough consistency to impose unobvious semantics on dropwhile(None, ...). From franciscouzo at gmail.com Thu Nov 3 12:10:51 2016 From: franciscouzo at gmail.com (Francisco Couzo) Date: Thu, 3 Nov 2016 13:10:51 -0300 Subject: [Python-Dev] itertools predicates In-Reply-To: <22555.3174.620662.967425@turnbull.sk.tsukuba.ac.jp> References: <22555.3174.620662.967425@turnbull.sk.tsukuba.ac.jp> Message-ID: Stephen, dropwhile(None, ...) was an example, takewhile(None, ...) doesn't works either and it makes a lot of sense IMO. On Thu, Nov 3, 2016 at 7:07 AM, Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp> wrote: > Francisco Couzo writes: > > > I'd be interested in writing a patch to make itertools more consistent > if > > there's a consensus. > > I don't understand what you mean by "consistent". I would argue that > in Python, an argument of None means "use the TOOWTDI default". For > "filterfalse", bool() is pretty obvious for the default predicate > since most Python classes do have a boolean interpretation. For > "groupby", I guess the identity is the "intuitive" default. But for > dropwhile, bool() or the identity seem like obvious choices, but > intuitively they're not very useful since the head of what's left is a > false-y. When would you want that? Typically false-ies are "nothing > to see here, people, move along" values. I guess that consideration > suggest lambda x: not(x), but I don't see why one would want to spell > that dropwhile(None, ...) rather than dropwhile(not, ...), especially > since in many contexts what you might really want (and expect None to > default to) is dropwhile(lambda x: x is None, ...). > > Bottom line for me is that this use (and non-use) of None is consistent > with Python practice regarding defaults. > > So, "accepts None" vs "doesn't accept None" doesn't seem to me to be > an important enough consistency to impose unobvious semantics on > dropwhile(None, ...). > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From fabiofz at gmail.com Thu Nov 3 15:06:21 2016 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Thu, 3 Nov 2016 17:06:21 -0200 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation Message-ID: Hi Python-Dev, I'm trying to get my head around on what's accepted in f-strings -- https://www.python.org/dev/peps/pep-0498/ seems very light on the details on what it does accept as an expression and how things should actually be parsed (and the current implementation still doesn't seem to be in a state for a final release, so, I thought asking on python-dev would be a reasonable option). I was thinking there'd be some grammar for it (something as https://docs.python.org/3.6/reference/grammar.html), but all I could find related to this is a quote saying that f-strings should be something as: f ' { } So, given that, is it safe to assume that would be equal to the "test" node from the official grammar? I initially thought it would obviously be, but the PEP says that using a lamda inside the expression would conflict because of the colon (which wouldn't happen if a proper grammar was actually used for this parsing as there'd be no conflict as the lamda would properly consume the colon), so, I guess some pre-parser steps takes place to separate the expression to then be parsed, so, I'm interested on knowing how exactly that should work when the implementation is finished -- lots of plus points if there's actually a grammar to back it up :) Thanks, Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Nov 3 18:51:05 2016 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 4 Nov 2016 09:51:05 +1100 Subject: [Python-Dev] itertools predicates In-Reply-To: References: Message-ID: <20161103225104.GS3365@ando.pearwood.info> On Wed, Nov 02, 2016 at 07:03:27PM -0300, Francisco Couzo wrote: > Some functions that take a predicate from itertools accept None as its > predicate: [...] > I'd be interested in writing a patch to make itertools more consistent if > there's a consensus. There's a line in PEP 8 about foolish consistency. Consistency just for the sake of consistency is not a virtue. I have no objection to adding support for None to individual itertools functions on the basis that it makes sense for that function. If you can suggest a reasonable and *useful* default predicate or key function to use for None, then sure, add it to the tracker as a feature enhancement. (And, of course, a patch would be appreciated.) But that will be because it is a useful enhancement, not because of "consistency". > I see two possible solutions: Before there can be a solution, there first has to be a problem that needs solving. "Lack of consistency" is not necessarily a problem. The intertools functions are quite different, they do different things with different APIs. The question here should not be "why don't these functions take None as an argument?", rather it should be "why should these functions take None as an argument?". -- Steve From rv at rasmusvillemoes.dk Thu Nov 3 19:16:30 2016 From: rv at rasmusvillemoes.dk (Rasmus Villemoes) Date: Fri, 04 Nov 2016 00:16:30 +0100 Subject: [Python-Dev] Implementing (parts of) copy module in C References: <878tt2x3t6.fsf@rasmusvillemoes.dk> <22553.34394.464326.178260@turnbull.sk.tsukuba.ac.jp> Message-ID: <87h97o40j5.fsf@rasmusvillemoes.dk> On Wed, Nov 02 2016, "Stephen J. Turnbull" wrote: > > Where/how should RFC patches be posted? > > To the issue tracker, bugs.python.org. A follow-up here (and if you > choose to post to python-list, there) to mention the issue number and > URL is a good idea. Thanks. Turns out to be a lot easier to forward-port than I feared, so I've put an RFC at . Rasmus From eric at trueblade.com Fri Nov 4 03:56:35 2016 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 4 Nov 2016 03:56:35 -0400 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: References: Message-ID: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> On 11/3/2016 3:06 PM, Fabio Zadrozny wrote: > Hi Python-Dev, > > I'm trying to get my head around on what's accepted in f-strings -- > https://www.python.org/dev/peps/pep-0498/ seems very light on the > details on what it does accept as an expression and how things should > actually be parsed (and the current implementation still doesn't seem to > be in a state for a final release, so, I thought asking on python-dev > would be a reasonable option). In what way do you think the implementation isn't ready for a final release? > I was thinking there'd be some grammar for it (something as > https://docs.python.org/3.6/reference/grammar.html), but all I could > find related to this is a quote saying that f-strings should be > something as: > > f ' { specifier> } > > So, given that, is it safe to assume that would be equal to > the "test" node from the official grammar? No. There are really three phases here: 1. The f-string is tokenized as a regular STRING token, like all other strings (f-, b-, u-, r-, etc). 2. The parser sees that it's an f-string, and breaks it into expression and text parts. 3. For each expression found, the expression is compiled with PyParser_ASTFromString(..., Py_eval_input, ...). Step 2 is the part that limits what types of expressions are allowed. While scanning for the end of an expression, it stops at the first '!', ':', or '}' that isn't inside of a string and isn't nested inside of parens, braces, and brackets. The nesting-tracking is why these work: >>> f'{(lambda x:3)}' ' at 0x000000000296E560>' >>> f'{(lambda x:3)!s:.20}' ' a' But this doesn't: >>> f'{lambda x:3}' File "", line 1 (lambda x) ^ SyntaxError: unexpected EOF while parsing Also, backslashes are not allowed anywhere inside of the expression. This was a late change right before beta 1 (I think), and differs from the PEP and docs. I have an open item to fix them. > I initially thought it would obviously be, but the PEP says that using a > lamda inside the expression would conflict because of the colon (which > wouldn't happen if a proper grammar was actually used for this parsing > as there'd be no conflict as the lamda would properly consume the > colon), so, I guess some pre-parser steps takes place to separate the > expression to then be parsed, so, I'm interested on knowing how exactly > that should work when the implementation is finished -- lots of plus > points if there's actually a grammar to back it up :) I've considered using the grammar and tokenizer to implement f-string parsing, but I doubt it will ever happen. It's a lot of work, and everything that produced or consumed tokens would have to be aware of it. As it stands, if you don't need to look inside of f-strings, you can just treat them as regular STRING tokens. I hope that helps. Eric. From hodgestar+pythondev at gmail.com Fri Nov 4 04:36:06 2016 From: hodgestar+pythondev at gmail.com (Simon Cross) Date: Fri, 4 Nov 2016 10:36:06 +0200 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> Message-ID: On Fri, Nov 4, 2016 at 9:56 AM, Eric V. Smith wrote: > 2. The parser sees that it's an f-string, and breaks it into expression and > text parts. I'm with Fabio here. It would be really nice to have a grammar specified and documented for this step, even if it's not implemented that way. Otherwise it's going to be very hard for, e.g., syntax highlighters to know what is intended to be allowed. From p.f.moore at gmail.com Fri Nov 4 05:36:20 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 4 Nov 2016 09:36:20 +0000 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> Message-ID: On 4 November 2016 at 08:36, Simon Cross wrote: > On Fri, Nov 4, 2016 at 9:56 AM, Eric V. Smith wrote: >> 2. The parser sees that it's an f-string, and breaks it into expression and >> text parts. > > I'm with Fabio here. It would be really nice to have a grammar > specified and documented for this step, even if it's not implemented > that way. Otherwise it's going to be very hard for, e.g., syntax > highlighters to know what is intended to be allowed. I think that if the docs explain the process, essentially as noted by Eric above: > Step 2 is the part that limits what types of expressions are allowed. > While scanning for the end of an expression, it stops at the first '!', ':', or '}' that > isn't inside of a string and isn't nested inside of parens, braces, and brackets. [...] > Also, backslashes are not allowed anywhere inside of the expression. then that would be fine. Possibly a bit more detail would be helpful, but I'm pretty sure I could reimplement the behaviour (for a syntax highlighter, for example) based on the above. I assume that the open item Eric mentions to fix the PEP and docs is sufficient to cover this, so it'll be documented in due course. Paul From victor.stinner at gmail.com Fri Nov 4 08:53:10 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 4 Nov 2016 13:53:10 +0100 Subject: [Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance Message-ID: (Re-post without the two attached files of 100 KB each.) Hi, You may know that I'm working on benchmarks. I regenerated all benchmark results of speed.python.org using performance 0.3.2 (benchmark suite). I started to analyze results. All results are available online on the website: https://speed.python.org/ To communicate on my work on benchmarks, I tweeted two pictures: "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python 2.7 #python #benchmark": https://twitter.com/VictorStinner/status/794289596683210760 "Python 3.6 is between 25% and 54% slower than Python 2.7 in the following benchmarks": https://twitter.com/VictorStinner/status/794305065708376069 Many people were disappointed that Python 3.6 can be up to 54% slower than Python 2.7. In fact, I know many reasons which explain that, but it's hard to summarize them in 140 characters ;-) For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark pycrypto_aes. This benchmark tests a pure Python implementation of the crypto cipher AES. You may know that CPython is slow for CPU intensive functions, especially on integer and floatting point numbers. "int" in Python 3 is now "long integers" by default, which is known to be a little bit slower than "short int" of Python 2. On a more realistic benchmark (see other benchmarks), the overhead of Python 3 "long int" is negligible. AES is a typical example stressing integers. For me, it's a dummy benchmark: it doesn't make sense to use Python for AES: modern CPUs have an *hardware* implemention which is super fast. Well, I didn't have time to analyze in depth individual benchmarks. If you want to help me, here is the source code of benchmarks: https://github.com/python/performance/blob/master/performance/benchmarks/ Raw results of Python 3.6 compared to Python 2.7: ------------------- $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 Slower (40): - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: 1.07x slower - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower Faster (15): - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster Benchmark hidden because not significant (8): 2to3, dulwich_log, nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json: hg_startup, pyflate, spambayes ------------------- Please ignore call_method, call_method_slots, call_method_unknown benchmarks: it seems like I had an issue on the benchmark server. I was unable to reproduce he 70% slowdown on my laptop. JSON files if you want to analyze them yourself: http://www.haypocalc.com/tmp/2016-11-03_15-36-2.7-91f024fc9b3a.json.gz http://www.haypocalc.com/tmp/2016-11-03_15-38-3.6-c4319c0d0131.json.gz I hope that my work on benchmarks will motive some developers to look closer at Python 3 performance to find interesting optimizations ;-) Victor From fabiofz at gmail.com Fri Nov 4 10:50:01 2016 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Fri, 4 Nov 2016 12:50:01 -0200 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> Message-ID: Answers inline... On Fri, Nov 4, 2016 at 5:56 AM, Eric V. Smith wrote: > On 11/3/2016 3:06 PM, Fabio Zadrozny wrote: > >> Hi Python-Dev, >> >> I'm trying to get my head around on what's accepted in f-strings -- >> https://www.python.org/dev/peps/pep-0498/ seems very light on the >> details on what it does accept as an expression and how things should >> actually be parsed (and the current implementation still doesn't seem to >> be in a state for a final release, so, I thought asking on python-dev >> would be a reasonable option). >> > > In what way do you think the implementation isn't ready for a final > release? Well, the cases listed in the docs? (https://hg.python.org/ cpython/file/default/Doc/reference/lexical_analysis.rst) don't work in the latest release (with SyntaxErrors) -- and the bug I created related to it: http://bugs.python.org/issue28597 was promptly closed as duplicate -- so, I assumed (maybe wrongly?) that the parsing still needs work. > I was thinking there'd be some grammar for it (something as >> https://docs.python.org/3.6/reference/grammar.html), but all I could >> find related to this is a quote saying that f-strings should be >> something as: >> >> f ' { > specifier> } >> >> So, given that, is it safe to assume that would be equal to >> the "test" node from the official grammar? >> > > No. There are really three phases here: > > 1. The f-string is tokenized as a regular STRING token, like all other > strings (f-, b-, u-, r-, etc). > 2. The parser sees that it's an f-string, and breaks it into expression > and text parts. > 3. For each expression found, the expression is compiled with > PyParser_ASTFromString(..., Py_eval_input, ...). > > Step 2 is the part that limits what types of expressions are allowed. > While scanning for the end of an expression, it stops at the first '!', > ':', or '}' that isn't inside of a string and isn't nested inside of > parens, braces, and brackets. > ?It'd be nice if at least this description could be added to the PEP (as all other language implementations and IDEs will have to work the same way and will probably reference it) -- a grammar example, even if not used would be helpful (personally, I think hand-crafted parsers are always worse in the long run compared to having a proper grammar with a parser, although I understand that if you're not really used to it, it may be more work to set it up). Also, I find it a bit troubling that PyParser_ASTFromString is used there and not just the node which would be related to an expression, although I understand it's probably an easier approach, although in the end you probably have to filter it and end up just accepting what's beneath the "test" from the grammar, no? (i.e.: that's what a lambda body accepts). > The nesting-tracking is why these work: > >>> f'{(lambda x:3)}' > ' at 0x000000000296E560>' > >>> f'{(lambda x:3)!s:.20}' > ' a' > > But this doesn't: > >>> f'{lambda x:3}' > File "", line 1 > (lambda x) > ^ > SyntaxError: unexpected EOF while parsing > > Also, backslashes are not allowed anywhere inside of the expression. This > was a late change right before beta 1 (I think), and differs from the PEP > and docs. I have an open item to fix them. > > I initially thought it would obviously be, but the PEP says that using a >> lamda inside the expression would conflict because of the colon (which >> wouldn't happen if a proper grammar was actually used for this parsing >> as there'd be no conflict as the lamda would properly consume the >> colon), so, I guess some pre-parser steps takes place to separate the >> expression to then be parsed, so, I'm interested on knowing how exactly >> that should work when the implementation is finished -- lots of plus >> points if there's actually a grammar to back it up :) >> > > I've considered using the grammar and tokenizer to implement f-string > parsing, but I doubt it will ever happen. It's a lot of work, and > everything that produced or consumed tokens would have to be aware of it. > As it stands, if you don't need to look inside of f-strings, you can just > treat them as regular STRING tokens. > ?Well, I think all language implementations / IDEs (or at least those which want to give syntax errors) will *have* to look inside f-strings. Also, you could still have a separate grammar saying how to look inside f-strings (this would make the lives of other implementors easier) even if it was a post-processing step as you're doing now. > > I hope that helps. > > Eric. > ?It does, thank you very much. ?Best Regards, Fabio? -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Nov 4 10:50:42 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 4 Nov 2016 15:50:42 +0100 Subject: [Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance In-Reply-To: References: Message-ID: The nice thing with benchmarks is that you can always take a subset of numbers to prove something :-) Another subset where Python 3.6 is faster: "Python 3.6 is between 12% (1.14x) and 97% (32x) FASTER than Python 2.7 in the following benchmarks:" https://twitter.com/VictorStinner/status/794525289623719937 Victor 2016-11-04 13:53 GMT+01:00 Victor Stinner : > (Re-post without the two attached files of 100 KB each.) > > Hi, > > You may know that I'm working on benchmarks. I regenerated all > benchmark results of speed.python.org using performance 0.3.2 > (benchmark suite). I started to analyze results. > > All results are available online on the website: > > https://speed.python.org/ > > > To communicate on my work on benchmarks, I tweeted two pictures: > > "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python > 2.7 #python #benchmark": > https://twitter.com/VictorStinner/status/794289596683210760 > > "Python 3.6 is between 25% and 54% slower than Python 2.7 in the > following benchmarks": > https://twitter.com/VictorStinner/status/794305065708376069 > > > Many people were disappointed that Python 3.6 can be up to 54% slower > than Python 2.7. In fact, I know many reasons which explain that, but > it's hard to summarize them in 140 characters ;-) > > For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark > pycrypto_aes. This benchmark tests a pure Python implementation of the > crypto cipher AES. You may know that CPython is slow for CPU intensive > functions, especially on integer and floatting point numbers. > > "int" in Python 3 is now "long integers" by default, which is known to > be a little bit slower than "short int" of Python 2. On a more > realistic benchmark (see other benchmarks), the overhead of Python 3 > "long int" is negligible. > > AES is a typical example stressing integers. For me, it's a dummy > benchmark: it doesn't make sense to use Python for AES: modern CPUs > have an *hardware* implemention which is super fast. > > > Well, I didn't have time to analyze in depth individual benchmarks. If > you want to help me, here is the source code of benchmarks: > https://github.com/python/performance/blob/master/performance/benchmarks/ > > > Raw results of Python 3.6 compared to Python 2.7: > ------------------- > $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz > 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 > Slower (40): > - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower > - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower > - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower > - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower > - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower > - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower > - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower > - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower > - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower > - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower > - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower > - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower > - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower > - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower > - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower > - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower > - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower > - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower > - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower > - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower > - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower > - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower > - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower > - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower > - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower > - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower > - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower > - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower > - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower > - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower > - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower > - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower > - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower > - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower > - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower > - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower > - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower > - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower > - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: > 1.07x slower > - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower > > Faster (15): > - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster > - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster > - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster > - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster > - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster > - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster > - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster > - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster > - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster > - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster > - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster > - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster > - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster > - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster > - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster > > Benchmark hidden because not significant (8): 2to3, dulwich_log, > nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle > Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json: > hg_startup, pyflate, spambayes > ------------------- > > Please ignore call_method, call_method_slots, call_method_unknown > benchmarks: it seems like I had an issue on the benchmark server. I > was unable to reproduce he 70% slowdown on my laptop. > > JSON files if you want to analyze them yourself: > > http://www.haypocalc.com/tmp/2016-11-03_15-36-2.7-91f024fc9b3a.json.gz > http://www.haypocalc.com/tmp/2016-11-03_15-38-3.6-c4319c0d0131.json.gz > > I hope that my work on benchmarks will motive some developers to look > closer at Python 3 performance to find interesting optimizations ;-) > > Victor From victor.stinner at gmail.com Fri Nov 4 08:28:36 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 4 Nov 2016 13:28:36 +0100 Subject: [Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance Message-ID: Hi, You may know that I'm working on benchmarks. I regenerated all benchmark results of speed.python.org using performance 0.3.2 (benchmark suite). I started to analyze results. All results are available online on the website: https://speed.python.org/ To communicate on my work on benchmarks, I tweeted two pictures: "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python 2.7 #python #benchmark": https://twitter.com/VictorStinner/status/794289596683210760 "Python 3.6 is between 25% and 54% slower than Python 2.7 in the following benchmarks": https://twitter.com/VictorStinner/status/794305065708376069 Many people were disappointed that Python 3.6 can be up to 54% slower than Python 2.7. In fact, I know many reasons which explain that, but it's hard to summarize them in 140 characters ;-) For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark pycrypto_aes. This benchmark tests a pure Python implementation of the crypto cipher AES. You may know that CPython is slow for CPU intensive functions, especially on integer and floatting point numbers. "int" in Python 3 is now "long integers" by default, which is known to be a little bit slower than "short int" of Python 2. On a more realistic benchmark (see other benchmarks), the overhead of Python 3 "long int" is negligible. AES is a typical example stressing integers. For me, it's a dummy benchmark: it doesn't make sense to use Python for AES: modern CPUs have an *hardware* implemention which is super fast. Well, I didn't have time to analyze in depth individual benchmarks. If you want to help me, here is the source code of benchmarks: https://github.com/python/performance/blob/master/performance/benchmarks/ Raw results of Python 3.6 compared to Python 2.7: ------------------- $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 Slower (40): - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x slower - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: 1.07x slower - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower Faster (15): - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster Benchmark hidden because not significant (8): 2to3, dulwich_log, nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json: hg_startup, pyflate, spambayes ------------------- Please ignore call_method, call_method_slots, call_method_unknown benchmarks: it seems like I had an issue on the benchmark server. I was unable to reproduce he 70% slowdown on my laptop. I attached the two compressed JSON files to this email if you want to analyze them yourself. I hope that my work on benchmarks will motive some developers to look closer at Python 3 performance to find interesting optimizations ;-) Victor -------------- next part -------------- A non-text attachment was scrubbed... Name: 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz Type: application/x-gzip Size: 107594 bytes Desc: not available URL: -------------- next part -------------- A non-text attachment was scrubbed... Name: 2016-11-03_15-38-3.6-c4319c0d0131.json.gz Type: application/x-gzip Size: 102546 bytes Desc: not available URL: From status at bugs.python.org Fri Nov 4 13:08:50 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 4 Nov 2016 18:08:50 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20161104170850.7F19756A85@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-10-28 - 2016-11-04) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5547 (+23) closed 34821 (+41) total 40368 (+64) Open issues with patches: 2405 Issues opened (43) ================== #28199: Compact dict resizing is doing too much work http://bugs.python.org/issue28199 reopened by ned.deily #28473: mailbox.MH crashes on certain Claws Mail .mh_sequences files http://bugs.python.org/issue28473 reopened by r.david.murray #28551: sysconfig.py wrong _PROJECT_BASE for Py2.7 Windows 64bit PC/VS http://bugs.python.org/issue28551 opened by anselm.kruis #28552: Distutils fail if sys.executable is None http://bugs.python.org/issue28552 opened by iamale #28554: Windows: _socket module fails to compile on "AMD64 Windows7 SP http://bugs.python.org/issue28554 opened by haypo #28555: provid also sha-1 and sha-256 also on download links http://bugs.python.org/issue28555 opened by Big Stone #28556: typing.py upgrades http://bugs.python.org/issue28556 opened by gvanrossum #28557: error message for bad raw readinto http://bugs.python.org/issue28557 opened by davidszotten #28559: Unclear error message when raising wrong type of exceptions http://bugs.python.org/issue28559 opened by Dimitri Merejkowsky #28562: test_asyncio fails on Android upon calling getaddrinfo() http://bugs.python.org/issue28562 opened by xdegaye #28563: Arbitrary code execution in gettext.c2py http://bugs.python.org/issue28563 opened by Carl Ekerot #28564: shutil.rmtree is inefficient due to listdir() instead of scand http://bugs.python.org/issue28564 opened by enkore #28565: datetime.strptime %Z doesn't produce aware object http://bugs.python.org/issue28565 opened by lilydjwg #28567: Bus error / segmentation fault on macOS debug build when using http://bugs.python.org/issue28567 opened by Alex Croitor #28569: mock.attach_mock should work with any return value of patch() http://bugs.python.org/issue28569 opened by anfedorov #28570: httplib mishandles unknown 1XX status codes http://bugs.python.org/issue28570 opened by Lukasa #28571: llist and scipy.stats conflicts, python segfault http://bugs.python.org/issue28571 opened by Llu??s #28572: IDLE: add tests for config dialog. http://bugs.python.org/issue28572 opened by terry.reedy #28573: Python 3.6.0b3 64-bit has no sys._mercurial info http://bugs.python.org/issue28573 opened by steve.dower #28574: Update bundled pip http://bugs.python.org/issue28574 opened by steve.dower #28576: Uninstalling Py352 x86 with /uninstall option does not remove http://bugs.python.org/issue28576 opened by jcrmatos #28577: ipaddress.ip_network(...).hosts() returns nothing for an IPv4 http://bugs.python.org/issue28577 opened by era #28584: ICC compiler check is too permissive http://bugs.python.org/issue28584 opened by struktured #28585: Restore docstring of os._isdir http://bugs.python.org/issue28585 opened by serhiy.storchaka #28586: Convert os.scandir to Argument Clinic http://bugs.python.org/issue28586 opened by serhiy.storchaka #28587: list.index documentation missing start and stop arguments http://bugs.python.org/issue28587 opened by ChrisRands #28588: Memory leak in OpenSSL thread state http://bugs.python.org/issue28588 opened by christian.heimes #28590: fstring's '{' from escape sequences does not start an expressi http://bugs.python.org/issue28590 opened by afg984 #28591: imghdr doesn't recognize some jpeg formats http://bugs.python.org/issue28591 opened by 4simple-org #28592: Installation freezes on C Runtime Install http://bugs.python.org/issue28592 opened by Ex Tracheese #28593: Inconsistent itertools' predicate behaviour http://bugs.python.org/issue28593 opened by franciscouzo #28595: shlex.shlex should not augment wordchars http://bugs.python.org/issue28595 opened by evan_ #28596: on Android _bootlocale on startup relies on too many library m http://bugs.python.org/issue28596 opened by xdegaye #28598: RHS not consulted in `str % subclass_of_str` case. http://bugs.python.org/issue28598 opened by mjpieters #28601: Ambiguous datetime comparisons should use == rather than 'is' http://bugs.python.org/issue28601 opened by p-ganssle #28602: `tzinfo.fromutc()` fails when used for a fold-aware tzinfo imp http://bugs.python.org/issue28602 opened by p-ganssle #28603: traceback module can't format/print unhashable exceptions http://bugs.python.org/issue28603 opened by Trundle #28604: Exception raised by python3.5 when using en_GB locale http://bugs.python.org/issue28604 opened by Guillaume Pasquet (Etenil) #28607: C implementation of parts of copy.deepcopy http://bugs.python.org/issue28607 opened by villemoes #28608: Support creating hardlink using `pathlib` http://bugs.python.org/issue28608 opened by cool-RR #28609: argparse claims '*' positional argument is required in error o http://bugs.python.org/issue28609 opened by rrt #28610: Provide PDB hook to customize how to find source files http://bugs.python.org/issue28610 opened by Pinku Surana #28612: str.translate needs a mapping example http://bugs.python.org/issue28612 opened by Jim.Jewett Most recent 15 issues with no replies (15) ========================================== #28612: str.translate needs a mapping example http://bugs.python.org/issue28612 #28610: Provide PDB hook to customize how to find source files http://bugs.python.org/issue28610 #28609: argparse claims '*' positional argument is required in error o http://bugs.python.org/issue28609 #28608: Support creating hardlink using `pathlib` http://bugs.python.org/issue28608 #28607: C implementation of parts of copy.deepcopy http://bugs.python.org/issue28607 #28603: traceback module can't format/print unhashable exceptions http://bugs.python.org/issue28603 #28596: on Android _bootlocale on startup relies on too many library m http://bugs.python.org/issue28596 #28595: shlex.shlex should not augment wordchars http://bugs.python.org/issue28595 #28593: Inconsistent itertools' predicate behaviour http://bugs.python.org/issue28593 #28588: Memory leak in OpenSSL thread state http://bugs.python.org/issue28588 #28586: Convert os.scandir to Argument Clinic http://bugs.python.org/issue28586 #28585: Restore docstring of os._isdir http://bugs.python.org/issue28585 #28569: mock.attach_mock should work with any return value of patch() http://bugs.python.org/issue28569 #28567: Bus error / segmentation fault on macOS debug build when using http://bugs.python.org/issue28567 #28562: test_asyncio fails on Android upon calling getaddrinfo() http://bugs.python.org/issue28562 Most recent 15 issues waiting for review (15) ============================================= #28607: C implementation of parts of copy.deepcopy http://bugs.python.org/issue28607 #28603: traceback module can't format/print unhashable exceptions http://bugs.python.org/issue28603 #28598: RHS not consulted in `str % subclass_of_str` case. http://bugs.python.org/issue28598 #28595: shlex.shlex should not augment wordchars http://bugs.python.org/issue28595 #28591: imghdr doesn't recognize some jpeg formats http://bugs.python.org/issue28591 #28587: list.index documentation missing start and stop arguments http://bugs.python.org/issue28587 #28586: Convert os.scandir to Argument Clinic http://bugs.python.org/issue28586 #28585: Restore docstring of os._isdir http://bugs.python.org/issue28585 #28584: ICC compiler check is too permissive http://bugs.python.org/issue28584 #28572: IDLE: add tests for config dialog. http://bugs.python.org/issue28572 #28570: httplib mishandles unknown 1XX status codes http://bugs.python.org/issue28570 #28563: Arbitrary code execution in gettext.c2py http://bugs.python.org/issue28563 #28562: test_asyncio fails on Android upon calling getaddrinfo() http://bugs.python.org/issue28562 #28559: Unclear error message when raising wrong type of exceptions http://bugs.python.org/issue28559 #28557: error message for bad raw readinto http://bugs.python.org/issue28557 Top 10 most discussed issues (10) ================================= #28199: Compact dict resizing is doing too much work http://bugs.python.org/issue28199 16 msgs #28542: document cross compilation http://bugs.python.org/issue28542 9 msgs #28591: imghdr doesn't recognize some jpeg formats http://bugs.python.org/issue28591 9 msgs #26163: FAIL: test_hash_effectiveness (test.test_set.TestFrozenSet) http://bugs.python.org/issue26163 8 msgs #28577: ipaddress.ip_network(...).hosts() returns nothing for an IPv4 http://bugs.python.org/issue28577 7 msgs #28123: _PyDict_GetItem_KnownHash ignores DKIX_ERROR return http://bugs.python.org/issue28123 6 msgs #28499: Logging module documentation needs a rework. http://bugs.python.org/issue28499 6 msgs #28532: Show sys.version when -V option is supplied twice. http://bugs.python.org/issue28532 6 msgs #28548: http.server parse_request() bug and error reporting http://bugs.python.org/issue28548 6 msgs #19899: No test for thread.interrupt_main() http://bugs.python.org/issue19899 5 msgs Issues closed (41) ================== #23693: timeit accuracy could be better http://bugs.python.org/issue23693 closed by haypo #25264: test_marshal always crashs on "AMD64 Windows10 2.7" buildbot http://bugs.python.org/issue25264 closed by serhiy.storchaka #26638: Avoid warnings about missing CLI options when building documen http://bugs.python.org/issue26638 closed by martin.panter #26980: The path argument of asyncio.BaseEventLoop.create_unix_connect http://bugs.python.org/issue26980 closed by gvanrossum #27517: LZMACompressor and LZMADecompressor raise exceptions if given http://bugs.python.org/issue27517 closed by serhiy.storchaka #27939: Tkinter mainloop raises when setting the value of ttk.LabeledS http://bugs.python.org/issue27939 closed by serhiy.storchaka #28088: Document Transport.set_protocol and get_protocol http://bugs.python.org/issue28088 closed by inada.naoki #28128: Improve the warning message for invalid escape sequences http://bugs.python.org/issue28128 closed by eric.smith #28208: update sqlite to 3.14.2 http://bugs.python.org/issue28208 closed by ned.deily #28248: Upgrade installers to OpenSSL 1.0.2j http://bugs.python.org/issue28248 closed by ned.deily #28385: Bytes objects should reject all formatting codes with an error http://bugs.python.org/issue28385 closed by serhiy.storchaka #28387: double free in io.TextIOWrapper http://bugs.python.org/issue28387 closed by serhiy.storchaka #28444: Missing extensions modules when cross compiling python 3.5.2 f http://bugs.python.org/issue28444 closed by xdegaye #28449: tarfile.open(mode = 'r:*', ignore_zeros = True) has 50% chance http://bugs.python.org/issue28449 closed by serhiy.storchaka #28513: Document zipfile CLI http://bugs.python.org/issue28513 closed by serhiy.storchaka #28522: can't make IDLEX work with python._pth and python-3.6.0b2 http://bugs.python.org/issue28522 closed by steve.dower #28530: Howto detect if an object is of type os.DirEntry http://bugs.python.org/issue28530 closed by brett.cannon #28541: Improve test coverage for json library - loading bytes http://bugs.python.org/issue28541 closed by serhiy.storchaka #28549: curses: calling addch() with an 1-length str segfaults with nc http://bugs.python.org/issue28549 closed by serhiy.storchaka #28550: if inline statement does not work with multiple assignment. http://bugs.python.org/issue28550 closed by eric.snow #28553: int.to_bytes docs logic error http://bugs.python.org/issue28553 closed by inada.naoki #28558: argparse Incorrect Handling of Store Actions http://bugs.python.org/issue28558 closed by xiang.zhang #28560: Implement `PurePath.startswith` and `PurePath.endswith` http://bugs.python.org/issue28560 closed by cool-RR #28561: Report surrogate characters range in utf8_encoder http://bugs.python.org/issue28561 closed by serhiy.storchaka #28566: Python installation fails on Windows because of pip feature http://bugs.python.org/issue28566 closed by steve.dower #28568: Build files in PC/VS9.0 contain an outdated sqlite version num http://bugs.python.org/issue28568 closed by python-dev #28575: Error 0x80070643 While installing in Win 10 32 Bit http://bugs.python.org/issue28575 closed by steve.dower #28578: '\n' escape character print before the Py_GetCompiler version http://bugs.python.org/issue28578 closed by SilentGhost #28579: nan != nan http://bugs.python.org/issue28579 closed by SilentGhost #28580: Optimize iterating split table values http://bugs.python.org/issue28580 closed by inada.naoki #28581: Allow running code without explicitly saving the file. http://bugs.python.org/issue28581 closed by terry.reedy #28582: Invalid backslash syntax errors are not always accurate as to http://bugs.python.org/issue28582 closed by eric.smith #28583: PyDict_SetDefault doesn't combine split table when needed http://bugs.python.org/issue28583 closed by inada.naoki #28589: get error when compile .py file during install stage if cross http://bugs.python.org/issue28589 closed by xdegaye #28594: List define and Change result http://bugs.python.org/issue28594 closed by martin.panter #28597: f-string behavior is conflicting with its documentation http://bugs.python.org/issue28597 closed by eric.smith #28599: AboutDialog set_name is ignored http://bugs.python.org/issue28599 closed by zach.ware #28600: asyncio: Optimize loop.call_soon http://bugs.python.org/issue28600 closed by yselivanov #28605: Remove mention of LTO when referencing --with-optimization in http://bugs.python.org/issue28605 closed by brett.cannon #28606: Suspected bug in python optimizer with decorators http://bugs.python.org/issue28606 closed by r.david.murray #28611: Syntax error when using raw strings ending with a backslash. http://bugs.python.org/issue28611 closed by ebarry From eric at trueblade.com Fri Nov 4 13:15:07 2016 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 4 Nov 2016 13:15:07 -0400 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> Message-ID: <3b718ea3-f416-057a-d8e1-634464dc7770@trueblade.com> On 11/4/2016 10:50 AM, Fabio Zadrozny wrote: > In what way do you think the implementation isn't ready for a final > release? > > > Well, the cases listed in the docs? > (https://hg.python.org/cpython/file/default/Doc/reference/lexical_analysis.rst > ) > don't work in the latest release (with SyntaxErrors) -- and the bug I > created related to it: http://bugs.python.org/issue28597 > was promptly closed as duplicate > -- so, I assumed (maybe wrongly?) that the parsing still needs work. It's not the parsing that needs work, it's the documentation. Those examples used to work, but the parser was deliberately changed to not support them. There's a long discussion on python-ideas about it, starting at https://mail.python.org/pipermail/python-ideas/2016-August/041727.html > ?It'd be nice if at least this description could be added to the PEP (as > all other language implementations and IDEs will have to work the same > way and will probably reference it) -- a grammar example, even if not > used would be helpful (personally, I think hand-crafted parsers are > always worse in the long run compared to having a proper grammar with a > parser, although I understand that if you're not really used to it, it > may be more work to set it up). I've written a parser generator just to understand how they work, so I'm completely sympathetic to this. However, in this case, I don't think it would be any easier. I'm basically writing a tokenizer, not an expression parser. It's much simpler. The actual parsing is handled by PyParser_ASTFromString. And as I state below, you have to also consider the parser consumers. > Also, I find it a bit troubling that PyParser_ASTFromString is used > there and not just the node which would be related to an expression, > although I understand it's probably an easier approach, although in the > end you probably have to filter it and end up just accepting what's > beneath the "test" from the grammar, no? (i.e.: that's what a lambda > body accepts). Using PyParser_ASTFromString is the easiest possible way to do this. Given a string, it returns an AST node. What could be simpler? > ?Well, I think all language implementations / IDEs (or at least those > which want to give syntax errors) will *have* to look inside f-strings. While it's probably true that IDEs (and definitely language implementations) will want to parse f-strings, I think there are many more code scanners that are not language implementations or IDEs. And by being "just" regular strings with a new prefix, it's trivial to get any parser that doesn't care about the internal structure to at least see f-strings as normal strings. > Also, you could still have a separate grammar saying how to look inside > f-strings (this would make the lives of other implementors easier) even > if it was a post-processing step as you're doing now. Yes. I've contemplated exposing the f-string scanner. That's the part that returns expressions (as strings) and literal strings. I realize that won't help 3.6. Eric. From raymond.hettinger at gmail.com Fri Nov 4 13:29:27 2016 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Fri, 4 Nov 2016 10:29:27 -0700 Subject: [Python-Dev] itertools predicates In-Reply-To: <20161103225104.GS3365@ando.pearwood.info> References: <20161103225104.GS3365@ando.pearwood.info> Message-ID: <449F716E-B602-4FAA-A6DE-61E7813FF060@gmail.com> > On Nov 3, 2016, at 3:51 PM, Steven D'Aprano wrote: > > Before there can be a solution, there first has to be a problem that > needs solving. "Lack of consistency" is not necessarily a problem. The > intertools functions are quite different, they do different things with > different APIs. The question here should not be "why don't these > functions take None as an argument?", rather it should be "why should > these functions take None as an argument?". I concur with Steven who articulated the issue perfectly. There isn't an actual problem here that needs to be solved (i.e. not a single user report in 15 years indicating that the API wasn't meeting the needs for real use-cases). I'm disinclined to churn the API unless there is a real need. FWIW, groupby() has the predicate as an optional argument so that you can write groupby('aaabbc') and have it group by value (much like the key-function on sorted() is optional). The two filter variants allow None as the first argument only for historical reasons -- once "bool" came along, it would have been better to write filter(bool, someiterable) in preference to using None which is less clear about its intention. The takewhile/dropwhile tools didn't have the same constraint to match a historical API, so there was an opportunity to have a clearer API with a simpler signature. As Terry suggested, if you have other itertools feature requests, please put them on the tracker an assign them to me. Thank you, Raymond From fabiofz at gmail.com Fri Nov 4 14:03:12 2016 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Fri, 4 Nov 2016 16:03:12 -0200 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: <3b718ea3-f416-057a-d8e1-634464dc7770@trueblade.com> References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> <3b718ea3-f416-057a-d8e1-634464dc7770@trueblade.com> Message-ID: On Fri, Nov 4, 2016 at 3:15 PM, Eric V. Smith wrote: > On 11/4/2016 10:50 AM, Fabio Zadrozny wrote: > >> In what way do you think the implementation isn't ready for a final >> release? >> >> >> Well, the cases listed in the docs? >> (https://hg.python.org/cpython/file/default/Doc/reference/ >> lexical_analysis.rst >> > lexical_analysis.rst>) >> don't work in the latest release (with SyntaxErrors) -- and the bug I >> created related to it: http://bugs.python.org/issue28597 >> was promptly closed as duplicate >> -- so, I assumed (maybe wrongly?) that the parsing still needs work. >> > > It's not the parsing that needs work, it's the documentation. Those > examples used to work, but the parser was deliberately changed to not > support them. There's a long discussion on python-ideas about it, starting > at https://mail.python.org/pipermail/python-ideas/2016-August/041727.html ?Understood ;) ? > > > ?It'd be nice if at least this description could be added to the PEP (as >> all other language implementations and IDEs will have to work the same >> way and will probably reference it) -- a grammar example, even if not >> used would be helpful (personally, I think hand-crafted parsers are >> always worse in the long run compared to having a proper grammar with a >> parser, although I understand that if you're not really used to it, it >> may be more work to set it up). >> > > I've written a parser generator just to understand how they work, so I'm > completely sympathetic to this. However, in this case, I don't think it > would be any easier. I'm basically writing a tokenizer, not an expression > parser. It's much simpler. The actual parsing is handled by > PyParser_ASTFromString. And as I state below, you have to also consider the > parser consumers. > > Also, I find it a bit troubling that >> ?? >> PyParser_ASTFromString is used >> there and not just the node which would be related to an expression, >> although I understand it's probably an easier approach, although in the >> end you probably have to filter it and end up just accepting what's >> beneath the "test" from the grammar, no? (i.e.: that's what a lambda >> body accepts). >> > > Using PyParser_ASTFromString is the easiest possible way to do this. Given > a string, it returns an AST node. What could be simpler? ?I think that for implementation purposes, given the python infrastructure, it's fine, but for specification purposes, probably incorrect... As I don't think f-strings should accept: f"start {import sys; sys.version_info[0];} end" (i.e.: ? PyParser_ASTFromString doesn't just return an expression, it accepts any valid Python code, even code which can't be used in an f-string). > ?Well, I think all language implementations / IDEs (or at least those >> which want to give syntax errors) will *have* to look inside f-strings. >> > > While it's probably true that IDEs (and definitely language > implementations) will want to parse f-strings, I think there are many more > code scanners that are not language implementations or IDEs. And by being > "just" regular strings with a new prefix, it's trivial to get any parser > that doesn't care about the internal structure to at least see f-strings as > normal strings. > > Also, you could still have a separate grammar saying how to look inside >> f-strings (this would make the lives of other implementors easier) even >> if it was a post-processing step as you're doing now. >> > > Yes. I've contemplated exposing the f-string scanner. That's the part that > returns expressions (as strings) and literal strings. I realize that won't > help 3.6. Nice... As a note, just for the record, my own interest on f-strings is knowing how exactly they are parsed for providing a preview of PyDev with syntax highlighting and preliminary support for f-strings (which at the very minimum besides syntax highlighting for the parts of f-strings should also show syntax errors inside them). ?Cheers,? ?Fabio -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Fri Nov 4 14:16:22 2016 From: eric at trueblade.com (Eric V. Smith) Date: Fri, 4 Nov 2016 14:16:22 -0400 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> <3b718ea3-f416-057a-d8e1-634464dc7770@trueblade.com> Message-ID: On 11/4/2016 2:03 PM, Fabio Zadrozny wrote: > Using PyParser_ASTFromString is the easiest possible way to do this. > Given a string, it returns an AST node. What could be simpler? > > > ?I think that for implementation purposes, given the python > infrastructure, it's fine, but for specification purposes, probably > incorrect... As I don't think f-strings should accept: > > f"start {import sys; sys.version_info[0];} end" (i.e.: > ? > PyParser_ASTFromString doesn't just return an expression, it accepts any > valid Python code, even code which can't be used in an f-string). Not so. It should only accept expressions, not statements: >>> f"start {import sys; sys.version_info[0];} end" File "", line 1 (import sys; sys.version_info[0];) ^ SyntaxError: invalid syntax > Also, you could still have a separate grammar saying how to look > inside > f-strings (this would make the lives of other implementors > easier) even > if it was a post-processing step as you're doing now. > > > Yes. I've contemplated exposing the f-string scanner. That's the > part that returns expressions (as strings) and literal strings. I > realize that won't help 3.6. > > > Nice... > > As a note, just for the record, my own interest on f-strings is knowing > how exactly they are parsed for providing a preview of PyDev with syntax > highlighting and preliminary support for f-strings (which at the very > minimum besides syntax highlighting for the parts of f-strings should > also show syntax errors inside them). I understand there's a need to make the specification more rigorous. Hopefully we'll get there. Eric. From tobami at gmail.com Fri Nov 4 15:18:48 2016 From: tobami at gmail.com (Miquel Torres) Date: Fri, 04 Nov 2016 19:18:48 +0000 Subject: [Python-Dev] [Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance In-Reply-To: References: Message-ID: Nice! For the record, I'll be giving a talk in PyCon Ireland about Codespeed. Would you mind me citing those tweets and screenshots, to highlight usage on speed.python.org? You mentioned new more reliable vs old results. How close are we to have an stable setup that gives us benchmarks numbers regularly? Cheers, Miquel El El vie, 4 nov 2016 a las 12:30, Victor Stinner escribi?: > Hi, > > You may know that I'm working on benchmarks. I regenerated all > benchmark results of speed.python.org using performance 0.3.2 > (benchmark suite). I started to analyze results. > > All results are available online on the website: > > https://speed.python.org/ > > > To communicate on my work on benchmarks, I tweeted two pictures: > > "sympy benchmarks: Python 3.6 is between 8% and 48% faster than Python > 2.7 #python #benchmark": > https://twitter.com/VictorStinner/status/794289596683210760 > > "Python 3.6 is between 25% and 54% slower than Python 2.7 in the > following benchmarks": > https://twitter.com/VictorStinner/status/794305065708376069 > > > Many people were disappointed that Python 3.6 can be up to 54% slower > than Python 2.7. In fact, I know many reasons which explain that, but > it's hard to summarize them in 140 characters ;-) > > For example, Python 3.6 is 54% slower than Python 2.7 on the benchmark > pycrypto_aes. This benchmark tests a pure Python implementation of the > crypto cipher AES. You may know that CPython is slow for CPU intensive > functions, especially on integer and floatting point numbers. > > "int" in Python 3 is now "long integers" by default, which is known to > be a little bit slower than "short int" of Python 2. On a more > realistic benchmark (see other benchmarks), the overhead of Python 3 > "long int" is negligible. > > AES is a typical example stressing integers. For me, it's a dummy > benchmark: it doesn't make sense to use Python for AES: modern CPUs > have an *hardware* implemention which is super fast. > > > Well, I didn't have time to analyze in depth individual benchmarks. If > you want to help me, here is the source code of benchmarks: > https://github.com/python/performance/blob/master/performance/benchmarks/ > > > Raw results of Python 3.6 compared to Python 2.7: > ------------------- > $ python3 -m perf compare_to 2016-11-03_15-36-2.7-91f024fc9b3a.json.gz > 2016-11-03_15-38-3.6-c4319c0d0131.json.gz -G --min-speed=5 > Slower (40): > - python_startup: 7.74 ms +- 0.28 ms -> 26.9 ms +- 0.6 ms: 3.47x slower > - python_startup_no_site: 4.43 ms +- 0.08 ms -> 10.4 ms +- 0.4 ms: 2.36x > slower > - unpickle_pure_python: 417 us +- 3 us -> 918 us +- 14 us: 2.20x slower > - call_method: 16.3 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.76x slower > - call_method_slots: 16.2 ms +- 0.4 ms -> 28.3 ms +- 0.7 ms: 1.75x slower > - call_method_unknown: 18.4 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower > - crypto_pyaes: 161 ms +- 2 ms -> 249 ms +- 2 ms: 1.54x slower > - xml_etree_parse: 201 ms +- 5 ms -> 298 ms +- 8 ms: 1.49x slower > - logging_simple: 26.4 us +- 0.3 us -> 38.4 us +- 0.7 us: 1.46x slower > - logging_format: 31.3 us +- 0.4 us -> 45.5 us +- 0.8 us: 1.45x slower > - pickle_pure_python: 986 us +- 9 us -> 1.41 ms +- 0.03 ms: 1.43x slower > - spectral_norm: 208 ms +- 2 ms -> 287 ms +- 2 ms: 1.38x slower > - logging_silent: 660 ns +- 7 ns -> 865 ns +- 31 ns: 1.31x slower > - chaos: 240 ms +- 2 ms -> 314 ms +- 4 ms: 1.31x slower > - go: 490 ms +- 2 ms -> 640 ms +- 26 ms: 1.31x slower > - xml_etree_iterparse: 178 ms +- 2 ms -> 230 ms +- 5 ms: 1.29x slower > - sqlite_synth: 8.29 us +- 0.16 us -> 10.6 us +- 0.2 us: 1.28x slower > - xml_etree_process: 210 ms +- 6 ms -> 268 ms +- 14 ms: 1.28x slower > - django_template: 387 ms +- 4 ms -> 484 ms +- 5 ms: 1.25x slower > - fannkuch: 830 ms +- 32 ms -> 1.04 sec +- 0.03 sec: 1.25x slower > - hexiom: 20.2 ms +- 0.1 ms -> 24.7 ms +- 0.2 ms: 1.22x slower > - chameleon: 26.1 ms +- 0.2 ms -> 31.9 ms +- 0.4 ms: 1.22x slower > - regex_compile: 395 ms +- 2 ms -> 482 ms +- 6 ms: 1.22x slower > - json_dumps: 25.8 ms +- 0.2 ms -> 31.0 ms +- 0.5 ms: 1.20x slower > - nqueens: 229 ms +- 2 ms -> 274 ms +- 2 ms: 1.20x slower > - genshi_text: 81.9 ms +- 0.6 ms -> 97.8 ms +- 1.1 ms: 1.19x slower > - raytrace: 1.17 sec +- 0.03 sec -> 1.39 sec +- 0.03 sec: 1.19x slower > - scimark_monte_carlo: 240 ms +- 7 ms -> 282 ms +- 10 ms: 1.17x slower > - scimark_sor: 441 ms +- 8 ms -> 517 ms +- 12 ms: 1.17x slower > - deltablue: 17.4 ms +- 0.1 ms -> 20.1 ms +- 0.6 ms: 1.16x slower > - sqlalchemy_declarative: 310 ms +- 3 ms -> 354 ms +- 6 ms: 1.14x slower > - call_simple: 12.2 ms +- 0.2 ms -> 13.9 ms +- 0.2 ms: 1.14x slower > - scimark_fft: 613 ms +- 19 ms -> 694 ms +- 23 ms: 1.13x slower > - meteor_contest: 191 ms +- 1 ms -> 215 ms +- 2 ms: 1.13x slower > - pathlib: 46.9 ms +- 0.4 ms -> 52.6 ms +- 0.9 ms: 1.12x slower > - richards: 181 ms +- 1 ms -> 201 ms +- 6 ms: 1.11x slower > - genshi_xml: 191 ms +- 2 ms -> 209 ms +- 2 ms: 1.10x slower > - float: 290 ms +- 5 ms -> 310 ms +- 7 ms: 1.07x slower > - scimark_sparse_mat_mult: 8.19 ms +- 0.22 ms -> 8.74 ms +- 0.15 ms: > 1.07x slower > - xml_etree_generate: 302 ms +- 3 ms -> 320 ms +- 8 ms: 1.06x slower > > Faster (15): > - telco: 707 ms +- 22 ms -> 22.1 ms +- 0.4 ms: 32.04x faster > - unpickle_list: 15.0 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.90x faster > - pickle_list: 14.7 us +- 0.2 us -> 9.12 us +- 0.38 us: 1.61x faster > - json_loads: 98.7 us +- 2.3 us -> 62.3 us +- 0.7 us: 1.58x faster > - pickle: 40.4 us +- 0.6 us -> 27.1 us +- 0.5 us: 1.49x faster > - sympy_sum: 361 ms +- 10 ms -> 244 ms +- 7 ms: 1.48x faster > - sympy_expand: 1.68 sec +- 0.02 sec -> 1.15 sec +- 0.03 sec: 1.47x faster > - regex_v8: 62.0 ms +- 0.5 ms -> 47.2 ms +- 0.6 ms: 1.31x faster > - sympy_str: 699 ms +- 22 ms -> 537 ms +- 15 ms: 1.30x faster > - regex_effbot: 6.67 ms +- 0.04 ms -> 5.23 ms +- 0.05 ms: 1.28x faster > - mako: 61.5 ms +- 0.7 ms -> 49.7 ms +- 2.5 ms: 1.24x faster > - html5lib: 298 ms +- 7 ms -> 261 ms +- 6 ms: 1.14x faster > - sympy_integrate: 55.9 ms +- 0.3 ms -> 51.8 ms +- 1.0 ms: 1.08x faster > - pickle_dict: 69.4 us +- 0.9 us -> 65.2 us +- 3.2 us: 1.06x faster > - scimark_lu: 551 ms +- 26 ms -> 523 ms +- 18 ms: 1.05x faster > > Benchmark hidden because not significant (8): 2to3, dulwich_log, > nbody, pidigits, regex_dna, tornado_http, unpack_sequence, unpickle > Ignored benchmarks (3) of 2016-11-03_15-36-2.7-91f024fc9b3a.json: > hg_startup, pyflate, spambayes > ------------------- > > Please ignore call_method, call_method_slots, call_method_unknown > benchmarks: it seems like I had an issue on the benchmark server. I > was unable to reproduce he 70% slowdown on my laptop. > > I attached the two compressed JSON files to this email if you want to > analyze them yourself. > > I hope that my work on benchmarks will motive some developers to look > closer at Python 3 performance to find interesting optimizations ;-) > > Victor > _______________________________________________ > Speed mailing list > Speed at python.org > https://mail.python.org/mailman/listinfo/speed > -------------- next part -------------- An HTML attachment was scrubbed... URL: From franciscouzo at gmail.com Fri Nov 4 15:35:17 2016 From: franciscouzo at gmail.com (Francisco Couzo) Date: Fri, 4 Nov 2016 16:35:17 -0300 Subject: [Python-Dev] itertools predicates In-Reply-To: <449F716E-B602-4FAA-A6DE-61E7813FF060@gmail.com> References: <20161103225104.GS3365@ando.pearwood.info> <449F716E-B602-4FAA-A6DE-61E7813FF060@gmail.com> Message-ID: Well sorry about it, I didn't know about that PEP 8 line, disregard this thread then :) On Fri, Nov 4, 2016 at 2:29 PM, Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > > > On Nov 3, 2016, at 3:51 PM, Steven D'Aprano wrote: > > > > Before there can be a solution, there first has to be a problem that > > needs solving. "Lack of consistency" is not necessarily a problem. The > > intertools functions are quite different, they do different things with > > different APIs. The question here should not be "why don't these > > functions take None as an argument?", rather it should be "why should > > these functions take None as an argument?". > > I concur with Steven who articulated the issue perfectly. There isn't an > actual problem here that needs to be solved (i.e. not a single user report > in 15 years indicating that the API wasn't meeting the needs for real > use-cases). I'm disinclined to churn the API unless there is a real need. > > FWIW, groupby() has the predicate as an optional argument so that you can > write groupby('aaabbc') and have it group by value (much like the > key-function on sorted() is optional). The two filter variants allow None > as the first argument only for historical reasons -- once "bool" came > along, it would have been better to write filter(bool, someiterable) in > preference to using None which is less clear about its intention. The > takewhile/dropwhile tools didn't have the same constraint to match a > historical API, so there was an opportunity to have a clearer API with a > simpler signature. > > As Terry suggested, if you have other itertools feature requests, please > put them on the tracker an assign them to me. > > Thank you, > > > Raymond > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > franciscouzo%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Nov 4 16:56:21 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 4 Nov 2016 21:56:21 +0100 Subject: [Python-Dev] [Speed] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance In-Reply-To: References: Message-ID: 2016-11-04 20:18 GMT+01:00 Miquel Torres : > Nice! For the record, I'll be giving a talk in PyCon Ireland about > Codespeed. Would you mind me citing those tweets and screenshots, to > highlight usage on speed.python.org? Sure. Keep me in touch in you publish your slides later. > You mentioned new more reliable vs old results. How close are we to have an > stable setup that gives us benchmarks numbers regularly? My plan for the short term is to analyze last (latest?) benchmarks hiccups and try to fix them. The fully automated script to run benchmarks is already written: https://github.com/python/performance/tree/master/scripts Then, the plan we decided with Zachary Ware is to run a script in a loop which compiles the default branch of CPython. Later, we may also do the same for 2.7 and 3.6 branches. And then add PyPy (and PyPy 3). Victor From solipsis at pitrou.net Sat Nov 5 05:56:04 2016 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 5 Nov 2016 10:56:04 +0100 Subject: [Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance References: Message-ID: <20161105105604.4a7e52c5@fsol> Hi Victor, On Fri, 4 Nov 2016 13:53:10 +0100 Victor Stinner wrote: > > Raw results of Python 3.6 compared to Python 2.7: That's interesting, but I would be personally more interested in a performance comparison of 3.5 and 3.6, to know if anything interesting (or worrying :-)) has happened there. The performance differences between 2.7 and 3.x are quite well-known by now, and none of them are really dramatic except for the increase in startup time. Regards Antoine. From ncoghlan at gmail.com Sat Nov 5 08:36:06 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 5 Nov 2016 22:36:06 +1000 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> <3b718ea3-f416-057a-d8e1-634464dc7770@trueblade.com> Message-ID: On 5 November 2016 at 04:03, Fabio Zadrozny wrote: > On Fri, Nov 4, 2016 at 3:15 PM, Eric V. Smith wrote: >> Using PyParser_ASTFromString is the easiest possible way to do this. Given >> a string, it returns an AST node. What could be simpler? > > > I think that for implementation purposes, given the python infrastructure, > it's fine, but for specification purposes, probably incorrect... As I don't > think f-strings should accept: > > f"start {import sys; sys.version_info[0];} end" (i.e.: > PyParser_ASTFromString doesn't just return an expression, it accepts any > valid Python code, even code which can't be used in an f-string). f-strings use the "eval" parsing mode, which starts from the "eval_input" node in the grammar (which is only a couple of nodes higher than 'test', allowing tuples via 'testlist' as well as trailing newlines and EOF): >>> ast.parse("import sys; sys.version_info[0];", mode="eval") Traceback (most recent call last): File "", line 1, in File "/usr/lib64/python3.5/ast.py", line 35, in parse return compile(source, filename, mode, PyCF_ONLY_AST) File "", line 1 import sys; sys.version_info[0]; ^ SyntaxError: invalid syntax You have to use "exec" mode to get the parser to allow statements, which is why f-strings don't do that: >>> ast.dump(ast.parse("import sys; sys.version_info[0];", mode="exec")) "Module(body=[Import(names=[alias(name='sys', asname=None)]), Expr(value=Subscript(value=Attribute(value=Name(id='sys', ctx=Load()), attr='version_info', ctx=Load()), slice=Index(value=Num(n=0)), ctx=Load()))])" The unique aspect for f-strings that means they don't permit some otherwise valid Python expressions is that it also does the initial pre-tokenisation based on: 1. Look for an opening '{' 2. Look for a closing '!', ':' or '}' accounting for balanced string quotes, parentheses, brackets and braces Ignoring the surrounding quotes, and using the `atom` node from Python's grammar to represent the nesting tracking, and TEXT to stand in for arbitrary text, it's something akin to: fstring: (TEXT ['{' maybe_pyexpr ('!' | ':' | '}')])+ maybe_pyexpr: (atom | TEXT)+ That isn't quite right, since it doesn't properly account for brace nesting, but it gives the general idea - there's an initial really simple tokenising pass that picks out the potential Python expressions, and then those are run through the AST parser's equivalent of eval(). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From wolfgang.maier at biologie.uni-freiburg.de Mon Nov 7 10:11:55 2016 From: wolfgang.maier at biologie.uni-freiburg.de (Wolfgang Maier) Date: Mon, 7 Nov 2016 16:11:55 +0100 Subject: [Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance In-Reply-To: <20161105105604.4a7e52c5@fsol> References: <20161105105604.4a7e52c5@fsol> Message-ID: On 05.11.2016 10:56, Antoine Pitrou wrote: > > Hi Victor, > > On Fri, 4 Nov 2016 13:53:10 +0100 > Victor Stinner wrote: >> >> Raw results of Python 3.6 compared to Python 2.7: > > That's interesting, but I would be personally more interested in > a performance comparison of 3.5 and 3.6, to know if anything > interesting (or worrying :-)) has happened there. > You can get this as well from https://speed.python.org/comparison/ and https://speed.python.org/timeline and looking at this, I think there is something worrying indeed: Startup time has increased by ~ 30 % between 3.5 and 3.6 again. More specifically, all this increase happened between Sep 09 and Sep 15. I have no clue why that is, but it is definitely the biggest effect far and wide. > The performance differences between 2.7 and 3.x are quite well-known by > now, and none of them are really dramatic except for the increase in > startup time. > > Regards > > Antoine. > > From guido at python.org Mon Nov 7 14:20:57 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 7 Nov 2016 11:20:57 -0800 Subject: [Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance In-Reply-To: References: <20161105105604.4a7e52c5@fsol> Message-ID: Re: https://speed.python.org/timeline/#/?exe=4&ben=python_startup&env=1&revs=50&equid=off&quarts=on&extr=on That's suspiciously close to the core sprint. Since the -S time stayed roughly the same I suspect that either a new module was added to the startup sequence or one of the (too many) modules already involved grew a lot. My money is on a new module. Using `python -v -c pass` on a Python built from the Sept. 9 tree and one built from Sept. 15, it shouldn't be too hard to figure out which new module(s). On Mon, Nov 7, 2016 at 7:11 AM, Wolfgang Maier < wolfgang.maier at biologie.uni-freiburg.de> wrote: > On 05.11.2016 10:56, Antoine Pitrou wrote: > >> >> Hi Victor, >> >> On Fri, 4 Nov 2016 13:53:10 +0100 >> Victor Stinner wrote: >> >>> >>> Raw results of Python 3.6 compared to Python 2.7: >>> >> >> That's interesting, but I would be personally more interested in >> a performance comparison of 3.5 and 3.6, to know if anything >> interesting (or worrying :-)) has happened there. >> >> > You can get this as well from https://speed.python.org/comparison/ > and https://speed.python.org/timeline and looking at this, I think there > is something worrying indeed: > Startup time has increased by ~ 30 % between 3.5 and 3.6 again. More > specifically, all this increase happened between Sep 09 and Sep 15. > > I have no clue why that is, but it is definitely the biggest effect far > and wide. > > > The performance differences between 2.7 and 3.x are quite well-known by >> now, and none of them are really dramatic except for the increase in >> startup time. >> >> Regards >> >> Antoine. >> >> >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% > 40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Mon Nov 7 14:24:34 2016 From: steve.dower at python.org (Steve Dower) Date: Mon, 7 Nov 2016 11:24:34 -0800 Subject: [Python-Dev] Installing Python on Windows Message-ID: Hi all Those of you who follow me on Twitter (@zooba) may have noticed that I posted one of my rare blog posts over the weekend about the increasing range of Python installers available for Windows. I figured I'd draw some attention here in case others are interested in my rationale for why the main installer is how it is, and why I'm also experimenting with other forms of installer or package: http://stevedower.id.au/blog/why-so-many-python-installers/ Arguably this would make a good section in the docs or a PEP, though it's very point-in-time content, so a blog seems fine for now. There's also not really anything in the docs about how Linux distributors package up CPython, so maybe it's best as entirely standalone content. Cheers, Steve (In case anyone is suspicious, I earn no income from my blog and there are no ads. So I hope I'm not promoting the link for selfish reasons :) ) From victor.stinner at gmail.com Mon Nov 7 18:11:31 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 8 Nov 2016 00:11:31 +0100 Subject: [Python-Dev] Benchmarks: Comparison between Python 2.7 and Python 3.6 performance In-Reply-To: References: <20161105105604.4a7e52c5@fsol> Message-ID: 2016-11-07 20:20 GMT+01:00 Guido van Rossum : > Re: > https://speed.python.org/timeline/#/?exe=4&ben=python_startup&env=1&revs=50&equid=off&quarts=on&extr=on > > That's suspiciously close to the core sprint. Since the -S time stayed > roughly the same I suspect that either a new module was added to the startup > sequence or one of the (too many) modules already involved grew a lot. My > money is on a new module. Using `python -v -c pass` on a Python built from > the Sept. 9 tree and one built from Sept. 15, it shouldn't be too hard to > figure out which new module(s). I identified the regression, I created: http://bugs.python.org/issue28637 It's a regression caused by: https://hg.python.org/cpython/rev/223731925d06/ http://bugs.python.org/issue28082 The change added "import enum" in Lib/re.py. I propose to revert this change for now, and discuss later a solution which doesn't impact performances. It's nice to see that my work of performance was useful to catch a performance regression. Victor From victor.stinner at gmail.com Mon Nov 7 18:25:49 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 8 Nov 2016 00:25:49 +0100 Subject: [Python-Dev] Benchmarks: Comparison between Python 3.5 and Python 3.6 performance Message-ID: Hi, Antoine Pitrou asked me to compare Python 3.6 to Python 3.5. Here are results on the speed-python server using LTO compilation (but not PGO): $ python3 -m perf compare_to 2016-11-03_15-37-3.5-89f7386104e2.json 2016-11-03_15-38-3.6-c4319c0d0131.json -G --min-speed=5 Slower (17): - call_method_slots: 16.7 ms +- 0.2 ms -> 28.3 ms +- 0.7 ms: 1.70x slower - call_method: 16.9 ms +- 0.2 ms -> 28.6 ms +- 0.8 ms: 1.69x slower - call_method_unknown: 18.5 ms +- 0.2 ms -> 30.8 ms +- 0.8 ms: 1.67x slower - python_startup: 20.3 ms +- 0.7 ms -> 26.9 ms +- 0.6 ms: 1.33x slower - regex_compile: 397 ms +- 4 ms -> 482 ms +- 6 ms: 1.21x slower - mako: 42.2 ms +- 0.5 ms -> 49.7 ms +- 2.5 ms: 1.18x slower - deltablue: 17.8 ms +- 0.2 ms -> 20.1 ms +- 0.6 ms: 1.13x slower - chameleon: 29.1 ms +- 0.4 ms -> 31.9 ms +- 0.4 ms: 1.10x slower - genshi_text: 89.8 ms +- 4.2 ms -> 97.8 ms +- 1.1 ms: 1.09x slower - pickle_pure_python: 1.30 ms +- 0.02 ms -> 1.41 ms +- 0.03 ms: 1.08x slower - logging_simple: 35.8 us +- 0.7 us -> 38.4 us +- 0.7 us: 1.07x slower - sqlalchemy_declarative: 331 ms +- 3 ms -> 354 ms +- 6 ms: 1.07x slower - logging_format: 42.6 us +- 0.5 us -> 45.5 us +- 0.8 us: 1.07x slower - call_simple: 13.1 ms +- 0.4 ms -> 13.9 ms +- 0.2 ms: 1.06x slower - genshi_xml: 197 ms +- 2 ms -> 209 ms +- 2 ms: 1.06x slower - richards: 190 ms +- 5 ms -> 201 ms +- 6 ms: 1.06x slower - go: 607 ms +- 19 ms -> 640 ms +- 26 ms: 1.05x slower Faster (15): - xml_etree_iterparse: 498 ms +- 11 ms -> 230 ms +- 5 ms: 2.17x faster - unpickle_list: 10.6 us +- 0.3 us -> 7.86 us +- 0.16 us: 1.35x faster - xml_etree_parse: 345 ms +- 6 ms -> 298 ms +- 8 ms: 1.16x faster - xml_etree_generate: 362 ms +- 5 ms -> 320 ms +- 8 ms: 1.13x faster - scimark_lu: 589 ms +- 24 ms -> 523 ms +- 18 ms: 1.13x faster - regex_effbot: 5.88 ms +- 0.06 ms -> 5.23 ms +- 0.05 ms: 1.13x faster - sympy_sum: 269 ms +- 7 ms -> 244 ms +- 7 ms: 1.10x faster - spectral_norm: 317 ms +- 6 ms -> 287 ms +- 2 ms: 1.10x faster - sympy_expand: 1.25 sec +- 0.04 sec -> 1.15 sec +- 0.03 sec: 1.09x faster - xml_etree_process: 291 ms +- 5 ms -> 268 ms +- 14 ms: 1.09x faster - float: 336 ms +- 8 ms -> 310 ms +- 7 ms: 1.08x faster - regex_dna: 323 ms +- 3 ms -> 298 ms +- 3 ms: 1.08x faster - unpickle: 38.4 us +- 0.6 us -> 35.5 us +- 1.2 us: 1.08x faster - crypto_pyaes: 267 ms +- 3 ms -> 249 ms +- 2 ms: 1.07x faster - nbody: 262 ms +- 3 ms -> 246 ms +- 3 ms: 1.07x faster Benchmark hidden because not significant (31): 2to3, chaos, (...) --- call_method_* slowdown: http://bugs.python.org/issue28618 python_startup slowdown: http://bugs.python.org/issue28637 I didn't analyze results yet. Victor From yselivanov.ml at gmail.com Wed Nov 9 10:16:38 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 9 Nov 2016 10:16:38 -0500 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (b78574cb00ab): sum=1120 In-Reply-To: <20161109090247.50140.35000.CEB8D404@psf.io> References: <20161109090247.50140.35000.CEB8D404@psf.io> Message-ID: I'm trying to fix refleaks in 3.6. So far: On 2016-11-09 4:02 AM, solipsis at pitrou.net wrote: > results for b78574cb00ab on branch "default" > -------------------------------------------- > > test_ast leaked [98, 98, 98] references, sum=294 > test_ast leaked [98, 98, 98] memory blocks, sum=294 > test_asyncio leaked [3, 0, 0] memory blocks, sum=3 > test_code leaked [2, 2, 2] references, sum=6 > test_code leaked [2, 2, 2] memory blocks, sum=6 > test_functools leaked [0, 3, 1] memory blocks, sum=4 > test_pydoc leaked [106, 106, 106] references, sum=318 > test_pydoc leaked [42, 42, 42] memory blocks, sum=126 > test_trace leaked [12, 12, 12] references, sum=36 > test_trace leaked [11, 11, 11] memory blocks, sum=33 > > test_ast, test_code and test_trace were fixed by https://hg.python.org/cpython/rev/2c6825c9ecfd test_pydoc leaks in test_typing_pydoc. I tried git bisect and it looks like that the first commit that introduced the refleak was the one that added test_typing_pydoc! 62127e60e7b0 doesn't modify any CPython internals, so it looks like that test_typing_pydoc exposed some bug that has existed before it. Any help tracking that down is welcome :) Yury From yselivanov.ml at gmail.com Wed Nov 9 11:01:08 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 9 Nov 2016 11:01:08 -0500 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (b78574cb00ab): sum=1120 In-Reply-To: References: <20161109090247.50140.35000.CEB8D404@psf.io> Message-ID: On 2016-11-09 10:16 AM, Yury Selivanov wrote: > I'm trying to fix refleaks in 3.6. So far: > > On 2016-11-09 4:02 AM, solipsis at pitrou.net wrote: > >> results for b78574cb00ab on branch "default" >> -------------------------------------------- >> >> test_ast leaked [98, 98, 98] references, sum=294 >> test_ast leaked [98, 98, 98] memory blocks, sum=294 >> test_asyncio leaked [3, 0, 0] memory blocks, sum=3 >> test_code leaked [2, 2, 2] references, sum=6 >> test_code leaked [2, 2, 2] memory blocks, sum=6 >> test_functools leaked [0, 3, 1] memory blocks, sum=4 >> test_pydoc leaked [106, 106, 106] references, sum=318 >> test_pydoc leaked [42, 42, 42] memory blocks, sum=126 >> test_trace leaked [12, 12, 12] references, sum=36 >> test_trace leaked [11, 11, 11] memory blocks, sum=33 >> >> > > test_ast, test_code and test_trace were fixed by > https://hg.python.org/cpython/rev/2c6825c9ecfd > > test_pydoc leaks in test_typing_pydoc. I tried git bisect and it looks > like that the first commit that introduced the refleak was the one > that added test_typing_pydoc! > > 62127e60e7b0 doesn't modify any CPython internals, so it looks like > that test_typing_pydoc exposed some bug that has existed before it. > Any help tracking that down is welcome :) I've created an issue to track the leak in pydocs/typing: http://bugs.python.org/issue28649 Yury From fabiofz at gmail.com Wed Nov 9 11:20:27 2016 From: fabiofz at gmail.com (Fabio Zadrozny) Date: Wed, 9 Nov 2016 14:20:27 -0200 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> <3b718ea3-f416-057a-d8e1-634464dc7770@trueblade.com> Message-ID: On Sat, Nov 5, 2016 at 10:36 AM, Nick Coghlan wrote: > On 5 November 2016 at 04:03, Fabio Zadrozny wrote: > > On Fri, Nov 4, 2016 at 3:15 PM, Eric V. Smith > wrote: > >> Using PyParser_ASTFromString is the easiest possible way to do this. > Given > >> a string, it returns an AST node. What could be simpler? > > > > > > I think that for implementation purposes, given the python > infrastructure, > > it's fine, but for specification purposes, probably incorrect... As I > don't > > think f-strings should accept: > > > > f"start {import sys; sys.version_info[0];} end" (i.e.: > > PyParser_ASTFromString doesn't just return an expression, it accepts any > > valid Python code, even code which can't be used in an f-string). > > f-strings use the "eval" parsing mode, which starts from the > "eval_input" node in the grammar (which is only a couple of nodes > higher than 'test', allowing tuples via 'testlist' as well as trailing > newlines and EOF): > > >>> ast.parse("import sys; sys.version_info[0];", mode="eval") > Traceback (most recent call last): > File "", line 1, in > File "/usr/lib64/python3.5/ast.py", line 35, in parse > return compile(source, filename, mode, PyCF_ONLY_AST) > File "", line 1 > import sys; sys.version_info[0]; > ^ > SyntaxError: invalid syntax > > You have to use "exec" mode to get the parser to allow statements, > which is why f-strings don't do that: > > >>> ast.dump(ast.parse("import sys; sys.version_info[0];", > mode="exec")) > "Module(body=[Import(names=[alias(name='sys', asname=None)]), > Expr(value=Subscript(value=Attribute(value=Name(id='sys', ctx=Load()), > attr='version_info', ctx=Load()), slice=Index(value=Num(n=0)), > ctx=Load()))])" > > The unique aspect for f-strings that means they don't permit some > otherwise valid Python expressions is that it also does the initial > pre-tokenisation based on: > > 1. Look for an opening '{' > 2. Look for a closing '!', ':' or '}' accounting for balanced string > quotes, parentheses, brackets and braces > > Ignoring the surrounding quotes, and using the `atom` node from > Python's grammar to represent the nesting tracking, and TEXT to stand > in for arbitrary text, it's something akin to: > > fstring: (TEXT ['{' maybe_pyexpr ('!' | ':' | '}')])+ > maybe_pyexpr: (atom | TEXT)+ > > That isn't quite right, since it doesn't properly account for brace > nesting, but it gives the general idea - there's an initial really > simple tokenising pass that picks out the potential Python > expressions, and then those are run through the AST parser's > equivalent of eval(). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > ?Hi Nick and Eric, Just wanted to say thanks for the feedback and point to a grammar I ended up doing on my side (in JavaCC), just in case someone else decides to do a formal grammar later on it can probably be used as a reference (shouldn't be hard to convert it to a bnf grammar): https://github.com/fabioz/Pydev/blob/master/plugins/org.python.pydev.parser/src/org/python/pydev/parser/grammar_fstrings/grammar_fstrings.jjt ? Also, as a feedback, I found it a bit odd that there can't be any space nor new line between the last format specifiers and '}' I.e.: f'''{ dict( a = 10 ) !r } ''' ?is not valid, whereas ? ? f'''{ dict( a = 10 ) !r} '''? is valid -- as a note, this means my grammar has a bug as both versions are accepted -- and I currently don't care enough about that change from the implementation to fix it ;) Cheers, Fabio? -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Nov 9 11:35:24 2016 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 9 Nov 2016 16:35:24 +0000 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> <3b718ea3-f416-057a-d8e1-634464dc7770@trueblade.com> Message-ID: On 9 November 2016 at 16:20, Fabio Zadrozny wrote: > Also, as a feedback, I found it a bit odd that there can't be any space nor > new line between the last format specifiers and '}' FWIW, that is the case for normal format strings, as well: >>> print("{!r\n}".format(12)) Traceback (most recent call last): File "", line 1, in ValueError: expected ':' after conversion specifier I guess the behaviour is simply inherited from there. Paul From eric at trueblade.com Wed Nov 9 17:14:37 2016 From: eric at trueblade.com (Eric V. Smith) Date: Wed, 9 Nov 2016 17:14:37 -0500 Subject: [Python-Dev] Parsing f-strings from PEP 498 -- Literal String Interpolation In-Reply-To: References: <63659b4e-475e-2ae6-caaf-5773cbc348c5@trueblade.com> <3b718ea3-f416-057a-d8e1-634464dc7770@trueblade.com> Message-ID: <4b55b05c-54a5-2d37-0980-380b8222b929@trueblade.com> On 11/9/2016 11:35 AM, Paul Moore wrote: > On 9 November 2016 at 16:20, Fabio Zadrozny wrote: >> Also, as a feedback, I found it a bit odd that there can't be any space nor >> new line between the last format specifiers and '}' > > FWIW, that is the case for normal format strings, as well: > >>>> print("{!r\n}".format(12)) > Traceback (most recent call last): > File "", line 1, in > ValueError: expected ':' after conversion specifier > > I guess the behaviour is simply inherited from there. Right. That and the fact that whitespace is significant inside the format specifier portion: >>> '{:%H:%M:%S\n}'.format(datetime.datetime.now()) '17:14:10\n' I don't think it's worth changing this to allow whitespace after the optional one-character conversion flag. Eric. From status at bugs.python.org Fri Nov 11 12:08:50 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 11 Nov 2016 18:08:50 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20161111170850.363F556A7F@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-11-04 - 2016-11-11) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5559 (+12) closed 34866 (+45) total 40425 (+57) Open issues with patches: 2418 Issues opened (37) ================== #4347: Circular dependency causes SystemError when adding new syntax http://bugs.python.org/issue4347 reopened by martin.panter #28614: Slice limit documentation is incorrect http://bugs.python.org/issue28614 opened by airwin #28615: Document clarification: Section 5.4 Complex Number http://bugs.python.org/issue28615 opened by David.Johnston #28617: Why isn't "in" called a comparison operation? http://bugs.python.org/issue28617 opened by wim.glenn #28618: Decorate hot functions using __attribute__((hot)) to optimize http://bugs.python.org/issue28618 opened by haypo #28619: [Patch] Stop using inet_ntoa() when possible. http://bugs.python.org/issue28619 opened by EdSchouten #28624: Make the `cwd` argument to `subprocess.Popen` accept a `PathLi http://bugs.python.org/issue28624 opened by cool-RR #28625: multiprocessing.Pool.imap swallows exceptions thrown by genera http://bugs.python.org/issue28625 opened by elias #28626: Tutorial: rearrange discussion of output formatting to encoura http://bugs.python.org/issue28626 opened by akuchling #28627: [alpine] shutil.copytree fail to copy a direcotry with broken http://bugs.python.org/issue28627 opened by fruch #28628: Failure to add signal handlers for any signal but SIGINT http://bugs.python.org/issue28628 opened by brotherBox #28629: Emit ResourceWarning when implicitly terminating a suspended f http://bugs.python.org/issue28629 opened by ncoghlan #28631: [2.7/3.5/3.6 Regression] crash using ctypes http://bugs.python.org/issue28631 opened by doko #28632: configparser does not close files in read http://bugs.python.org/issue28632 opened by PetrPy #28635: Update What's New for 3.6 http://bugs.python.org/issue28635 opened by Elvis.Pranskevichus #28637: Python startup performance regression http://bugs.python.org/issue28637 opened by haypo #28643: Broken makefile depends for profile-opt target http://bugs.python.org/issue28643 opened by nascheme #28644: Document recent changes in typing.py http://bugs.python.org/issue28644 opened by levkivskyi #28645: Drop __aiter__ compatibility layer from 3.7 http://bugs.python.org/issue28645 opened by yselivanov #28646: Using a read-only buffer. http://bugs.python.org/issue28646 opened by mrolle #28647: python --help: -u is misdocumented as binary mode http://bugs.python.org/issue28647 opened by arigo #28648: False assert in _Py_DecodeUTF8_surrogateescape http://bugs.python.org/issue28648 opened by xiang.zhang #28650: concurrent.futures.ThreadPoolExecutor: tasks in queue not mark http://bugs.python.org/issue28650 opened by pyfm #28651: Make objects with empty __slots__ GC types http://bugs.python.org/issue28651 opened by yselivanov #28654: sys.stdout.isatty() returns True even if redirected to NUL http://bugs.python.org/issue28654 opened by kmeyer #28655: Tests altered the execution environment in isolated mode http://bugs.python.org/issue28655 opened by serhiy.storchaka #28657: cmd.Cmd.get_help() implementation can't see do_*() methods add http://bugs.python.org/issue28657 opened by B??a??ej Michalik #28658: MacOsX idle don't run http://bugs.python.org/issue28658 opened by Andrew Kontokanis #28660: TextWrapper break_long_words=True, break_on_hyphens=True on lo http://bugs.python.org/issue28660 opened by maubp #28661: Fix code example in Python 3.5 telnetlib documentation http://bugs.python.org/issue28661 opened by tiabc #28662: catch also PermissionError in tests when spawning a non existe http://bugs.python.org/issue28662 opened by xdegaye #28663: Higher virtual memory usage on recent Linux versions http://bugs.python.org/issue28663 opened by Valentin.Lorentz #28665: Harmonize STORE_DEREF with STORE_FAST and LOAD_DEREF http://bugs.python.org/issue28665 opened by rhettinger #28666: Make test.support.rmtree() able to remove non-writable directo http://bugs.python.org/issue28666 opened by serhiy.storchaka #28667: FD_SETSIZE is unsigned on FreeBSD http://bugs.python.org/issue28667 opened by serhiy.storchaka #28668: instanciation of multiprocessing.Queue raises ImportError in t http://bugs.python.org/issue28668 opened by xdegaye #28669: Math Library Dos Attack http://bugs.python.org/issue28669 opened by Stone Most recent 15 issues with no replies (15) ========================================== #28667: FD_SETSIZE is unsigned on FreeBSD http://bugs.python.org/issue28667 #28666: Make test.support.rmtree() able to remove non-writable directo http://bugs.python.org/issue28666 #28663: Higher virtual memory usage on recent Linux versions http://bugs.python.org/issue28663 #28660: TextWrapper break_long_words=True, break_on_hyphens=True on lo http://bugs.python.org/issue28660 #28658: MacOsX idle don't run http://bugs.python.org/issue28658 #28657: cmd.Cmd.get_help() implementation can't see do_*() methods add http://bugs.python.org/issue28657 #28655: Tests altered the execution environment in isolated mode http://bugs.python.org/issue28655 #28650: concurrent.futures.ThreadPoolExecutor: tasks in queue not mark http://bugs.python.org/issue28650 #28648: False assert in _Py_DecodeUTF8_surrogateescape http://bugs.python.org/issue28648 #28646: Using a read-only buffer. http://bugs.python.org/issue28646 #28625: multiprocessing.Pool.imap swallows exceptions thrown by genera http://bugs.python.org/issue28625 #28624: Make the `cwd` argument to `subprocess.Popen` accept a `PathLi http://bugs.python.org/issue28624 #28619: [Patch] Stop using inet_ntoa() when possible. http://bugs.python.org/issue28619 #28608: Support creating hardlink using `pathlib` http://bugs.python.org/issue28608 #28596: on Android _bootlocale on startup relies on too many library m http://bugs.python.org/issue28596 Most recent 15 issues waiting for review (15) ============================================= #28668: instanciation of multiprocessing.Queue raises ImportError in t http://bugs.python.org/issue28668 #28667: FD_SETSIZE is unsigned on FreeBSD http://bugs.python.org/issue28667 #28666: Make test.support.rmtree() able to remove non-writable directo http://bugs.python.org/issue28666 #28665: Harmonize STORE_DEREF with STORE_FAST and LOAD_DEREF http://bugs.python.org/issue28665 #28662: catch also PermissionError in tests when spawning a non existe http://bugs.python.org/issue28662 #28651: Make objects with empty __slots__ GC types http://bugs.python.org/issue28651 #28650: concurrent.futures.ThreadPoolExecutor: tasks in queue not mark http://bugs.python.org/issue28650 #28648: False assert in _Py_DecodeUTF8_surrogateescape http://bugs.python.org/issue28648 #28645: Drop __aiter__ compatibility layer from 3.7 http://bugs.python.org/issue28645 #28644: Document recent changes in typing.py http://bugs.python.org/issue28644 #28643: Broken makefile depends for profile-opt target http://bugs.python.org/issue28643 #28637: Python startup performance regression http://bugs.python.org/issue28637 #28635: Update What's New for 3.6 http://bugs.python.org/issue28635 #28626: Tutorial: rearrange discussion of output formatting to encoura http://bugs.python.org/issue28626 #28619: [Patch] Stop using inet_ntoa() when possible. http://bugs.python.org/issue28619 Top 10 most discussed issues (10) ================================= #28637: Python startup performance regression http://bugs.python.org/issue28637 32 msgs #28635: Update What's New for 3.6 http://bugs.python.org/issue28635 15 msgs #28618: Decorate hot functions using __attribute__((hot)) to optimize http://bugs.python.org/issue28618 11 msgs #19569: Use __attribute__((deprecated)) to warn usage of deprecated fu http://bugs.python.org/issue19569 9 msgs #24329: __qualname__ and __slots__ http://bugs.python.org/issue24329 9 msgs #26934: android: test_faulthandler fails http://bugs.python.org/issue26934 8 msgs #28609: argparse claims '*' positional argument is required in error o http://bugs.python.org/issue28609 8 msgs #19717: resolve() fails when the path doesn't exist http://bugs.python.org/issue19717 7 msgs #27854: Installed 2.7: IDLE Help disabled because help.html is missing http://bugs.python.org/issue27854 7 msgs #28617: Why isn't "in" called a comparison operation? http://bugs.python.org/issue28617 7 msgs Issues closed (42) ================== #1778: SyntaxError.offset sometimes wrong http://bugs.python.org/issue1778 closed by akuchling #4713: Installing sgmlop causes xmlrpclib error http://bugs.python.org/issue4713 closed by akuchling #10605: ElementTree documentation http://bugs.python.org/issue10605 closed by akuchling #18317: gettext: DoS via crafted Plural-Forms http://bugs.python.org/issue18317 closed by serhiy.storchaka #19398: test_trace fails with -S http://bugs.python.org/issue19398 closed by serhiy.storchaka #20629: Python ctypes BigEndianStructure bitfield assignment misbehavi http://bugs.python.org/issue20629 closed by xiang.zhang #21590: Systemtap and DTrace support http://bugs.python.org/issue21590 closed by lukasz.langa #21864: Error in documentation of point 9.8 'Exceptions are classes to http://bugs.python.org/issue21864 closed by berker.peksag #23839: Clear caches after every test http://bugs.python.org/issue23839 closed by serhiy.storchaka #26969: ascynio should provide a policy to address pass-loop-everywher http://bugs.python.org/issue26969 closed by yselivanov #27781: Change sys.getfilesystemencoding() on Windows to UTF-8 http://bugs.python.org/issue27781 closed by python-dev #28123: _PyDict_GetItem_KnownHash ignores DKIX_ERROR return http://bugs.python.org/issue28123 closed by serhiy.storchaka #28485: compileall.compile_dir(workers=) does not raise Valu http://bugs.python.org/issue28485 closed by martin.panter #28543: Incomplete fast path codecs aliases in codecs doc http://bugs.python.org/issue28543 closed by berker.peksag #28555: provid also sha-1 and sha-256 also on download links http://bugs.python.org/issue28555 closed by benjamin.peterson #28563: Arbitrary code execution in gettext.c2py http://bugs.python.org/issue28563 closed by serhiy.storchaka #28574: Update bundled pip http://bugs.python.org/issue28574 closed by berker.peksag #28585: Restore docstring of os._isdir http://bugs.python.org/issue28585 closed by serhiy.storchaka #28586: Convert os.scandir to Argument Clinic http://bugs.python.org/issue28586 closed by serhiy.storchaka #28590: fstring's '{' from escape sequences does not start an expressi http://bugs.python.org/issue28590 closed by eric.smith #28593: Inconsistent itertools' predicate behaviour http://bugs.python.org/issue28593 closed by rhettinger #28613: Make get_event_loop() return the current loop if called from c http://bugs.python.org/issue28613 closed by yselivanov #28616: sys.version_info.releaselevel - 'final' or 'release' http://bugs.python.org/issue28616 closed by ned.deily #28620: Build Memory Leak http://bugs.python.org/issue28620 closed by r.david.murray #28621: Refactor duplicate code calculating digit's bit length http://bugs.python.org/issue28621 closed by serhiy.storchaka #28622: Remove redundant variable annotation test from test_grammar http://bugs.python.org/issue28622 closed by gvanrossum #28623: Let `shlex.quote` accept a `PathLike` object http://bugs.python.org/issue28623 closed by berker.peksag #28630: setup.py bdist_egg --without-source-files does not allow conso http://bugs.python.org/issue28630 closed by berker.peksag #28633: Concatenating bytes literal and f-string causes segmentation f http://bugs.python.org/issue28633 closed by eric.smith #28634: asyncio.isfuture() should support Mocks http://bugs.python.org/issue28634 closed by yselivanov #28636: strange issue with Pandas-0.19.1 on Python-3.6.0b3 http://bugs.python.org/issue28636 closed by Big Stone #28638: Creating namedtuple is too slow to be used in common stdlib (e http://bugs.python.org/issue28638 closed by rhettinger #28639: inspect.isawaitable(native_coro) returns int http://bugs.python.org/issue28639 closed by yselivanov #28640: UnicodeDecodeError: 'cp949' http://bugs.python.org/issue28640 closed by steve.dower #28641: Describe PEP 495 features in "What's New in Python 3.6" docume http://bugs.python.org/issue28641 closed by yselivanov #28642: csv reader losing rows with big files and tab delimiter http://bugs.python.org/issue28642 closed by datapythonista #28649: refleak in typing.py http://bugs.python.org/issue28649 closed by yselivanov #28652: Make loop methods reject socket kinds they do not support. http://bugs.python.org/issue28652 closed by yselivanov #28653: refleak in C functools.lru_cache http://bugs.python.org/issue28653 closed by yselivanov #28656: build error in cygwin64 ,because pthread_key_t http://bugs.python.org/issue28656 closed by serhiy.storchaka #28659: xml.etree.cElementTree.write misses opening tag http://bugs.python.org/issue28659 closed by xiang.zhang #28664: test_bz2 fails with BrokenPipeError when bunzip2 is missing http://bugs.python.org/issue28664 closed by serhiy.storchaka From christoph at grothesque.org Mon Nov 14 08:58:36 2016 From: christoph at grothesque.org (Christoph Groth) Date: Mon, 14 Nov 2016 14:58:36 +0100 Subject: [Python-Dev] Warnings behave differently in Python 2 and Python 3? Message-ID: <87zil2uprn.fsf@grothesque.org> (I assume that this list is appropriate for this topic, but if it isn't, I will be grateful for being redirected to the appropriate place.) It seems that warnings behave differently in Python 2 and Python 3. Consider the following script. **************** def func(): warnings.warn("Blabla", RuntimeWarning) warnings.simplefilter("ignore") func() warnings.simplefilter("always") func() **************** When run with CPython 2.7, there is no output, when run with CPython 3.5, there is the following output: **************** /tmp/test.py:4: RuntimeWarning: Blabla warnings.warn("Blabla", RuntimeWarning) **************** Was there indeed a change of how warnings behave in Python 3? I searched, but couldn't find any documentation for this. Understanding how warnings work is especially important when testing for them. This is how I stumbled across this, in case anyone is interested: https://gitlab.kwant-project.org/kwant/kwant/issues/1#note_2608 I also note that the current documentation still uses DeprecationWarning in the example of how to suppress a warning: https://docs.python.org/dev/library/warnings.html#temporarily-suppressing-warnings. Since DeprecationWarning is disabled by default in modern Python, perhaps a different warning would provide a better example? From victor.stinner at gmail.com Mon Nov 14 12:16:45 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 14 Nov 2016 18:16:45 +0100 Subject: [Python-Dev] Warnings behave differently in Python 2 and Python 3? In-Reply-To: <87zil2uprn.fsf@grothesque.org> References: <87zil2uprn.fsf@grothesque.org> Message-ID: You are looking for the bug fix http://bugs.python.org/issue4180: "warnings.simplefilter("always") does not make warnings always show up" Python 2.7 has a bug, Python 3.4.3+ behaves correctly. It is surprising that Python 3.4.0 and Python 3.4.3 behaves differently :-) (Ubuntu 14.04 uses Python 3.4.0.. Victor 2016-11-14 14:58 GMT+01:00 Christoph Groth : > (I assume that this list is appropriate for this topic, but if it isn't, I > will be grateful for being redirected to the appropriate place.) > > It seems that warnings behave differently in Python 2 and Python 3. > Consider the following script. > > **************** > def func(): > warnings.warn("Blabla", RuntimeWarning) > > warnings.simplefilter("ignore") > func() > warnings.simplefilter("always") > func() > **************** > > When run with CPython 2.7, there is no output, when run with CPython 3.5, > there is the following output: > > **************** > /tmp/test.py:4: RuntimeWarning: Blabla > warnings.warn("Blabla", RuntimeWarning) > **************** > > Was there indeed a change of how warnings behave in Python 3? I searched, > but couldn't find any documentation for this. > > Understanding how warnings work is especially important when testing for > them. > This is how I stumbled across this, in case anyone is interested: > https://gitlab.kwant-project.org/kwant/kwant/issues/1#note_2608 > > I also note that the current documentation still uses DeprecationWarning in > the example of how to suppress a warning: > https://docs.python.org/dev/library/warnings.html#temporarily-suppressing-warnings. > Since DeprecationWarning is disabled by default in modern Python, perhaps a > different warning would provide a better example? > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/victor.stinner%40gmail.com From christoph at grothesque.org Mon Nov 14 16:44:22 2016 From: christoph at grothesque.org (Christoph Groth) Date: Mon, 14 Nov 2016 22:44:22 +0100 Subject: [Python-Dev] Warnings behave differently in Python 2 and Python 3? In-Reply-To: (Victor Stinner's message of "Mon, 14 Nov 2016 18:16:45 +0100") References: <87zil2uprn.fsf@grothesque.org> Message-ID: <87vavpenyh.fsf@grothesque.org> Thanks, Victor, for the link to the bug fix! I suspected that the original mechanism was something like that, but I believed that it was so by design. I find it a bit surprising that CPython gets changed between versions in backwards-incompatible ways (even if it?s a bug fix) without a notice in "what's new"... Christoph From tjreedy at udel.edu Mon Nov 14 23:19:32 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 14 Nov 2016 23:19:32 -0500 Subject: [Python-Dev] Warnings behave differently in Python 2 and Python 3? In-Reply-To: <87vavpenyh.fsf@grothesque.org> References: <87zil2uprn.fsf@grothesque.org> <87vavpenyh.fsf@grothesque.org> Message-ID: On 11/14/2016 4:44 PM, Christoph Groth wrote: > Thanks, Victor, for the link to the bug fix! I suspected that the > original mechanism was something like that, but I believed that it was > so by design. > > I find it a bit surprising that CPython gets changed between versions in > backwards-incompatible ways (even if it?s a bug fix) without a notice in > "what's new"... It doesn't. What's New proper only lists new features, which only appear in x.y.0. But is includes a link to an updated 'changelog', which includes bug fixes, listed in separate sections for each release, including release candidates. "See the changelog for a full list of changes.": 2nd line of What's New in Python 3.5. -- Terry Jan Reedy From ned at nedbatchelder.com Wed Nov 16 19:18:16 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Wed, 16 Nov 2016 19:18:16 -0500 Subject: [Python-Dev] Recent changes to PyCodeObject Message-ID: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> When I added Python 3.6 support to coverage.py, I posted a Mac wheel to PyPI: https://pypi.python.org/pypi/coverage/ That wheel was built against 3.6a3, the latest version at the time. When I use it now on 3.6b3, it doesn't work right. The reason is that the co_firstlineno field in PyCodeObject moved in September, as part of the PEP 523 work: https://github.com/python/cpython/commit/f1e6d88b3ca2b56d51d87b6b38ea1870c5d9490c The docs say that PyCodeObject can change at any time, but I don't see why the field had to move in the first place. Was this needed? Am I doing the wrong thing by using PyCodeObject fields directly in the coverage.py C trace function? It seems like this was an unnecessary breaking change, even if it is a non-guaranteed interface. BTW, there are some comments that should be updated based on that commit, I can submit a patch to fix them. --Ned. From steve.dower at python.org Wed Nov 16 19:44:46 2016 From: steve.dower at python.org (Steve Dower) Date: Wed, 16 Nov 2016 16:44:46 -0800 Subject: [Python-Dev] Recent changes to PyCodeObject In-Reply-To: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> References: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> Message-ID: On 16Nov2016 1618, Ned Batchelder wrote: > When I added Python 3.6 support to coverage.py, I posted a Mac wheel to > PyPI: https://pypi.python.org/pypi/coverage/ That wheel was built > against 3.6a3, the latest version at the time. When I use it now on > 3.6b3, it doesn't work right. The reason is that the co_firstlineno > field in PyCodeObject moved in September, as part of the PEP 523 work: > https://github.com/python/cpython/commit/f1e6d88b3ca2b56d51d87b6b38ea1870c5d9490c > > The docs say that PyCodeObject can change at any time, but I don't see > why the field had to move in the first place. Was this needed? IIRC, by reordering the fields we saved enough memory in padding that the new field did not take up any extra space (assuming 32-bit int and 64-bit pointers). > Am I doing the wrong thing by using PyCodeObject fields directly in the > coverage.py C trace function? It seems like this was an unnecessary > breaking change, even if it is a non-guaranteed interface. I suspect the "wrong thing" here is expecting to not see any changes between alpha/beta versions. It certainly should not change within 3.6.x, so your wheel build will be fine, but changing during the prerelease versions is a possibility for anything not in the limited ABI. Cheers, Steve From ncoghlan at gmail.com Wed Nov 16 22:49:15 2016 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 17 Nov 2016 13:49:15 +1000 Subject: [Python-Dev] Recent changes to PyCodeObject In-Reply-To: References: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> Message-ID: On 17 November 2016 at 10:44, Steve Dower wrote: > On 16Nov2016 1618, Ned Batchelder wrote: >> Am I doing the wrong thing by using PyCodeObject fields directly in the >> coverage.py C trace function? It seems like this was an unnecessary >> breaking change, even if it is a non-guaranteed interface. > > I suspect the "wrong thing" here is expecting to not see any changes between > alpha/beta versions. It certainly should not change within 3.6.x, so your > wheel build will be fine, but changing during the prerelease versions is a > possibility for anything not in the limited ABI. Right, and we don't have an explicit policy on when the ABI gets locked prior to the x.y.0 release other than the general rule of "minimise change after the first release candidate". This question also came up on the wheel-builders list in the context of "When can Python 3.6 support be added to the published manylinux1 build environments without risking ABI incompatible wheels on final release?". Perhaps it would make sense to declare the final beta the last opportunity for public ABI changes? That would give folks a few weeks to start preparing launch binaries if they want to do so, rather than the single week planned between the rc and the final release. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From njs at pobox.com Wed Nov 16 23:18:51 2016 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 16 Nov 2016 20:18:51 -0800 Subject: [Python-Dev] Recent changes to PyCodeObject In-Reply-To: References: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> Message-ID: Didn't 3.5 have to roll an extra last minute RC for an emergency abi-breaking bug fix, though? (Thinking of the windows runtime stuff.) On Nov 16, 2016 7:51 PM, "Nick Coghlan" wrote: > On 17 November 2016 at 10:44, Steve Dower wrote: > > On 16Nov2016 1618, Ned Batchelder wrote: > >> Am I doing the wrong thing by using PyCodeObject fields directly in the > >> coverage.py C trace function? It seems like this was an unnecessary > >> breaking change, even if it is a non-guaranteed interface. > > > > I suspect the "wrong thing" here is expecting to not see any changes > between > > alpha/beta versions. It certainly should not change within 3.6.x, so your > > wheel build will be fine, but changing during the prerelease versions is > a > > possibility for anything not in the limited ABI. > > Right, and we don't have an explicit policy on when the ABI gets > locked prior to the x.y.0 release other than the general rule of > "minimise change after the first release candidate". > > This question also came up on the wheel-builders list in the context > of "When can Python 3.6 support be added to the published manylinux1 > build environments without risking ABI incompatible wheels on final > release?". > > Perhaps it would make sense to declare the final beta the last > opportunity for public ABI changes? That would give folks a few weeks > to start preparing launch binaries if they want to do so, rather than > the single week planned between the rc and the final release. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > njs%40pobox.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Thu Nov 17 00:31:15 2016 From: steve.dower at python.org (Steve Dower) Date: Wed, 16 Nov 2016 21:31:15 -0800 Subject: [Python-Dev] Recent changes to PyCodeObject In-Reply-To: References: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> Message-ID: Yeah, we did. That issue had been around since the betas though, it just wasn't discovered until late. This is why we've been encouraging people to treat the betas like they used to treat the RCs (at least, I've been doing what I can). They're ready for testing against, just not ready for general distribution (or for producing wheels for general distribution). Cheers, Steve Top-posted from my Windows Phone -----Original Message----- From: "Nathaniel Smith" Sent: ?11/?16/?2016 20:19 To: "Nick Coghlan" Cc: "Python Dev" ; "Steve Dower" Subject: Re: [Python-Dev] Recent changes to PyCodeObject Didn't 3.5 have to roll an extra last minute RC for an emergency abi-breaking bug fix, though? (Thinking of the windows runtime stuff.) On Nov 16, 2016 7:51 PM, "Nick Coghlan" wrote: On 17 November 2016 at 10:44, Steve Dower wrote: > On 16Nov2016 1618, Ned Batchelder wrote: >> Am I doing the wrong thing by using PyCodeObject fields directly in the >> coverage.py C trace function? It seems like this was an unnecessary >> breaking change, even if it is a non-guaranteed interface. > > I suspect the "wrong thing" here is expecting to not see any changes between > alpha/beta versions. It certainly should not change within 3.6.x, so your > wheel build will be fine, but changing during the prerelease versions is a > possibility for anything not in the limited ABI. Right, and we don't have an explicit policy on when the ABI gets locked prior to the x.y.0 release other than the general rule of "minimise change after the first release candidate". This question also came up on the wheel-builders list in the context of "When can Python 3.6 support be added to the published manylinux1 build environments without risking ABI incompatible wheels on final release?". Perhaps it would make sense to declare the final beta the last opportunity for public ABI changes? That would give folks a few weeks to start preparing launch binaries if they want to do so, rather than the single week planned between the rc and the final release. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/njs%40pobox.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From cody.piersall at gmail.com Thu Nov 17 10:09:15 2016 From: cody.piersall at gmail.com (Cody Piersall) Date: Thu, 17 Nov 2016 09:09:15 -0600 Subject: [Python-Dev] Recent changes to PyCodeObject In-Reply-To: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> References: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> Message-ID: On Wed, Nov 16, 2016 at 6:18 PM, Ned Batchelder wrote: > > When I added Python 3.6 support to coverage.py, I posted a Mac wheel to > PyPI: https://pypi.python.org/pypi/coverage/ That wheel was built > against 3.6a3, the latest version at the time. When I use it now on > 3.6b3, it doesn't work right. The reason is that the co_firstlineno > field in PyCodeObject moved in September, as part of the PEP 523 work: > https://github.com/python/cpython/commit/f1e6d88b3ca2b56d51d87b6b38ea1870c5d9490c > > The docs say that PyCodeObject can change at any time, but I don't see > why the field had to move in the first place. Was this needed? Christian Heimes had the idea so that memory consumption wouldn't be impacted by adding co_extra to the struct: https://mail.python.org/pipermail/python-dev/2016-August/145961.html Cody From ned at nedbatchelder.com Thu Nov 17 10:54:13 2016 From: ned at nedbatchelder.com (Ned Batchelder) Date: Thu, 17 Nov 2016 10:54:13 -0500 Subject: [Python-Dev] Recent changes to PyCodeObject In-Reply-To: References: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> Message-ID: <69da1c01-0708-8809-f2f1-f0e6870bd6bc@nedbatchelder.com> On 11/17/16 10:09 AM, Cody Piersall wrote: > On Wed, Nov 16, 2016 at 6:18 PM, Ned Batchelder wrote: >> When I added Python 3.6 support to coverage.py, I posted a Mac wheel to >> PyPI: https://pypi.python.org/pypi/coverage/ That wheel was built >> against 3.6a3, the latest version at the time. When I use it now on >> 3.6b3, it doesn't work right. The reason is that the co_firstlineno >> field in PyCodeObject moved in September, as part of the PEP 523 work: >> https://github.com/python/cpython/commit/f1e6d88b3ca2b56d51d87b6b38ea1870c5d9490c >> >> The docs say that PyCodeObject can change at any time, but I don't see >> why the field had to move in the first place. Was this needed? > Christian Heimes had the idea so that memory consumption wouldn't be > impacted by adding co_extra to the struct: > https://mail.python.org/pipermail/python-dev/2016-August/145961.html > Thanks, I should have searched :) --Ned. From status at bugs.python.org Fri Nov 18 12:08:49 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 18 Nov 2016 18:08:49 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20161118170849.F3D5A5663B@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-11-11 - 2016-11-18) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5584 (+25) closed 34909 (+43) total 40493 (+68) Open issues with patches: 2421 Issues opened (51) ================== #28032: --with-lto builds segfault in many situations http://bugs.python.org/issue28032 reopened by koobs #28671: SSL server requesting client certificates should send CA list http://bugs.python.org/issue28671 opened by kchen #28672: Explain in the "Data Model" document why arguments to __init__ http://bugs.python.org/issue28672 opened by Jo??o.Sebasti??o.de.Oliveira.Bueno #28673: When using pyro4 with more than 15 threads, python 2.7.12 core http://bugs.python.org/issue28673 opened by pwp333 #28675: about PEP 528 / PEP 529 http://bugs.python.org/issue28675 opened by Big Stone #28677: difficult to parse sentence structure in "When an instance att http://bugs.python.org/issue28677 opened by viorel #28680: bdist_wininst generated 64-bit executable looks for 3.5-32 http://bugs.python.org/issue28680 opened by Timothy Widrick #28681: About function renaming in the tutorial http://bugs.python.org/issue28681 opened by xfq #28682: Bytes support in os.fwalk() http://bugs.python.org/issue28682 opened by serhiy.storchaka #28683: bind() on a unix socket raises PermissionError on Android for http://bugs.python.org/issue28683 opened by xdegaye #28684: [asyncio] bind() on a unix socket raises PermissionError on An http://bugs.python.org/issue28684 opened by xdegaye #28685: Optimizing list.sort() by performing safety checks in advance http://bugs.python.org/issue28685 opened by elliot.gorokhovsky #28686: py.exe ignored PATH when using python3 shebang http://bugs.python.org/issue28686 opened by wdhwg001 #28687: Python 2.7.12 windows x64 installer fails after previous bad u http://bugs.python.org/issue28687 opened by Igor.Skochinsky #28688: Warning -- warnings.filters was modified by test_warnings http://bugs.python.org/issue28688 opened by haypo #28689: OpenSSL 1.1.0c test failures http://bugs.python.org/issue28689 opened by christian.heimes #28692: gettext: deprecate selecting plural form by fractional numbers http://bugs.python.org/issue28692 opened by serhiy.storchaka #28693: No HKSCS support in Windows cp950 http://bugs.python.org/issue28693 opened by Artoria2e5 #28694: tkinter interface to fontchooser http://bugs.python.org/issue28694 opened by Lance Ware #28695: Add SSL_CTX_set_client_cert_engine http://bugs.python.org/issue28695 opened by christian.heimes #28696: imap from ThreadPool hangs by an exception in a generator func http://bugs.python.org/issue28696 opened by lev-veshnyakov #28697: asyncio.Lock, Condition, Semaphore docs don't mention `async w http://bugs.python.org/issue28697 opened by ulope #28698: Python 3.x ctypes c_wchar_p return is different from official http://bugs.python.org/issue28698 opened by Alex Wang #28699: Imap from ThreadPool behaves unexpectedly http://bugs.python.org/issue28699 opened by lev-veshnyakov #28700: test_dbm failure: KeyError: b'0' (intermittent in 3.5, reliabl http://bugs.python.org/issue28700 opened by gward #28702: Confusing error message when None used in expressions, eg. "'N http://bugs.python.org/issue28702 opened by gward #28705: Clean up design FAQ question about compiling to C http://bugs.python.org/issue28705 opened by brett.cannon #28706: msvc9compiler does not find a vcvarsall.bat of Visual C++ for http://bugs.python.org/issue28706 opened by jyrkih #28707: add 'directory' option to the http.server module http://bugs.python.org/issue28707 opened by matrixise #28708: Low FD_SETSIZE limit on Windows http://bugs.python.org/issue28708 opened by David Hirschfeld #28709: PyStructSequence_NewType is broken; makes GC type without sett http://bugs.python.org/issue28709 opened by josh.r #28710: Sphinx incompatible markup in configparser.ConfigParser. http://bugs.python.org/issue28710 opened by Patrick Lehmann #28711: IDLE doesn't open http://bugs.python.org/issue28711 opened by bg90299 #28712: Non-Windows mappings for a couple of Windows code pages http://bugs.python.org/issue28712 opened by Artoria2e5 #28714: Addition to Documentation of configparser.ConfigParser.write() http://bugs.python.org/issue28714 opened by georgefischhof #28715: Check result of PyUnicode_AsUTF8 http://bugs.python.org/issue28715 opened by serhiy.storchaka #28716: Fractions instantiation revisited http://bugs.python.org/issue28716 opened by wolma #28718: '*' matches entire path in fnmatch.translate http://bugs.python.org/issue28718 opened by Jim Nasby #28719: zipfile increase in size http://bugs.python.org/issue28719 opened by X-Istence #28722: doctest example exit status http://bugs.python.org/issue28722 opened by Brainix #28724: Add method send_io, recv_io to the socket module. http://bugs.python.org/issue28724 opened by tokibito #28727: Implement comparison (x==y and x!=y) for _sre.SRE_Pattern http://bugs.python.org/issue28727 opened by haypo #28728: test_host_resolution in test_socket fails on duplicate assert http://bugs.python.org/issue28728 opened by SilentGhost #28729: Exception from sqlite3 adapter masked by sqlite3.InterfaceErro http://bugs.python.org/issue28729 opened by inglesp #28730: __reduce__ not being called in dervied extension class from da http://bugs.python.org/issue28730 opened by Jeff Reback #28731: _PyDict_NewPresized() creates too small dict http://bugs.python.org/issue28731 opened by inada.naoki #28732: spawnl crash on windows7 http://bugs.python.org/issue28732 opened by srinim #28733: Show how to use mock_open in modules other that __main__ http://bugs.python.org/issue28733 opened by butla #28735: Mock is equal to ANY but MagicMock is not http://bugs.python.org/issue28735 opened by rafael.fonseca #28736: multiprocessing.Lock() no longer has .acquire() http://bugs.python.org/issue28736 opened by Eric Leadbetter #28737: Document that tp_dealloc handler must call PyObject_GC_UnTrack http://bugs.python.org/issue28737 opened by colesbury Most recent 15 issues with no replies (15) ========================================== #28737: Document that tp_dealloc handler must call PyObject_GC_UnTrack http://bugs.python.org/issue28737 #28736: multiprocessing.Lock() no longer has .acquire() http://bugs.python.org/issue28736 #28735: Mock is equal to ANY but MagicMock is not http://bugs.python.org/issue28735 #28733: Show how to use mock_open in modules other that __main__ http://bugs.python.org/issue28733 #28722: doctest example exit status http://bugs.python.org/issue28722 #28714: Addition to Documentation of configparser.ConfigParser.write() http://bugs.python.org/issue28714 #28706: msvc9compiler does not find a vcvarsall.bat of Visual C++ for http://bugs.python.org/issue28706 #28705: Clean up design FAQ question about compiling to C http://bugs.python.org/issue28705 #28697: asyncio.Lock, Condition, Semaphore docs don't mention `async w http://bugs.python.org/issue28697 #28695: Add SSL_CTX_set_client_cert_engine http://bugs.python.org/issue28695 #28687: Python 2.7.12 windows x64 installer fails after previous bad u http://bugs.python.org/issue28687 #28683: bind() on a unix socket raises PermissionError on Android for http://bugs.python.org/issue28683 #28682: Bytes support in os.fwalk() http://bugs.python.org/issue28682 #28680: bdist_wininst generated 64-bit executable looks for 3.5-32 http://bugs.python.org/issue28680 #28672: Explain in the "Data Model" document why arguments to __init__ http://bugs.python.org/issue28672 Most recent 15 issues waiting for review (15) ============================================= #28733: Show how to use mock_open in modules other that __main__ http://bugs.python.org/issue28733 #28731: _PyDict_NewPresized() creates too small dict http://bugs.python.org/issue28731 #28728: test_host_resolution in test_socket fails on duplicate assert http://bugs.python.org/issue28728 #28727: Implement comparison (x==y and x!=y) for _sre.SRE_Pattern http://bugs.python.org/issue28727 #28724: Add method send_io, recv_io to the socket module. http://bugs.python.org/issue28724 #28716: Fractions instantiation revisited http://bugs.python.org/issue28716 #28715: Check result of PyUnicode_AsUTF8 http://bugs.python.org/issue28715 #28710: Sphinx incompatible markup in configparser.ConfigParser. http://bugs.python.org/issue28710 #28707: add 'directory' option to the http.server module http://bugs.python.org/issue28707 #28699: Imap from ThreadPool behaves unexpectedly http://bugs.python.org/issue28699 #28692: gettext: deprecate selecting plural form by fractional numbers http://bugs.python.org/issue28692 #28688: Warning -- warnings.filters was modified by test_warnings http://bugs.python.org/issue28688 #28685: Optimizing list.sort() by performing safety checks in advance http://bugs.python.org/issue28685 #28684: [asyncio] bind() on a unix socket raises PermissionError on An http://bugs.python.org/issue28684 #28683: bind() on a unix socket raises PermissionError on Android for http://bugs.python.org/issue28683 Top 10 most discussed issues (10) ================================= #28712: Non-Windows mappings for a couple of Windows code pages http://bugs.python.org/issue28712 22 msgs #28618: Decorate hot functions using __attribute__((hot)) to optimize http://bugs.python.org/issue28618 15 msgs #28731: _PyDict_NewPresized() creates too small dict http://bugs.python.org/issue28731 15 msgs #24379: Add operator.subscript as a convenience for creating slices http://bugs.python.org/issue24379 11 msgs #28688: Warning -- warnings.filters was modified by test_warnings http://bugs.python.org/issue28688 11 msgs #28727: Implement comparison (x==y and x!=y) for _sre.SRE_Pattern http://bugs.python.org/issue28727 11 msgs #24339: iso6937 encoding missing http://bugs.python.org/issue24339 10 msgs #21145: Add the @cached_property decorator http://bugs.python.org/issue21145 9 msgs #27945: Various segfaults with dict http://bugs.python.org/issue27945 9 msgs #28673: When using pyro4 with more than 15 threads, python 2.7.12 core http://bugs.python.org/issue28673 9 msgs Issues closed (43) ================== #19717: resolve() fails when the path doesn't exist http://bugs.python.org/issue19717 closed by steve.dower #21449: Replace _PyUnicode_CompareWithId with _PyUnicode_EqualToASCIII http://bugs.python.org/issue21449 closed by serhiy.storchaka #22495: merge large parts of test_binop.py and test_fractions.py http://bugs.python.org/issue22495 closed by haypo #26920: android: test_sys fails http://bugs.python.org/issue26920 closed by xdegaye #26926: test_io large file test failure on 32 bits Android platforms http://bugs.python.org/issue26926 closed by xdegaye #26929: android: test_strptime fails http://bugs.python.org/issue26929 closed by xdegaye #26931: android: test_distutils fails http://bugs.python.org/issue26931 closed by xdegaye #26934: android: test_faulthandler fails http://bugs.python.org/issue26934 closed by xdegaye #26935: android: test_os fails http://bugs.python.org/issue26935 closed by xdegaye #27854: Installed 2.7: IDLE Help disabled because help.html is missing http://bugs.python.org/issue27854 closed by steve.dower #28000: Build fails on AIX with _LINUX_SOURCE_COMPAT flag http://bugs.python.org/issue28000 closed by martin.panter #28016: test_fileio fails on AIX http://bugs.python.org/issue28016 closed by martin.panter #28531: Improve utf7 encoder memory usage http://bugs.python.org/issue28531 closed by haypo #28573: Python 3.6.0b3 64-bit has no sys._mercurial info http://bugs.python.org/issue28573 closed by steve.dower #28614: Slice limit documentation could be misinterpreted in Python 2. http://bugs.python.org/issue28614 closed by rhettinger #28631: [2.7/3.5/3.6 Regression] crash using ctypes http://bugs.python.org/issue28631 closed by doko #28632: configparser does not close files in read http://bugs.python.org/issue28632 closed by PetrPy #28637: Python startup performance regression http://bugs.python.org/issue28637 closed by haypo #28644: Document recent changes in typing.py http://bugs.python.org/issue28644 closed by gvanrossum #28648: False assert in _Py_DecodeUTF8_surrogateescape http://bugs.python.org/issue28648 closed by serhiy.storchaka #28658: IDLE: catch user cfg file error and improve error message http://bugs.python.org/issue28658 closed by terry.reedy #28662: catch also PermissionError in tests when spawning a non existe http://bugs.python.org/issue28662 closed by xdegaye #28665: Harmonize STORE_DEREF with STORE_FAST and LOAD_DEREF http://bugs.python.org/issue28665 closed by rhettinger #28668: instanciation of multiprocessing.Queue raises ImportError in t http://bugs.python.org/issue28668 closed by xdegaye #28669: Math Library Dos Attack http://bugs.python.org/issue28669 closed by haypo #28670: PEP 235: Implement on every POSIX system, and clean up the C c http://bugs.python.org/issue28670 closed by brett.cannon #28674: Spam http://bugs.python.org/issue28674 closed by zach.ware #28676: On macOS Sierra, warning: implicit declaration of function 'ge http://bugs.python.org/issue28676 closed by ned.deily #28678: tarfile extract docs inconsistent naming of parameter numeric_ http://bugs.python.org/issue28678 closed by martin.panter #28679: CGIHTTPServer displays raw python code when the url contains ' http://bugs.python.org/issue28679 closed by Yudai Fujiwara #28690: Loop in re (regular expression) processing http://bugs.python.org/issue28690 closed by haypo #28691: python3.6 -Werror -c '"\c"' fails with an assertion error http://bugs.python.org/issue28691 closed by haypo #28701: Replace PyUnicode_CompareWithASCIIString with _PyUnicode_Equal http://bugs.python.org/issue28701 closed by serhiy.storchaka #28703: Fix asyncio.iscoroutinefunction to handle Mock objects http://bugs.python.org/issue28703 closed by yselivanov #28704: Fix create_unix_server to support Path-like objects (PEP 519) http://bugs.python.org/issue28704 closed by yselivanov #28713: Recent tutorial for recent Python3 still uses IOError. http://bugs.python.org/issue28713 closed by python-dev #28717: rounding error in % operator http://bugs.python.org/issue28717 closed by mark.dickinson #28720: Add collections.abc.AsyncGenerator http://bugs.python.org/issue28720 closed by yselivanov #28721: Fix asynchronous generators athrow() and aclose() to handle St http://bugs.python.org/issue28721 closed by yselivanov #28723: tkinter, radiobutton, indicatoron=0 has no effect. http://bugs.python.org/issue28723 closed by serhiy.storchaka #28725: Awaiting an awaitable returned from an async function does not http://bugs.python.org/issue28725 closed by yselivanov #28726: py.exe launches wrong version http://bugs.python.org/issue28726 closed by pravic #28734: argparse: successive parsing wipes out nargs=? values http://bugs.python.org/issue28734 closed by ajstewart From brett at python.org Fri Nov 18 12:47:49 2016 From: brett at python.org (Brett Cannon) Date: Fri, 18 Nov 2016 17:47:49 +0000 Subject: [Python-Dev] Recent changes to PyCodeObject In-Reply-To: <69da1c01-0708-8809-f2f1-f0e6870bd6bc@nedbatchelder.com> References: <07a847cf-d144-d2fd-69df-32ce1e9bd518@nedbatchelder.com> <69da1c01-0708-8809-f2f1-f0e6870bd6bc@nedbatchelder.com> Message-ID: I know you got your answer to why the change occurred, but a patch to update any outdated comments would be appreciated. :) On Thu, 17 Nov 2016 at 07:55 Ned Batchelder wrote: > On 11/17/16 10:09 AM, Cody Piersall wrote: > > On Wed, Nov 16, 2016 at 6:18 PM, Ned Batchelder > wrote: > >> When I added Python 3.6 support to coverage.py, I posted a Mac wheel to > >> PyPI: https://pypi.python.org/pypi/coverage/ That wheel was built > >> against 3.6a3, the latest version at the time. When I use it now on > >> 3.6b3, it doesn't work right. The reason is that the co_firstlineno > >> field in PyCodeObject moved in September, as part of the PEP 523 work: > >> > https://github.com/python/cpython/commit/f1e6d88b3ca2b56d51d87b6b38ea1870c5d9490c > >> > >> The docs say that PyCodeObject can change at any time, but I don't see > >> why the field had to move in the first place. Was this needed? > > Christian Heimes had the idea so that memory consumption wouldn't be > > impacted by adding co_extra to the struct: > > https://mail.python.org/pipermail/python-dev/2016-August/145961.html > > > > Thanks, I should have searched :) > > --Ned. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Nov 18 14:48:50 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 18 Nov 2016 14:48:50 -0500 Subject: [Python-Dev] New problem accessing tracker with Firefox 50.0 Message-ID: Starting today, while reviewing the "Summary of Python tracker issues", I get the following for about half the clicks. """ Secure Connection Failed An error occurred during a connection to bugs.python.org. A PKCS #11 module returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR The page you are trying to view cannot be shown because the authenticity of the received data could not be verified. Please contact the website owners to inform them of this problem. [ Try Again ] """ So far, the Try Again button has always worked. I believe I upgraded Firefox (to 50.0) just last night, or possibly the night before. -- Terry Jan Reedy From phd at phdru.name Fri Nov 18 14:55:06 2016 From: phd at phdru.name (Oleg Broytman) Date: Fri, 18 Nov 2016 20:55:06 +0100 Subject: [Python-Dev] New problem accessing tracker with Firefox 50.0 In-Reply-To: References: Message-ID: <20161118195506.GA31943@phdru.name> Hi! On Fri, Nov 18, 2016 at 02:48:50PM -0500, Terry Reedy wrote: > Starting today, while reviewing the "Summary of Python tracker issues", I > get the following for about half the clicks. > > """ > Secure Connection Failed > > An error occurred during a connection to bugs.python.org. A PKCS #11 module > returned CKR_DEVICE_ERROR, indicating that a problem has occurred with the > token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR > > The page you are trying to view cannot be shown because the authenticity > of the received data could not be verified. > Please contact the website owners to inform them of this problem. > > [ Try Again ] > """ > > So far, the Try Again button has always worked. > > I believe I upgraded Firefox (to 50.0) just last night, or possibly the > night before. I upgraded to Firefox 50.0 a day ago and can connect to https://bugs.python.org without any problem. > -- > Terry Jan Reedy Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From vadmium at gmail.com Fri Nov 18 16:28:21 2016 From: vadmium at gmail.com (Martin Panter) Date: Fri, 18 Nov 2016 21:28:21 +0000 Subject: [Python-Dev] New problem accessing tracker with Firefox 50.0 In-Reply-To: References: Message-ID: On 18/11/2016, Terry Reedy wrote: > Starting today, while reviewing the "Summary of Python tracker issues", > I get the following for about half the clicks. > > """ > Secure Connection Failed > > An error occurred during a connection to bugs.python.org. A PKCS #11 > module returned CKR_DEVICE_ERROR, indicating that a problem has occurred > with the token or slot. Error code: SEC_ERROR_PKCS11_DEVICE_ERROR > > The page you are trying to view cannot be shown because the > authenticity of the received data could not be verified. > Please contact the website owners to inform them of this problem. > > [ Try Again ] > """ > > So far, the Try Again button has always worked. > > I believe I upgraded Firefox (to 50.0) just last night, or possibly the > night before. Yesterday I started seeing this behaviour too. I have Firefox 43 and haven?t updated any relevant libraries recently, so I assumed it was the server?s fault :) From kmod at dropbox.com Sat Nov 19 14:46:36 2016 From: kmod at dropbox.com (Kevin Modzelewski) Date: Sat, 19 Nov 2016 11:46:36 -0800 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (b78574cb00ab): sum=1120 In-Reply-To: References: <20161109090247.50140.35000.CEB8D404@psf.io> Message-ID: Hi Yury, you may be interested in some leak-finding code that wrote for Pyston. It uses the GC infrastructure to show you objects that were directly leaked, ignoring indirect leaks -- ie objects that were only leaked because they were referenced by a leaked object. It can often give you a very small list of objects to look into (depending on how many non-gc objects were leaked). If you're interested I can try porting it to CPython. https://github.com/dropbox/pyston/blob/master/from_cpython/Modules/gcmodule.c#L894 kmod On Wed, Nov 9, 2016 at 7:16 AM, Yury Selivanov wrote: > I'm trying to fix refleaks in 3.6. So far: > > On 2016-11-09 4:02 AM, solipsis at pitrou.net wrote: > > results for b78574cb00ab on branch "default" >> -------------------------------------------- >> >> test_ast leaked [98, 98, 98] references, sum=294 >> test_ast leaked [98, 98, 98] memory blocks, sum=294 >> test_asyncio leaked [3, 0, 0] memory blocks, sum=3 >> test_code leaked [2, 2, 2] references, sum=6 >> test_code leaked [2, 2, 2] memory blocks, sum=6 >> test_functools leaked [0, 3, 1] memory blocks, sum=4 >> test_pydoc leaked [106, 106, 106] references, sum=318 >> test_pydoc leaked [42, 42, 42] memory blocks, sum=126 >> test_trace leaked [12, 12, 12] references, sum=36 >> test_trace leaked [11, 11, 11] memory blocks, sum=33 >> >> >> > test_ast, test_code and test_trace were fixed by > https://hg.python.org/cpython/rev/2c6825c9ecfd > > test_pydoc leaks in test_typing_pydoc. I tried git bisect and it looks > like that the first commit that introduced the refleak was the one that > added test_typing_pydoc! > > 62127e60e7b0 doesn't modify any CPython internals, so it looks like that > test_typing_pydoc exposed some bug that has existed before it. Any help > tracking that down is welcome :) > > Yury > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/kmod% > 40dropbox.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Sun Nov 20 16:31:51 2016 From: greg at krypto.org (Gregory P. Smith) Date: Sun, 20 Nov 2016 21:31:51 +0000 Subject: [Python-Dev] Would someone please update the profile-opt buildbot flag settings? Message-ID: For http://buildbot.python.org/all/buildslaves/gps-debian-profile-opt in particular (and any others we have using it - if any): Our new --with-optimizations configure flag has been renamed to *--enable-optimizations* to be more consistent with preferred autoconf terminology (see https://bugs.python.org/issue28032 for why). thanks, -gregory.p.smith -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Sun Nov 20 16:58:27 2016 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Sun, 20 Nov 2016 15:58:27 -0600 Subject: [Python-Dev] Would someone please update the profile-opt buildbot flag settings? In-Reply-To: References: Message-ID: On Sun, Nov 20, 2016 at 3:31 PM, Gregory P. Smith wrote: > For http://buildbot.python.org/all/buildslaves/gps-debian-profile-opt in > particular (and any others we have using it - if any): Our new > --with-optimizations configure flag has been renamed to > --enable-optimizations to be more consistent with preferred autoconf > terminology (see https://bugs.python.org/issue28032 for why). Done. -- Zach From nad at python.org Mon Nov 21 05:35:42 2016 From: nad at python.org (Ned Deily) Date: Mon, 21 Nov 2016 05:35:42 -0500 Subject: [Python-Dev] IMPORTANT: 3.6.0b4 snapshot today, workflow changes for the final weeks of 3.6.0 Message-ID: <541563DC-61B5-4DE8-91DA-11DC68B667B8@python.org> The final beta snapshot planned for the 3.6 release cycle is scheduled to be tagged today within the next 12 hours. We then begin the Release Candidate stage, the final days leading up to 3.6.0, with the release candidate scheduled to be tagged in about 2 weeks. As I have noted previously, with the tagging today and release of beta 4, *all* non-critical bug fixes for 3.6.0 should now have been checked in. From the b4 tag on, *only* release critical and doc fixes should be pushed for the release candidate. A reminder from the Developer's Guide regarding the Release Candidate stage of the release cycle: "A branch preparing for an RC release can only have bugfixes applied that have been reviewed by other core developers. Generally, these issues must be severe enough (e.g. crashes) that they deserve fixing before the final release. All other issues should be deferred to the next development cycle, since stability is the strongest concern at this point. You cannot skip the peer review during an RC, no matter how small! Even if it is a simple copy-and-paste change, everything requires peer review from a core developer." A goal for this release has been to have no changes between the release candidate and the final release other than the tag itself. Remember that every time we make a change at this point, it puts a burden on and adds risk to our testing efforts and, more importantly, the efforts of our third-party package developers, downstream distributors, and end users helping us ensure that 3.6.0 will be a high-quality release. I think everyone has been doing a great job so far in exercising good judgement about what is appropriate at each stage of our release cycle. Thank you! Now, after b4, unless a code change addresses a truly release critical issue, please target bug fixes for 3.6.1 and, of course, continue to target new features for 3.7. This raises the issue of what process to follow for changes to the 3.6 branch following the b4 tag and up to the final 3.6.0 release. During the last feature release cycle (3.5.0), we tried something a little bit different: having a 3.5 branch open during the beta phase and then using a special Bitbucket repo with pull requests for changes in the release candidate phase. This was intended to ease the burden on us release managers trying to cherry pick a set of fixes into the RC and/or final releases while allowing the 3.5 branch to remain open to everyone for 3.5.1 changes. My sense is that, while we all appreciated keeping the branch open for bug fixes, many of us were new to the Bitbucket PR process and found it more confusing than expected. So for 3.6.0, I am trying to avoid going down that route. But I do need your help. First, throughout the release cycle I have tried to set expectations that the release candidate will be exactly what it says, a candidate for final release, and thus there should be at most a *very* small number of release critical fixes and final doc updates going in after b4 and with the goal of no changes after rc1. Second, by keeping the rc phase changes to a minimum, I have also tried to keep the release candidate phase shorter so that we all can move on to focus on the next feature release and on maintenance releases for 3.6 (schedules for both will be proposed soon). Also, 3.6.0 is the last feature release cycle using our current development workflow. Sometime after the 3.6.0 release, we will be phasing in our revised development workflow (thanks to Brett and crew!) and with it will come the opportunity for greater flexibility throughout the release cycle, with new tools like pull requests workflows and continuous integration testing. So I don't think it makes sense to spend a lot of time tweaking our current process just for the final weeks of one release. Therefore, what I am planning to do and asking you to do is: 1. Sometime after 1800 UTC today when we are ready to start the 3.6.0b4 tagging and release manufacturing process, I will send an email announcement. 2. Following that b4 tag notification and until rc1 is tagged on 2016-12-05 (in about two weeks), the 3.6 branch will *temporarily* be open only for fixes appropriate for 3.6.0, e.g. reviewed release critical items and final doc changes. Don't forget to mark such items as "release blocker" in the bug tracker. 3. Hold off pushing changes that are not 3.6.0 release critical until after rc1 is tagged. You can continue to push bug fixes as appropriate to other open branches if you are willing to back port them to the 3.6 branch, as necessary, after rc1 is tagged. Or, if you prefer to follow the normal development flow, take a bugfix holiday for the next two weeks and just hold off pushing any changes that affect 3.6.1 until after rc1 is tagged. With those (hopefully) small inconveniences to you, we won't have to go through a special Bitbucket PR process and it will ensure that our 3.6 buildbots continue to test what's going into 3.6.0. 4. Once rc1 is tagged, the 3.6 branch will again be open for the usual maintenance-release-appropriate changes destined for 3.6.1. Any emergency fixes that might arise after rc1 and prior to 3.6.0 will be approved and merged from the 3.6 branch into the 3.6.0 release by me. I'm really hoping we won't have to do that! I hope that you find the process for the few remaining weeks leading up to the 3.6.0 release to not be too burdensome. Please contact me if you have any questions about the 3.6.0 schedule or about whether a change is appropriate at this point in the 3.6.0 cycle. To recap, the remaining milestones for 3.6.0: 2016-11-21, ~1800 UTC: 3.6.0 beta 4 (important bug fixes and doc fixes) 2016-11-21 to 2016-12-05: 3.6 branch open *only* for 3.6.0 release critical and doc fixes 2016-12-05 (2 weeks from now) 3.6.0 release candidate 1 (3.6.0 code freeze, release critical bug fixes, doc fixes; 3.6 branch reopens for 3.6.1) 2016-12-16 3.6.0 release (3.6.0rc1 plus any necessary emergency fixes) Thank you all again for your efforts so far on 3.6. Beethoven and I are looking forward to celebrating 3.6.0 on 12-16! --Ned http://cpython-devguide.readthedocs.io/en/latest/devcycle.html#release-candidate-rc https://www.python.org/dev/peps/pep-0494/ -- Ned Deily nad at python.org -- [] From yselivanov.ml at gmail.com Mon Nov 21 16:57:05 2016 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 21 Nov 2016 16:57:05 -0500 Subject: [Python-Dev] [Python-checkins] Daily reference leaks (b78574cb00ab): sum=1120 In-Reply-To: References: <20161109090247.50140.35000.CEB8D404@psf.io> Message-ID: <61773042-543a-8593-8dd0-65e95fe20fd9@gmail.com> On 2016-11-19 2:46 PM, Kevin Modzelewski wrote: > Hi Yury, you may be interested in some leak-finding code that wrote for > Pyston. It uses the GC infrastructure to show you objects that were > directly leaked, ignoring indirect leaks -- ie objects that were only > leaked because they were referenced by a leaked object. It can often give > you a very small list of objects to look into (depending on how many non-gc > objects were leaked). If you're interested I can try porting it to CPython. > > https://github.com/dropbox/pyston/blob/master/from_cpython/Modules/gcmodule.c#L894 Hi Kevin, Nice work! Anything to help to find refleaks is welcome, it's really a painful process sometimes. As for porting it to CPython - I don't see why not. We can add this new API to debug builds only, at least in the beginning. Thank you! Yury From nad at python.org Mon Nov 21 23:06:24 2016 From: nad at python.org (Ned Deily) Date: Mon, 21 Nov 2016 23:06:24 -0500 Subject: [Python-Dev] IMPORTANT: 3.6 branch in Release Candidate stage, check-ins restricted! In-Reply-To: <541563DC-61B5-4DE8-91DA-11DC68B667B8@python.org> References: <541563DC-61B5-4DE8-91DA-11DC68B667B8@python.org> Message-ID: The tagging and manufacturing of 3.6.0 beta 4 is now underway and we are entering the Release Candidate stage. From now until further notice, you should only push changes to 3.6 that meet the 3.6.0 release critical or final doc changes criteria described below. All code changes pushed to 3.6 during this release candidate stage must be reviewed. The restriction to only release-critical fixes will be lifted after the tagging of 3.6.0rc1 which is expected in about 2 weeks. As usual, the actual v3.6.0b4 tag and other release-related updates will be pushed to the cpython repo once the release build process has been completed and tested. This will be signaled by the 3.6.0b4 release availability email. By the way: I have just noticed that I made an error in the cleanup steps following the previous beta, 3.6.0b3. The patch level for post-b3 development builds was mistakenly set to 3.6.0b4+ instead of 3.6.0b3+. (The release tarballs and installers are not affected.) So don't be surprised when the patch level for post-b4 development builds remains 3.6.0b4+. As far as I can anticipate that should not cause any problems. My apologies! --Ned On Nov 21, 2016, at 05:35, Ned Deily wrote: > The final beta snapshot planned for the 3.6 release cycle is scheduled to be tagged today within the next 12 hours. We then begin the Release Candidate stage, the final days leading up to 3.6.0, with the release candidate scheduled to be tagged in about 2 weeks. As I have noted previously, with the tagging today and release of beta 4, *all* non-critical bug fixes for 3.6.0 should now have been checked in. From the b4 tag on, *only* release critical and doc fixes should be pushed for the release candidate. A reminder from the Developer's Guide regarding the Release Candidate stage of the release cycle: > > "A branch preparing for an RC release can only have bugfixes applied that have been reviewed by other core developers. Generally, these issues must be severe enough (e.g. crashes) that they deserve fixing before the final release. All other issues should be deferred to the next development cycle, since stability is the strongest concern at this point. You cannot skip the peer review during an RC, no matter how small! Even if it is a simple copy-and-paste change, everything requires peer review from a core developer." > > A goal for this release has been to have no changes between the release candidate and the final release other than the tag itself. Remember that every time we make a change at this point, it puts a burden on and adds risk to our testing efforts and, more importantly, the efforts of our third-party package developers, downstream distributors, and end users helping us ensure that 3.6.0 will be a high-quality release. I think everyone has been doing a great job so far in exercising good judgement about what is appropriate at each stage of our release cycle. Thank you! Now, after b4, unless a code change addresses a truly release critical issue, please target bug fixes for 3.6.1 and, of course, continue to target new features for 3.7. > > This raises the issue of what process to follow for changes to the 3.6 branch following the b4 tag and up to the final 3.6.0 release. During the last feature release cycle (3.5.0), we tried something a little bit different: having a 3.5 branch open during the beta phase and then using a special Bitbucket repo with pull requests for changes in the release candidate phase. This was intended to ease the burden on us release managers trying to cherry pick a set of fixes into the RC and/or final releases while allowing the 3.5 branch to remain open to everyone for 3.5.1 changes. My sense is that, while we all appreciated keeping the branch open for bug fixes, many of us were new to the Bitbucket PR process and found it more confusing than expected. So for 3.6.0, I am trying to avoid going down that route. But I do need your help. First, throughout the release cycle I have tried to set expectations that the release candidate will be exactly what it says, a candidate for final release, and thus there should be at most a *very* small number of release critical fixes and final doc updates going in after b4 and with the goal of no changes after rc1. Second, by keeping the rc phase changes to a minimum, I have also tried to keep the release candidate phase shorter so that we all can move on to focus on the next feature release and on maintenance releases for 3.6 (schedules for both will be proposed soon). Also, 3.6.0 is the last feature release cycle using our current development workflow. Sometime after the 3.6.0 release, we will be phasing in our revised development workflow (thanks to Brett and crew!) and with it will come the opportunity for greater flexibility throughout the release cycle, with new tools like pull requests workflows and continuous integration testing. So I don't think it makes sense to spend a lot of time tweaking our current process just for the final weeks of one release. > > Therefore, what I am planning to do and asking you to do is: > > 1. Sometime after 1800 UTC today when we are ready to start the 3.6.0b4 tagging and release manufacturing process, I will send an email announcement. > > 2. Following that b4 tag notification and until rc1 is tagged on 2016-12-05 (in about two weeks), the 3.6 branch will *temporarily* be open only for fixes appropriate for 3.6.0, e.g. reviewed release critical items and final doc changes. Don't forget to mark such items as "release blocker" in the bug tracker. > > 3. Hold off pushing changes that are not 3.6.0 release critical until after rc1 is tagged. You can continue to push bug fixes as appropriate to other open branches if you are willing to back port them to the 3.6 branch, as necessary, after rc1 is tagged. Or, if you prefer to follow the normal development flow, take a bugfix holiday for the next two weeks and just hold off pushing any changes that affect 3.6.1 until after rc1 is tagged. With those (hopefully) small inconveniences to you, we won't have to go through a special Bitbucket PR process and it will ensure that our 3.6 buildbots continue to test what's going into 3.6.0. > > 4. Once rc1 is tagged, the 3.6 branch will again be open for the usual maintenance-release-appropriate changes destined for 3.6.1. Any emergency fixes that might arise after rc1 and prior to 3.6.0 will be approved and merged from the 3.6 branch into the 3.6.0 release by me. I'm really hoping we won't have to do that! > > I hope that you find the process for the few remaining weeks leading up to the 3.6.0 release to not be too burdensome. Please contact me if you have any questions about the 3.6.0 schedule or about whether a change is appropriate at this point in the 3.6.0 cycle. > > To recap, the remaining milestones for 3.6.0: > > 2016-11-21, ~1800 UTC: 3.6.0 beta 4 (important bug fixes and doc fixes) > > 2016-11-21 to 2016-12-05: 3.6 branch open *only* for 3.6.0 release critical and doc fixes > > 2016-12-05 (2 weeks from now) 3.6.0 release candidate 1 (3.6.0 code freeze, release critical bug fixes, doc fixes; 3.6 branch reopens for 3.6.1) > > 2016-12-16 3.6.0 release (3.6.0rc1 plus any necessary emergency fixes) > > Thank you all again for your efforts so far on 3.6. Beethoven and I are looking forward to celebrating 3.6.0 on 12-16! > > --Ned > > http://cpython-devguide.readthedocs.io/en/latest/devcycle.html#release-candidate-rc > > https://www.python.org/dev/peps/pep-0494/ -- Ned Deily nad at python.org -- [] From nad at python.org Tue Nov 22 02:02:51 2016 From: nad at python.org (Ned Deily) Date: Tue, 22 Nov 2016 02:02:51 -0500 Subject: [Python-Dev] [RELEASE] Python 3.6.0b4 is now available Message-ID: On behalf of the Python development community and the Python 3.6 release team, I'm pleased to announce the availability of Python 3.6.0b4. 3.6.0b4 is the last planned beta release of Python 3.6, the next major release of Python. Among the new major new features in Python 3.6 are: * PEP 468 - Preserving the order of **kwargs in a function * PEP 487 - Simpler customization of class creation * PEP 495 - Local Time Disambiguation * PEP 498 - Literal String Formatting * PEP 506 - Adding A Secrets Module To The Standard Library * PEP 509 - Add a private version to dict * PEP 515 - Underscores in Numeric Literals * PEP 519 - Adding a file system path protocol * PEP 520 - Preserving Class Attribute Definition Order * PEP 523 - Adding a frame evaluation API to CPython * PEP 524 - Make os.urandom() blocking on Linux (during system startup) * PEP 525 - Asynchronous Generators (provisional) * PEP 526 - Syntax for Variable Annotations (provisional) * PEP 528 - Change Windows console encoding to UTF-8 * PEP 529 - Change Windows filesystem encoding to UTF-8 * PEP 530 - Asynchronous Comprehensions Please see "What?s New In Python 3.6" for more information: https://docs.python.org/3.6/whatsnew/3.6.html You can find Python 3.6.0b4 here: https://www.python.org/downloads/release/python-360b4/ Beta releases are intended to give the wider community the opportunity to test new features and bug fixes and to prepare their projects to support the new feature release. We strongly encourage maintainers of third-party Python projects to test with 3.6 during the beta phase and report issues found to bugs.python.org as soon as possible. While the release is feature complete entering the beta phase, it is possible that features may be modified or, in rare cases, deleted up until the start of the release candidate phase (2016-12-05). Our goal is have no changes after rc1. To achieve that, it will be extremely important to get as much exposure for 3.6 as possible during the beta phase. Please keep in mind that this is a preview release and its use is not recommended for production environments The next pre-release of Python 3.6 will be 3.6.0rc1, the release candidate, currently scheduled for 2016-12-05. The official release of Python 3.6.0 is currently scheduled for 2016-12-16. More information about the release schedule can be found here: https://www.python.org/dev/peps/pep-0494/ -- Ned Deily nad at python.org -- [] From nad at python.org Tue Nov 22 02:24:37 2016 From: nad at python.org (Ned Deily) Date: Tue, 22 Nov 2016 02:24:37 -0500 Subject: [Python-Dev] 3.6 branch now open only for 3.6.0 release critical fixes and doc updates! In-Reply-To: References: Message-ID: <9BB9B066-3FE6-4546-9ED9-770DBA691513@python.org> On Nov 22, 2016, at 02:02, Ned Deily wrote: > On behalf of the Python development community and the Python 3.6 release > team, I'm pleased to announce the availability of Python 3.6.0b4. 3.6.0b4 > is the last planned beta release of Python 3.6, the next major release of > Python. [...] OK, all of the release engineering for 3.6.0b4 is complete. The 3.6 branch in the cpython repo is now available again but, as noted, *only* for reviewed release critical fixes appropriate for the 3.6.0 final and for final 3.6.0 doc updates! Again, until 3.6.0rc1 is tagged, scheduled to happen in two weeks on 2016-12-05, please do *not* push non release-critical code of any kind to the 3.6 branch; after rc1 is tagged, the 3.6 branch will be open for 3.6.1 changes. Please use these two weeks to thoroughly review and, as necessary, update the What's New In Python 3.6 document (thanks, Elvis and Yuri, for editing it once again!) and all other release documentation. Any additional testing you can do and/or encourage others to do of the new and old components of 3.6 and with with third-party packages will be appreciated. Please document anything you think *might* be a release critical problem in the bug tracker marking them as "release blocker". Assuming no unresolved release critical problems arise, the final two steps will be the release candidate in 2 weeks and then, again assuming no release critical problems are identified in it, the final release on 12-16. Please contact me if you have any questions about what may or may not be appropriate to push during these final days before the release. So close! --Ned -- Ned Deily nad at python.org -- [] From victor.stinner at gmail.com Tue Nov 22 09:57:26 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 22 Nov 2016 15:57:26 +0100 Subject: [Python-Dev] [python-committers] 3.6 branch now open only for 3.6.0 release critical fixes and doc updates! In-Reply-To: <9BB9B066-3FE6-4546-9ED9-770DBA691513@python.org> References: <9BB9B066-3FE6-4546-9ED9-770DBA691513@python.org> Message-ID: Hi, 2016-11-22 8:24 GMT+01:00 Ned Deily : > OK, all of the release engineering for 3.6.0b4 is complete. The 3.6 branch in the cpython repo is now available again but, as noted, *only* for reviewed release critical fixes appropriate for the 3.6.0 final and for final 3.6.0 doc updates! Sorry, I pushed changes before reading your email(s). I expected that release critical changes would have to be pushed to a different repository using cherry-pick or something else. To be clear: Python 3.5 will be frozen as well? Pushing to 3.5 requires to merge into 3.6 (and then merge into default). Four changes have been pushed after the tag: (*) https://hg.python.org/cpython/rev/4f6fb9e47f6b Issue #28023: Fix python-gdb.py didn't support new dict implementation [#28023] I reviewed Naoki's patch and it LGTM. python-gdb.py was simply 100% broken without this fix, and I consider that this tool is important for Python (but I didn't understood correctly the RC rules, sorry)! By the way, I also broke python-gdb.py with fast calls! I started to work on a fix, http://bugs.python.org/issue28770 (*) https://hg.python.org/cpython/rev/9b974f988c95 Issue #28023: Fix python-gdb.py on old GDB versions ... My fix for the previous commit. Sadly, it's hard to test on all GDB versions, but buildbots are here for that :-) (*) https://hg.python.org/cpython/rev/6b43d15fd2d7 Issue #28727: Fix typo in pattern_richcompare() Obvious bugfix spotted by Serhiy. (*) https://hg.python.org/cpython/rev/c2cb70c97163 Issue #28727: Optimize pattern_richcompare() for a==a This one is a minor optimization suggested by Serhiy. Should I revert these changes? Or can someone please review them one more time? Victor From stefan.scherfke at energymeteo.de Tue Nov 22 06:52:59 2016 From: stefan.scherfke at energymeteo.de (Stefan Scherfke) Date: Tue, 22 Nov 2016 12:52:59 +0100 Subject: [Python-Dev] Bus error in Python 3.6.0beta Message-ID: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> Hi all, I am trying to build a custom Conda installer for Python 3.6.0b4. I could successfully build an run Python. However, when I run the generated Conda installer, it dies with a "Bus error". It happens when Conda's meta-installer script tries to replace the build-prefix (e.g., /home/stefan/conda/asdf) with the prefix of the actual installation (e.g., /tmp/py36). Therefore, it opens all files in binary mode, reads their contents, replaces stuff, and writes the new contents back to the original file. This works fine with text files but it dies when it hits the first binary file. Here is a minimal example that reproduces the error: $ /tmp/py36/bin/python Python 3.6.0b4 (default, Nov 22 2016, 10:32:29) [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux >>> >>> path = '/tmp/py36/lib/libpython3.6m.so.1.0' >>> f = open(path, 'wb') BusError Can anyone reproduce this? Is this a compilation error or an issue with Python itself? Cheers, Stefan PS: If need, I can upload the Conda installer somewhere. It's ~40MiB. From stefan.scherfke at energymeteo.de Tue Nov 22 11:00:21 2016 From: stefan.scherfke at energymeteo.de (Stefan Scherfke) Date: Tue, 22 Nov 2016 17:00:21 +0100 Subject: [Python-Dev] Bus error in Python 3.6.0beta In-Reply-To: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> References: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> Message-ID: On 22.11.2016 12:52, Stefan Scherfke wrote: > > Here is a minimal example that reproduces the error: > > $ /tmp/py36/bin/python > Python 3.6.0b4 (default, Nov 22 2016, 10:32:29) > [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux >>>> >>>> path = '/tmp/py36/lib/libpython3.6m.so.1.0' >>>> f = open(path, 'wb') > BusError > This can, of course, not work. But the Conda installer does something like this and it works on Python 3.5 but not on 3.6. I'm a bit baffled... Cheers, Stefan From solipsis at pitrou.net Tue Nov 22 12:03:31 2016 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 22 Nov 2016 18:03:31 +0100 Subject: [Python-Dev] Bus error in Python 3.6.0beta References: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> Message-ID: <20161122180331.1527f6bb@fsol> On Tue, 22 Nov 2016 12:52:59 +0100 Stefan Scherfke wrote: > Hi all, > > I am trying to build a custom Conda installer for Python 3.6.0b4. > > I could successfully build an run Python. However, when I run > the generated Conda installer, it dies with a "Bus error". > > It happens when Conda's meta-installer script tries to replace > the build-prefix (e.g., /home/stefan/conda/asdf) with the prefix > of the actual installation (e.g., /tmp/py36). > > Therefore, it opens all files in binary mode, reads their contents, > replaces stuff, and writes the new contents back to the original file. > > This works fine with text files but it dies when it hits the first > binary file. > > Here is a minimal example that reproduces the error: > > $ /tmp/py36/bin/python > Python 3.6.0b4 (default, Nov 22 2016, 10:32:29) > [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux > >>> > >>> path = '/tmp/py36/lib/libpython3.6m.so.1.0' > >>> f = open(path, 'wb') > BusError You'll have to investigate yourself. First enable core dumps ("ulimit -c unlimited"), then run gdb on the resulting core dump. (or run Python directly under gdb) Regards Antoine. From songofacandy at gmail.com Tue Nov 22 12:53:15 2016 From: songofacandy at gmail.com (INADA Naoki) Date: Wed, 23 Nov 2016 02:53:15 +0900 Subject: [Python-Dev] Bus error in Python 3.6.0beta In-Reply-To: References: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> Message-ID: On Wed, Nov 23, 2016 at 1:00 AM, Stefan Scherfke wrote: > > > On 22.11.2016 12:52, Stefan Scherfke wrote: >> >> >> Here is a minimal example that reproduces the error: >> >> $ /tmp/py36/bin/python >> Python 3.6.0b4 (default, Nov 22 2016, 10:32:29) >> [GCC 6.2.1 20160916 (Red Hat 6.2.1-2)] on linux >>>>> >>>>> >>>>> path = '/tmp/py36/lib/libpython3.6m.so.1.0' >>>>> f = open(path, 'wb') >> >> BusError >> > > This can, of course, not work. But the Conda installer does something like > this and it works on Python 3.5 but not on 3.6. I'm a bit baffled... > I don't know about Conda. But this BusError is normal behavior of Linux. While executing binary, truncating the binary cause BusError. I can reproduce it with Python 3.5 $ ldd bin/python3.5 ... libpython3.5m.so.1.0 => /home/inada-n/local/py35/lib/libpython3.5m.so.1.0 (0x00007fdefa188000) ... $ ll lib/ ... lrwxrwxrwx 1 inada-n inada-n 20 11? 23 02:46 libpython3.5m.so -> libpython3.5m.so.1.0* -r-xr-xr-x 1 inada-n inada-n 12497584 11? 23 02:46 libpython3.5m.so.1.0* ... $ chmod u+w lib/libpython3.5m.so.1.0 $ bin/python3.5 Python 3.5.2+ (3.5:0ee76f3afd70, Nov 23 2016, 02:39:08) [GCC 6.2.0 20161005] on linux Type "help", "copyright", "credits" or "license" for more information. >>> f = open('./lib/libpython3.5m.so.1.0', 'wb') Bus error (core dumped) -- INADA Naoki From nad at python.org Tue Nov 22 13:53:28 2016 From: nad at python.org (Ned Deily) Date: Tue, 22 Nov 2016 13:53:28 -0500 Subject: [Python-Dev] Bus error in Python 3.6.0beta In-Reply-To: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> References: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> Message-ID: On Nov 22, 2016, at 06:52, Stefan Scherfke wrote: > /tmp/py36/bin/python In addition to the other comments, I'd check whether the new python is built with --enable-shared. If so, it's probably not a good idea to be writing to its library while running whereas, if it's not a shared build, it may not matter. -- Ned Deily nad at python.org -- [] From victor.stinner at gmail.com Tue Nov 22 14:20:51 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Tue, 22 Nov 2016 20:20:51 +0100 Subject: [Python-Dev] Bus error in Python 3.6.0beta In-Reply-To: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> References: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> Message-ID: It's not bug but a feature :-) Python doesn't protect yourself against mistakes :-) Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Tue Nov 22 14:25:56 2016 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 23 Nov 2016 06:25:56 +1100 Subject: [Python-Dev] Bus error in Python 3.6.0beta In-Reply-To: References: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> Message-ID: On Wed, Nov 23, 2016 at 6:20 AM, Victor Stinner wrote: > It's not bug but a feature :-) Python doesn't protect yourself against > mistakes :-) AIUI the normal way to protect yourself is to unlink (remove) the file and create it from scratch, rather than truncate it. ChrisA From steve.dower at python.org Tue Nov 22 19:02:28 2016 From: steve.dower at python.org (Steve Dower) Date: Tue, 22 Nov 2016 16:02:28 -0800 Subject: [Python-Dev] Python 3.6.0 download stats Message-ID: For anyone who's interested, I ran a quick analysis of the download stats from python.org for CPython 3.6.0 releases since September. As Mac and Linux users typically get their downloads from elsewhere, those stats are on the low side. However, we've been getting roughly 10k downloads/day for Windows, and even though it's been out less than 24 hours, 3.6.0b4 is already There's a PDF with charts at: https://pythonbuilds.blob.core.windows.net/temp/Python%203.6.0%20Beta%20Downloads.pdf My spreadsheet is at https://pythonbuilds.blob.core.windows.net/temp/360Beta%20Analysis.xlsx (Excel format), if you want to see actual numbers. But there is likely some lost data in here and almost certainly some hits counted that shouldn't be (e.g. the spike on the 19th of September is *probably* a broken browser update causing multiple hits for each download - I haven't gone through UA data to figure it out yet, but Chrome has definitely done this in the past). Treat the actual numbers with a grain of salt, but overall it looks like we've had pretty good usage of the prerelease versions of 3.6. Cheers, Steve From raymond.hettinger at gmail.com Tue Nov 22 23:25:50 2016 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Tue, 22 Nov 2016 20:25:50 -0800 Subject: [Python-Dev] [python-committers] 3.6 branch now open only for 3.6.0 release critical fixes and doc updates! In-Reply-To: References: <9BB9B066-3FE6-4546-9ED9-770DBA691513@python.org> Message-ID: <8AF4D9BD-1FA9-43A6-92C4-987B9748F6EE@gmail.com> > On Nov 22, 2016, at 6:57 AM, Victor Stinner wrote: > > Should I revert these changes? I don't think reverting any of these would improve the release. I vote for them to stay. Raymond From victor.stinner at gmail.com Wed Nov 23 01:43:36 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 23 Nov 2016 07:43:36 +0100 Subject: [Python-Dev] [python-committers] 3.6 branch now open only for 3.6.0 release critical fixes and doc updates! In-Reply-To: <8AF4D9BD-1FA9-43A6-92C4-987B9748F6EE@gmail.com> References: <9BB9B066-3FE6-4546-9ED9-770DBA691513@python.org> <8AF4D9BD-1FA9-43A6-92C4-987B9748F6EE@gmail.com> Message-ID: Ok, thank you Raymond for checking. Victor Le 23 nov. 2016 05:25, "Raymond Hettinger" a ?crit : > > > On Nov 22, 2016, at 6:57 AM, Victor Stinner > wrote: > > > > Should I revert these changes? > > I don't think reverting any of these would improve the release. > I vote for them to stay. > > > Raymond > > > > > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan.scherfke at energymeteo.de Wed Nov 23 07:41:20 2016 From: stefan.scherfke at energymeteo.de (Stefan Scherfke) Date: Wed, 23 Nov 2016 13:41:20 +0100 Subject: [Python-Dev] Bus error in Python 3.6.0beta In-Reply-To: <20161122180331.1527f6bb@fsol> References: <7a06603d-42e0-46f0-29fa-c2772820800f@energymeteo.de> <20161122180331.1527f6bb@fsol> Message-ID: On 22.11.2016 18:03, Antoine Pitrou wrote: > You'll have to investigate yourself. First enable core dumps ("ulimit > -c unlimited"), then run gdb on the resulting core dump. > > (or run Python directly under gdb) It's seems indeed not to be a Problem with 3.6. I now have it with newer 3.5 builds, too. It currently looks like it has something to do with my Conda build env. The libpython files of older builds don't have a build prefix (like /opt/conda/placehold_placehold_...) in them while newer builds do. I am still investigating, but Python does not seem to be the culprit :) Cheers, Stefan > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/stefan.scherfke%40energymeteo.de > From rwilliams at lyft.com Wed Nov 23 21:11:02 2016 From: rwilliams at lyft.com (Roy Williams) Date: Wed, 23 Nov 2016 18:11:02 -0800 Subject: [Python-Dev] Expose environment variable for Py_Py3kWarningFlag (Issue 28288) Message-ID: Howdy all, I'd love to be able to get http://bugs.python.org/issue28288 into 2.7.12 . I've found the -3 flag to be immensely useful when porting Python code to Python 3, but unfortunately it can be difficult to use with services that run python in subprocesses (like Gunicorn or Xdist with py.test). With this patch I'd be able to set the `PYTHON3WARNINGS` environment variable to ensure I get warnings everywhere. Thanks, let me know what you think, Roy -------------- next part -------------- An HTML attachment was scrubbed... URL: From victor.stinner at gmail.com Fri Nov 25 04:28:46 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Fri, 25 Nov 2016 10:28:46 +0100 Subject: [Python-Dev] Does anyone use CALL_PROFILE special build? Message-ID: Hi, I proposed a patch to drop the CALL_PROFILE special build to simplify Python/ceval.c: http://bugs.python.org/issue28799 I modified ceval.c for fast calls in Python 3.6, and I'm not sure that the feature still works correctly. The feature is not well documented and not tested at all. For fast calls, I moved some code, so it's more complex to update call statistics since statistics are implemented in ceval.c. I would like to completely remove the feature to be able to implement further optimizations. Victor From status at bugs.python.org Fri Nov 25 12:08:52 2016 From: status at bugs.python.org (Python tracker) Date: Fri, 25 Nov 2016 18:08:52 +0100 (CET) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20161125170852.E66B15657E@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2016-11-18 - 2016-11-25) Python tracker at http://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 5588 ( +4) closed 34971 (+62) total 40559 (+66) Open issues with patches: 2421 Issues opened (40) ================== #10414: Python does not work on an IPv6 only host http://bugs.python.org/issue10414 reopened by gregory.p.smith #27100: Attempting to use class with both __enter__ & __exit__ undefin http://bugs.python.org/issue27100 reopened by martin.panter #28739: PEP 498: docstrings as f-strings http://bugs.python.org/issue28739 opened by yselivanov #28740: Add sys.getandroidapilevel() http://bugs.python.org/issue28740 opened by haypo #28742: argparse.ArgumentDefaultsHelpFormatter sometimes provides inac http://bugs.python.org/issue28742 opened by pde #28747: Expose SSL_CTX_set_cert_verify_callback http://bugs.python.org/issue28747 opened by steve.dower #28749: Fixed the documentation of the mapping codec APIs http://bugs.python.org/issue28749 opened by serhiy.storchaka #28753: Clinic: Converting Your First Function is not up to date http://bugs.python.org/issue28753 opened by mdk #28754: Argument Clinic for bisect.bisect_left http://bugs.python.org/issue28754 opened by mdk #28755: Rework syntax highlighing in howto/clinic.rst http://bugs.python.org/issue28755 opened by mdk #28758: UnicodeDecodeError: 'gbk' codec can't decode byte 0xab in posi http://bugs.python.org/issue28758 opened by dontbugme #28759: access to mkfifo, mknod and hard links is controled by SELinux http://bugs.python.org/issue28759 opened by xdegaye #28762: configure links with lockf and F_LOCK is not declared in Andro http://bugs.python.org/issue28762 opened by xdegaye #28763: Use en-dashes for ranges in docs http://bugs.python.org/issue28763 opened by serhiy.storchaka #28764: test_mailbox fails when run as a non-root user on Android API http://bugs.python.org/issue28764 opened by xdegaye #28765: _sre.compile(): be more strict on types of indexgroup and grou http://bugs.python.org/issue28765 opened by haypo #28767: Readd __index__ support on ipaddress objects http://bugs.python.org/issue28767 opened by josh.r #28768: Warning: implicit declaration of function '_setmode' http://bugs.python.org/issue28768 opened by masamoto #28769: Make PyUnicode_AsUTF8 returning "const char *" rather of "char http://bugs.python.org/issue28769 opened by serhiy.storchaka #28770: Update python-gdb.py for fastcalls http://bugs.python.org/issue28770 opened by haypo #28771: Update documented signatures of tp_get/setattr http://bugs.python.org/issue28771 opened by martin.panter #28775: Option to set startup directory in IDLE http://bugs.python.org/issue28775 opened by rhettinger #28777: asinc iter queue http://bugs.python.org/issue28777 opened by RekGRpth #28778: wsgiref HTTP Response Header Injection: CRLF Injection http://bugs.python.org/issue28778 opened by RAUSHAN RAJ #28779: set_forkserver_preload() can crash the forkserver if preloaded http://bugs.python.org/issue28779 opened by pitrou #28780: netrc throws NetrcParseError for record without 'password' http://bugs.python.org/issue28780 opened by Mark Wood #28781: On Installation of 3.5 Python get error message http://bugs.python.org/issue28781 opened by Marcopolo #28784: shlex.shlex punctuation_chars documentation should use posix=T http://bugs.python.org/issue28784 opened by evan_ #28785: Clarify the behavior of NotImplemented http://bugs.python.org/issue28785 opened by max #28787: Out of tree --with--dtrace builds fail with a traceback http://bugs.python.org/issue28787 opened by cstratak #28788: Feature request: ConfigParser should be able to write config t http://bugs.python.org/issue28788 opened by georgefischhof #28789: valgrind shows "invalid file descriptor" when calling platform http://bugs.python.org/issue28789 opened by Mathieu_Du #28790: Error when using Generic and __slots__ http://bugs.python.org/issue28790 opened by gvanrossum #28791: update sqlite to 3.15.2 http://bugs.python.org/issue28791 opened by Big Stone #28794: inspect.isasyncgen and inspect.isasyncgenfunction aren't docum http://bugs.python.org/issue28794 opened by njs #28795: Misleading stating, that SIGINT handler is installed by defaul http://bugs.python.org/issue28795 opened by Jan Veleck?? #28797: Modifying class __dict__ inside __set_name__ http://bugs.python.org/issue28797 opened by Wombatz #28799: Drop CALL_PROFILE special build? http://bugs.python.org/issue28799 opened by haypo #28800: Add RETURN_NONE bytecode instruction http://bugs.python.org/issue28800 opened by haypo #28802: Python 3.6.0b4 Reports ncurses present in Cygwin but fails to http://bugs.python.org/issue28802 opened by eclectic9509 Most recent 15 issues with no replies (15) ========================================== #28802: Python 3.6.0b4 Reports ncurses present in Cygwin but fails to http://bugs.python.org/issue28802 #28799: Drop CALL_PROFILE special build? http://bugs.python.org/issue28799 #28791: update sqlite to 3.15.2 http://bugs.python.org/issue28791 #28790: Error when using Generic and __slots__ http://bugs.python.org/issue28790 #28789: valgrind shows "invalid file descriptor" when calling platform http://bugs.python.org/issue28789 #28788: Feature request: ConfigParser should be able to write config t http://bugs.python.org/issue28788 #28787: Out of tree --with--dtrace builds fail with a traceback http://bugs.python.org/issue28787 #28784: shlex.shlex punctuation_chars documentation should use posix=T http://bugs.python.org/issue28784 #28780: netrc throws NetrcParseError for record without 'password' http://bugs.python.org/issue28780 #28778: wsgiref HTTP Response Header Injection: CRLF Injection http://bugs.python.org/issue28778 #28777: asinc iter queue http://bugs.python.org/issue28777 #28762: configure links with lockf and F_LOCK is not declared in Andro http://bugs.python.org/issue28762 #28759: access to mkfifo, mknod and hard links is controled by SELinux http://bugs.python.org/issue28759 #28758: UnicodeDecodeError: 'gbk' codec can't decode byte 0xab in posi http://bugs.python.org/issue28758 #28755: Rework syntax highlighing in howto/clinic.rst http://bugs.python.org/issue28755 Most recent 15 issues waiting for review (15) ============================================= #28800: Add RETURN_NONE bytecode instruction http://bugs.python.org/issue28800 #28799: Drop CALL_PROFILE special build? http://bugs.python.org/issue28799 #28797: Modifying class __dict__ inside __set_name__ http://bugs.python.org/issue28797 #28794: inspect.isasyncgen and inspect.isasyncgenfunction aren't docum http://bugs.python.org/issue28794 #28787: Out of tree --with--dtrace builds fail with a traceback http://bugs.python.org/issue28787 #28779: set_forkserver_preload() can crash the forkserver if preloaded http://bugs.python.org/issue28779 #28771: Update documented signatures of tp_get/setattr http://bugs.python.org/issue28771 #28770: Update python-gdb.py for fastcalls http://bugs.python.org/issue28770 #28769: Make PyUnicode_AsUTF8 returning "const char *" rather of "char http://bugs.python.org/issue28769 #28768: Warning: implicit declaration of function '_setmode' http://bugs.python.org/issue28768 #28765: _sre.compile(): be more strict on types of indexgroup and grou http://bugs.python.org/issue28765 #28764: test_mailbox fails when run as a non-root user on Android API http://bugs.python.org/issue28764 #28763: Use en-dashes for ranges in docs http://bugs.python.org/issue28763 #28762: configure links with lockf and F_LOCK is not declared in Andro http://bugs.python.org/issue28762 #28759: access to mkfifo, mknod and hard links is controled by SELinux http://bugs.python.org/issue28759 Top 10 most discussed issues (10) ================================= #28765: _sre.compile(): be more strict on types of indexgroup and grou http://bugs.python.org/issue28765 15 msgs #28747: Expose SSL_CTX_set_cert_verify_callback http://bugs.python.org/issue28747 13 msgs #27100: Attempting to use class with both __enter__ & __exit__ undefin http://bugs.python.org/issue27100 12 msgs #28740: Add sys.getandroidapilevel() http://bugs.python.org/issue28740 12 msgs #28754: Argument Clinic for bisect.bisect_left http://bugs.python.org/issue28754 11 msgs #28763: Use en-dashes for ranges in docs http://bugs.python.org/issue28763 11 msgs #28753: Clinic: Converting Your First Function is not up to date http://bugs.python.org/issue28753 10 msgs #28518: execute("begin immediate") throwing OperationalError http://bugs.python.org/issue28518 9 msgs #28728: test_host_resolution in test_socket fails http://bugs.python.org/issue28728 8 msgs #28739: PEP 498: docstrings as f-strings http://bugs.python.org/issue28739 8 msgs Issues closed (57) ================== #5384: mmap and exception type http://bugs.python.org/issue5384 closed by rhettinger #10656: "Out of tree" build fails on AIX http://bugs.python.org/issue10656 closed by martin.panter #17528: Implement dumps/loads for lru_cache http://bugs.python.org/issue17528 closed by rhettinger #17577: Add lookahead/peek wrapper to itertools http://bugs.python.org/issue17577 closed by rhettinger #22298: Lib/warnings.py _show_warning does not protect against being c http://bugs.python.org/issue22298 closed by terry.reedy #23106: Remove smalltable from set objects http://bugs.python.org/issue23106 closed by rhettinger #25659: ctypes.Array.from_buffer segmentation fault when trying to cre http://bugs.python.org/issue25659 closed by martin.panter #26530: tracemalloc: add C API to manually track/untrack memory alloca http://bugs.python.org/issue26530 closed by haypo #26647: ceval: use Wordcode, 16-bit bytecode http://bugs.python.org/issue26647 closed by haypo #27414: http.server.BaseHTTPRequestHandler inconsistence with Content- http://bugs.python.org/issue27414 closed by haypo #27825: Make the documentation for statistics' data argument clearer. http://bugs.python.org/issue27825 closed by rhettinger #28004: Optimize bytes.join(sequence) http://bugs.python.org/issue28004 closed by haypo #28023: python-gdb.py must be updated for the new Python 3.6 compact d http://bugs.python.org/issue28023 closed by inada.naoki #28082: re: convert re flags to (much friendlier) IntFlag constants http://bugs.python.org/issue28082 closed by python-dev #28172: Upper-case all example enum members http://bugs.python.org/issue28172 closed by python-dev #28475: Misleading error on random.sample when k < 0 http://bugs.python.org/issue28475 closed by rhettinger #28532: Show sys.version when -V option is supplied twice. http://bugs.python.org/issue28532 closed by inada.naoki #28548: http.server parse_request() bug and error reporting http://bugs.python.org/issue28548 closed by martin.panter #28587: list.index documentation missing start and stop arguments http://bugs.python.org/issue28587 closed by rhettinger #28666: Make test.support.rmtree() able to remove non-writable directo http://bugs.python.org/issue28666 closed by serhiy.storchaka #28687: Python 2.7.12 windows x64 installer fails after previous bad u http://bugs.python.org/issue28687 closed by steve.dower #28688: Warning -- warnings.filters was modified by test_warnings http://bugs.python.org/issue28688 closed by martin.panter #28705: Clean up design FAQ question about compiling to C http://bugs.python.org/issue28705 closed by brett.cannon #28714: Addition to Documentation of configparser.ConfigParser.write() http://bugs.python.org/issue28714 closed by georgefischhof #28715: Check result of PyUnicode_AsUTF8 http://bugs.python.org/issue28715 closed by serhiy.storchaka #28722: doctest example exit status http://bugs.python.org/issue28722 closed by berker.peksag #28727: Implement comparison (x==y and x!=y) for _sre.SRE_Pattern http://bugs.python.org/issue28727 closed by haypo #28732: spawnl crash on windows7 http://bugs.python.org/issue28732 closed by steve.dower #28736: multiprocessing.Lock() no longer has .acquire() http://bugs.python.org/issue28736 closed by Eric Leadbetter #28738: Document SIGBREAK as argument for signal() under Windows. http://bugs.python.org/issue28738 closed by berker.peksag #28741: PEP 498: single '}' is not allowed http://bugs.python.org/issue28741 closed by eric.smith #28743: test_choices_algorithms() in test_random uses lots of memory http://bugs.python.org/issue28743 closed by rhettinger #28744: Basic precision calc error http://bugs.python.org/issue28744 closed by zach.ware #28745: Python 3.5.2 "from ... import" statement is different from off http://bugs.python.org/issue28745 closed by eric.smith #28746: cannot set_inheritable() for a file descriptor on Android http://bugs.python.org/issue28746 closed by xdegaye #28748: Make _Py_PackageContext of type "const char *" http://bugs.python.org/issue28748 closed by serhiy.storchaka #28750: Replace string with bytes in doc of unicode-escape an raw-unic http://bugs.python.org/issue28750 closed by serhiy.storchaka #28751: Fix comments in code.h http://bugs.python.org/issue28751 closed by rhettinger #28752: datetime object fails to restore from reduction http://bugs.python.org/issue28752 closed by belopolsky #28756: robotfileparser always uses default Python user-agent http://bugs.python.org/issue28756 closed by xiang.zhang #28757: Installation Failure http://bugs.python.org/issue28757 closed by eric.smith #28760: Cleanup PyUnicode_AsUnicodeEscapeString http://bugs.python.org/issue28760 closed by serhiy.storchaka #28761: Add the const qualifier to fields name and doc of public struc http://bugs.python.org/issue28761 closed by serhiy.storchaka #28766: Remove the semicolon in source code http://bugs.python.org/issue28766 closed by martin.panter #28772: Bus error in Python 3.6.0beta http://bugs.python.org/issue28772 closed by Stefan Scherfke #28773: typing.FrozenSet missing in documentation. http://bugs.python.org/issue28773 closed by gvanrossum #28774: Better start and end position for unicodeerror in unicode_enco http://bugs.python.org/issue28774 closed by serhiy.storchaka #28776: Duplicate method names should be an error http://bugs.python.org/issue28776 closed by rhettinger #28782: SEGFAULT when running a given coroutine http://bugs.python.org/issue28782 closed by haypo #28783: Embedded/nuget packages incorrectly reference bdist_wininst http://bugs.python.org/issue28783 closed by steve.dower #28786: Warnings in Misc/NEWS http://bugs.python.org/issue28786 closed by SilentGhost #28792: bisect: implement aliases in Python, remove C aliases http://bugs.python.org/issue28792 closed by haypo #28793: Copy-paste error in collections.abc docs for AsyncGenerator http://bugs.python.org/issue28793 closed by berker.peksag #28796: FIX warnings in documentation build http://bugs.python.org/issue28796 closed by berker.peksag #28798: Describe character as a string of length one instead of size o http://bugs.python.org/issue28798 closed by rhettinger #28801: configparser: before_get() method of class Interpolation has p http://bugs.python.org/issue28801 closed by r.david.murray #28803: Python 3.6.0b4 Reports ncurses present in Cygwin but fails to http://bugs.python.org/issue28803 closed by berker.peksag From raymond.hettinger at gmail.com Sat Nov 26 03:34:48 2016 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sat, 26 Nov 2016 00:34:48 -0800 Subject: [Python-Dev] Does anyone use CALL_PROFILE special build? In-Reply-To: References: Message-ID: <37BE00A6-264C-438D-9278-6A8F52C9AD09@gmail.com> > On Nov 25, 2016, at 1:28 AM, Victor Stinner wrote: > > I would like to completely remove the feature to be able to implement > further optimizations. This seems reasonable to me. I've never used or needed this special build; StackOverflow has no mention of it; and a Google search comes up nearly empty. That said, it might be worthwhile to check with Jeremy to get his thoughts before removing his code. Raymond From noreply at academia-mail.com Sat Nov 26 20:43:07 2016 From: noreply at academia-mail.com (Academia.edu) Date: Sat, 26 Nov 2016 17:43:07 -0800 Subject: [Python-Dev] Jean-marie Burkhardt's co-author tag is pending your approval Message-ID: <583a3a2b42813_1bca3fa427bca9e8601656@send-emails-sidekiq-20161126-swathed.z0o.us.mail> Hi Python, Jean-marie Burkhardt tagged you as a co-author on: "A Methodological Framework for Socio-Cognitive Analyses of Collaborative Design of Open Source Software" To approve this tag, please visit the link below: https://www.academia.edu/t/cjCVm-LfBn0m5-t5u0J/8891732/A_Methodological_Framework_for_Socio-Cognitive_Analyses_of_Collaborative_Design_of_Open_Source_Software?coauthor_invite=614adbf50eea519b7a69b8ece180b02c ----- You can opt out of receiving these kinds of emails from Academia.edu with the link below: https://www.academia.edu/t/cjCVm-LfBn0m5-t5u0J/optout/595851d739c19e8e98b5d6ca0ad8ea96 Academia.edu, 251 Kearny St., Suite 520, San Francisco, CA, 94108 -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy at alum.mit.edu Sun Nov 27 10:37:11 2016 From: jeremy at alum.mit.edu (Jeremy Hylton) Date: Sun, 27 Nov 2016 15:37:11 +0000 Subject: [Python-Dev] Does anyone use CALL_PROFILE special build? In-Reply-To: <37BE00A6-264C-438D-9278-6A8F52C9AD09@gmail.com> References: <37BE00A6-264C-438D-9278-6A8F52C9AD09@gmail.com> Message-ID: I don't think there's any need to keep this feature. I hope we have better ways to profile function and method calls now than we did when this code was submitted. Jeremy On Sat, Nov 26, 2016 at 3:34 AM Raymond Hettinger < raymond.hettinger at gmail.com> wrote: > > > On Nov 25, 2016, at 1:28 AM, Victor Stinner > wrote: > > > > I would like to completely remove the feature to be able to implement > > further optimizations. > > This seems reasonable to me. I've never used or needed this special > build; StackOverflow has no mention of it; and a Google search comes up > nearly empty. That said, it might be worthwhile to check with Jeremy to > get his thoughts before removing his code. > > > Raymond -------------- next part -------------- An HTML attachment was scrubbed... URL: From benjamin at python.org Mon Nov 28 02:06:31 2016 From: benjamin at python.org (Benjamin Peterson) Date: Sun, 27 Nov 2016 23:06:31 -0800 Subject: [Python-Dev] Python 2.7.13 release dates Message-ID: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> I've have just updated PEP 373 to say that Python 2.7.13 release candidate 1 will be released on December 3. The final will follow two weeks later on December 17. If there are delays in the process, the final will likely to pushed into January. Servus, Benjamin From victor.stinner at gmail.com Mon Nov 28 06:06:38 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Mon, 28 Nov 2016 12:06:38 +0100 Subject: [Python-Dev] Does anyone use CALL_PROFILE special build? In-Reply-To: References: <37BE00A6-264C-438D-9278-6A8F52C9AD09@gmail.com> Message-ID: Ok, thanks. I pushed my patch. Victor 2016-11-27 16:37 GMT+01:00 Jeremy Hylton : > I don't think there's any need to keep this feature. I hope we have better > ways to profile function and method calls now than we did when this code was > submitted. > > Jeremy > > > On Sat, Nov 26, 2016 at 3:34 AM Raymond Hettinger > wrote: >> >> >> > On Nov 25, 2016, at 1:28 AM, Victor Stinner >> > wrote: >> > >> > I would like to completely remove the feature to be able to implement >> > further optimizations. >> >> This seems reasonable to me. I've never used or needed this special >> build; StackOverflow has no mention of it; and a Google search comes up >> nearly empty. That said, it might be worthwhile to check with Jeremy to >> get his thoughts before removing his code. >> >> >> Raymond From cstratak at redhat.com Mon Nov 28 10:35:36 2016 From: cstratak at redhat.com (Charalampos Stratakis) Date: Mon, 28 Nov 2016 10:35:36 -0500 (EST) Subject: [Python-Dev] Tests breakage with latest openssl (issue28689) In-Reply-To: <1960099934.363800.1480346129534.JavaMail.zimbra@redhat.com> Message-ID: <1827231094.697192.1480347336308.JavaMail.zimbra@redhat.com> Hello, Escalating here an issue that seems that it should be tagged as blocker. Currently the latest version of openssl-1.1.0c breaks Python's test suite. The problematic commit was identified [0] and reverted [1] at openssl's upstream, however when running the test suite on a Fedora Rawhide machine, which includes the fix, the tests currently hang (not just failing like before). The issue, with some more details, is tracked here: https://bugs.python.org/issue28689 [0] https://github.com/openssl/openssl/issues/1903 [1] https://github.com/openssl/openssl/commit/beacb0f0c1ae7b0542fe053b95307f515b578eb7 Regards, Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat From christian at python.org Mon Nov 28 12:03:34 2016 From: christian at python.org (Christian Heimes) Date: Mon, 28 Nov 2016 18:03:34 +0100 Subject: [Python-Dev] Tests breakage with latest openssl (issue28689) In-Reply-To: <1827231094.697192.1480347336308.JavaMail.zimbra@redhat.com> References: <1960099934.363800.1480346129534.JavaMail.zimbra@redhat.com> <1827231094.697192.1480347336308.JavaMail.zimbra@redhat.com> Message-ID: On 2016-11-28 16:35, Charalampos Stratakis wrote: > Hello, > > Escalating here an issue that seems that it should be tagged as blocker. > > Currently the latest version of openssl-1.1.0c breaks Python's test suite. > > The problematic commit was identified [0] and reverted [1] at openssl's upstream, however when running the test suite on a Fedora Rawhide machine, which includes the fix, the tests currently hang (not just failing like before). > > The issue, with some more details, is tracked here: https://bugs.python.org/issue28689 > > [0] https://github.com/openssl/openssl/issues/1903 > [1] https://github.com/openssl/openssl/commit/beacb0f0c1ae7b0542fe053b95307f515b578eb7 Hi Charalampos, Python's 3.6 and default (3.7) tests suite is passing with OpenSSL 1.1.0d-dev (OpenSSL_1_1_0-stable branch). Maybe your backport is missing a fix? Christian From cstratak at redhat.com Mon Nov 28 12:16:57 2016 From: cstratak at redhat.com (Charalampos Stratakis) Date: Mon, 28 Nov 2016 12:16:57 -0500 (EST) Subject: [Python-Dev] Tests breakage with latest openssl (issue28689) In-Reply-To: References: <1960099934.363800.1480346129534.JavaMail.zimbra@redhat.com> <1827231094.697192.1480347336308.JavaMail.zimbra@redhat.com> Message-ID: <1127697490.3309907.1480353417910.JavaMail.zimbra@redhat.com> Hi Christian and thanks for the fast reply, It's great to hear that the latest version is working fine. Do you have anymore details on the fix/breakage? The latest commit at Fedora's rawhide openssl package is at [0]. Is it missing something? [0] http://pkgs.fedoraproject.org/cgit/rpms/openssl.git/commit/?id=e443a79334446ac0dc14fdf7c062386f92bbc7a0 Regards, Charalampos Stratakis Associate Software Engineer Python Maintenance Team, Red Hat ----- Original Message ----- From: "Christian Heimes" To: python-dev at python.org Sent: Monday, November 28, 2016 6:03:34 PM Subject: Re: [Python-Dev] Tests breakage with latest openssl (issue28689) On 2016-11-28 16:35, Charalampos Stratakis wrote: > Hello, > > Escalating here an issue that seems that it should be tagged as blocker. > > Currently the latest version of openssl-1.1.0c breaks Python's test suite. > > The problematic commit was identified [0] and reverted [1] at openssl's upstream, however when running the test suite on a Fedora Rawhide machine, which includes the fix, the tests currently hang (not just failing like before). > > The issue, with some more details, is tracked here: https://bugs.python.org/issue28689 > > [0] https://github.com/openssl/openssl/issues/1903 > [1] https://github.com/openssl/openssl/commit/beacb0f0c1ae7b0542fe053b95307f515b578eb7 Hi Charalampos, Python's 3.6 and default (3.7) tests suite is passing with OpenSSL 1.1.0d-dev (OpenSSL_1_1_0-stable branch). Maybe your backport is missing a fix? Christian _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/cstratak%40redhat.com From storchaka at gmail.com Mon Nov 28 13:36:54 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 28 Nov 2016 20:36:54 +0200 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> Message-ID: On 28.11.16 09:06, Benjamin Peterson wrote: > I've have just updated PEP 373 to say that Python 2.7.13 release > candidate 1 will be released on December 3. The final will follow two > weeks later on December 17. If there are delays in the process, the > final will likely to pushed into January. Could it be delayed until 3.6.0 released? I paused fixing non-critical and non-documentation bugs while 3.6 in pre-release stage and this could include bugs that affect 2.7. In additional, we always receive increased number of bug reports in the first one or two weeks after releasing new Python version. Some of these reports are about regressions introduced by bugfixes. If delay bugfix releases after new version release, we could fix regressions caused by backported bugfixes and make bugfix releases more reliable. From guido at python.org Mon Nov 28 13:42:12 2016 From: guido at python.org (Guido van Rossum) Date: Mon, 28 Nov 2016 10:42:12 -0800 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> Message-ID: Perhaps (un)related, is the release schedule for 3.5.3 set yet? (I'm holding off putting a new version of typing.py on typeshed until that's out.) On Mon, Nov 28, 2016 at 10:36 AM, Serhiy Storchaka wrote: > On 28.11.16 09:06, Benjamin Peterson wrote: > >> I've have just updated PEP 373 to say that Python 2.7.13 release >> candidate 1 will be released on December 3. The final will follow two >> weeks later on December 17. If there are delays in the process, the >> final will likely to pushed into January. >> > > Could it be delayed until 3.6.0 released? I paused fixing non-critical and > non-documentation bugs while 3.6 in pre-release stage and this could > include bugs that affect 2.7. > > In additional, we always receive increased number of bug reports in the > first one or two weeks after releasing new Python version. Some of these > reports are about regressions introduced by bugfixes. If delay bugfix > releases after new version release, we could fix regressions caused by > backported bugfixes and make bugfix releases more reliable. > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido% > 40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Mon Nov 28 14:38:57 2016 From: larry at hastings.org (Larry Hastings) Date: Mon, 28 Nov 2016 11:38:57 -0800 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> Message-ID: On 11/28/2016 10:42 AM, Guido van Rossum wrote: > Perhaps (un)related, is the release schedule for 3.5.3 set yet? (I'm > holding off putting a new version of typing.py on typeshed until > that's out.) Not set in stone, but I have a rough plan. I want the dust to settle around 3.6 first. So, have an RC1 in January, and then the final release two weeks later. I'm traveling the second week of January (#DSDad) so the RC would be either before or after that. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Mon Nov 28 23:50:08 2016 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 28 Nov 2016 20:50:08 -0800 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> Message-ID: > On Nov 28, 2016, at 10:36 AM, Serhiy Storchaka wrote: > > On 28.11.16 09:06, Benjamin Peterson wrote: >> I've have just updated PEP 373 to say that Python 2.7.13 release >> candidate 1 will be released on December 3. The final will follow two >> weeks later on December 17. If there are delays in the process, the >> final will likely to pushed into January. > > Could it be delayed until 3.6.0 released? I paused fixing non-critical and non-documentation bugs while 3.6 in pre-release stage and this could include bugs that affect 2.7. > > In additional, we always receive increased number of bug reports in the first one or two weeks after releasing new Python version. Some of these reports are about regressions introduced by bugfixes. If delay bugfix releases after new version release, we could fix regressions caused by backported bugfixes and make bugfix releases more reliable. +1 on delaying 2.7.13 for a bit. As long as it doesn't muck up Benjamin's schedule, the extra time would be helpful (Python 3.6.0 got all the focus recently). Raymond From benjamin at python.org Tue Nov 29 03:01:40 2016 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 29 Nov 2016 00:01:40 -0800 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> Message-ID: <1480406500.3889434.802141153.0E325DD2@webmail.messagingengine.com> Okay, by popular demand, 2.7.13 now happens in January. I'm curious what people are planning to do to 2.7 with the extra 5 weeks. The 2.7 branch is a place to put occasional conservative bug fixes, which we aggregate and release every 6 months. It shouldn't really need special attention or become less stable depending on the release stage of Python 3. On Mon, Nov 28, 2016, at 20:50, Raymond Hettinger wrote: > > > On Nov 28, 2016, at 10:36 AM, Serhiy Storchaka wrote: > > > > On 28.11.16 09:06, Benjamin Peterson wrote: > >> I've have just updated PEP 373 to say that Python 2.7.13 release > >> candidate 1 will be released on December 3. The final will follow two > >> weeks later on December 17. If there are delays in the process, the > >> final will likely to pushed into January. > > > > Could it be delayed until 3.6.0 released? I paused fixing non-critical and non-documentation bugs while 3.6 in pre-release stage and this could include bugs that affect 2.7. > > > > In additional, we always receive increased number of bug reports in the first one or two weeks after releasing new Python version. Some of these reports are about regressions introduced by bugfixes. If delay bugfix releases after new version release, we could fix regressions caused by backported bugfixes and make bugfix releases more reliable. > > +1 on delaying 2.7.13 for a bit. As long as it doesn't muck up > Benjamin's schedule, the extra time would be helpful (Python 3.6.0 got > all the focus recently). > > > Raymond > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org From stephane at wirtel.be Tue Nov 29 08:26:53 2016 From: stephane at wirtel.be (Stephane Wirtel) Date: Tue, 29 Nov 2016 14:26:53 +0100 Subject: [Python-Dev] Last call for the Call For Proposals of PythonFOSDEM 2017 Message-ID: <20161129132653.e3vlya75shg2a7k2@sg1> Because the deadline is imminent and because we have only received some proposals, we will extend the current deadline. The new submission deadline is 2016-12-18. Call For Proposals ================== This is the official call for sessions for the Python devroom at FOSDEM 2017. FOSDEM is the Free and Open source Software Developers' European Meeting, a free and non-commercial two-day week-end that offers open source contributors a place to meet, share ideas and collaborate. It's the biggest event in Europe with +5000 hackers, +400 speakers. For this edition, Python will be represented by its Community. If you want to discuss with a lot of Python Users, it's the place to be! Important dates =============== * Submission deadlines: 2016-12-18 * Acceptance notifications: 2016-12-23 Practical ========= * The duration for talks will be 30 minutes, including presentations and questions and answers. * Presentation can be recorded and streamed, sending your proposal implies giving permission to be recorded. * A mailing list for the Python devroom is available for discussions about devroom organisation. You can register at this address: https://lists.fosdem.org/listinfo/python-devroom How to submit ============= All submissions are made in the Pentabarf event planning tool at https://penta.fosdem.org/submission/FOSDEM17 When submitting your talk in Pentabarf, make sure to select the Python devroom as the Track. Of course, if you already have a user account, please reuse it. Questions ========= Any questions, please send an email to info AT python-fosdem DOT org Thank you for submitting your sessions and see you soon in Brussels to talk about Python. If you want to keep informed for this edition, you can follow our twitter account @PythonFOSDEM. * FOSDEM 2017: https://fosdem.org/2017 * Python Devroom: http://python-fosdem.org * Twitter: https://twitter.com/PythonFOSDEM Stephane -- St?phane Wirtel - http://wirtel.be - @matrixise From steve.dower at python.org Tue Nov 29 09:12:55 2016 From: steve.dower at python.org (Steve Dower) Date: Tue, 29 Nov 2016 06:12:55 -0800 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: <1480406500.3889434.802141153.0E325DD2@webmail.messagingengine.com> References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> <1480406500.3889434.802141153.0E325DD2@webmail.messagingengine.com> Message-ID: I would much rather keep it in December, as I've already made scheduling decisions around the planned release date and there are fixes in 2.7.13 that I was expecting to be available by the end of the year. One month is highly impactful for me. Is this schedule change going to remove the month from 2.7.14? Or are we slipping all the rest of the releases (apart from the very last one at the end of 2019, presumably)? I too would like to know the intended use of the extra time. Top-posted from my Windows Phone -----Original Message----- From: "Benjamin Peterson" Sent: ?11/?29/?2016 0:04 To: "Raymond Hettinger" ; "Serhiy Storchaka" Cc: "Python-Dev at Python. Org" Subject: Re: [Python-Dev] Python 2.7.13 release dates Okay, by popular demand, 2.7.13 now happens in January. I'm curious what people are planning to do to 2.7 with the extra 5 weeks. The 2.7 branch is a place to put occasional conservative bug fixes, which we aggregate and release every 6 months. It shouldn't really need special attention or become less stable depending on the release stage of Python 3. On Mon, Nov 28, 2016, at 20:50, Raymond Hettinger wrote: > > > On Nov 28, 2016, at 10:36 AM, Serhiy Storchaka wrote: > > > > On 28.11.16 09:06, Benjamin Peterson wrote: > >> I've have just updated PEP 373 to say that Python 2.7.13 release > >> candidate 1 will be released on December 3. The final will follow two > >> weeks later on December 17. If there are delays in the process, the > >> final will likely to pushed into January. > > > > Could it be delayed until 3.6.0 released? I paused fixing non-critical and non-documentation bugs while 3.6 in pre-release stage and this could include bugs that affect 2.7. > > > > In additional, we always receive increased number of bug reports in the first one or two weeks after releasing new Python version. Some of these reports are about regressions introduced by bugfixes. If delay bugfix releases after new version release, we could fix regressions caused by backported bugfixes and make bugfix releases more reliable. > > +1 on delaying 2.7.13 for a bit. As long as it doesn't muck up > Benjamin's schedule, the extra time would be helpful (Python 3.6.0 got > all the focus recently). > > > Raymond > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From christian at python.org Tue Nov 29 09:27:56 2016 From: christian at python.org (Christian Heimes) Date: Tue, 29 Nov 2016 15:27:56 +0100 Subject: [Python-Dev] Tests breakage with latest openssl (issue28689) In-Reply-To: <1127697490.3309907.1480353417910.JavaMail.zimbra@redhat.com> References: <1960099934.363800.1480346129534.JavaMail.zimbra@redhat.com> <1827231094.697192.1480347336308.JavaMail.zimbra@redhat.com> <1127697490.3309907.1480353417910.JavaMail.zimbra@redhat.com> Message-ID: On 2016-11-28 18:16, Charalampos Stratakis wrote: > Hi Christian and thanks for the fast reply, > > It's great to hear that the latest version is working fine. Do you have anymore details on the fix/breakage? > > The latest commit at Fedora's rawhide openssl package is at [0]. Is it missing something? > > [0] http://pkgs.fedoraproject.org/cgit/rpms/openssl.git/commit/?id=e443a79334446ac0dc14fdf7c062386f92bbc7a0 Hi Charalampos, Sorry, I don't have more details other than the Python and OpenSSL tickets. It sounds like the Fedora rawhide fix is not sufficient and may lack something. I don't have any spare cycles to look at the patch. A distribution patch doesn't work for users of other distributions, too. Let's see if I can get the OpenSSL team to release another version soonish. Christian From doko at ubuntu.com Tue Nov 29 09:35:50 2016 From: doko at ubuntu.com (Matthias Klose) Date: Tue, 29 Nov 2016 15:35:50 +0100 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> Message-ID: On 28.11.2016 20:38, Larry Hastings wrote: > On 11/28/2016 10:42 AM, Guido van Rossum wrote: >> Perhaps (un)related, is the release schedule for 3.5.3 set yet? (I'm holding >> off putting a new version of typing.py on typeshed until that's out.) > > Not set in stone, but I have a rough plan. I want the dust to settle around 3.6 > first. So, have an RC1 in January, and then the final release two weeks later. > I'm traveling the second week of January (#DSDad) so the RC would be either > before or after that. https://wiki.debian.org/DebianStretch shows the final Debian release freeze for 2016-02-05, which would be a bit close for a 3.5.3 release. So anything with some buffer to Feb 05 would be appreciated. From doko at ubuntu.com Tue Nov 29 09:34:25 2016 From: doko at ubuntu.com (Matthias Klose) Date: Tue, 29 Nov 2016 15:34:25 +0100 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> <1480406500.3889434.802141153.0E325DD2@webmail.messagingengine.com> Message-ID: <41b7f3d0-952a-87a2-6dce-62cba199d46d@ubuntu.com> same here. https://wiki.debian.org/DebianStretch shows the final Debian release freeze for 2016-02-05, which would be a bit close for a 2.7.13 release. On 29.11.2016 15:12, Steve Dower wrote: > I would much rather keep it in December, as I've already made scheduling decisions around the planned release date and there are fixes in 2.7.13 that I was expecting to be available by the end of the year. One month is highly impactful for me. > > Is this schedule change going to remove the month from 2.7.14? Or are we slipping all the rest of the releases (apart from the very last one at the end of 2019, presumably)? I too would like to know the intended use of the extra time. > > Top-posted from my Windows Phone > > -----Original Message----- > From: "Benjamin Peterson" > Sent: ?11/?29/?2016 0:04 > To: "Raymond Hettinger" ; "Serhiy Storchaka" > Cc: "Python-Dev at Python. Org" > Subject: Re: [Python-Dev] Python 2.7.13 release dates > > Okay, by popular demand, 2.7.13 now happens in January. > > I'm curious what people are planning to do to 2.7 with the extra 5 > weeks. The 2.7 branch is a place to put occasional conservative bug > fixes, which we aggregate and release every 6 months. It shouldn't > really need special attention or become less stable depending on the > release stage of Python 3. > > On Mon, Nov 28, 2016, at 20:50, Raymond Hettinger wrote: >> >>> On Nov 28, 2016, at 10:36 AM, Serhiy Storchaka wrote: >>> >>> On 28.11.16 09:06, Benjamin Peterson wrote: >>>> I've have just updated PEP 373 to say that Python 2.7.13 release >>>> candidate 1 will be released on December 3. The final will follow two >>>> weeks later on December 17. If there are delays in the process, the >>>> final will likely to pushed into January. >>> >>> Could it be delayed until 3.6.0 released? I paused fixing non-critical and non-documentation bugs while 3.6 in pre-release stage and this could include bugs that affect 2.7. >>> >>> In additional, we always receive increased number of bug reports in the first one or two weeks after releasing new Python version. Some of these reports are about regressions introduced by bugfixes. If delay bugfix releases after new version release, we could fix regressions caused by backported bugfixes and make bugfix releases more reliable. >> >> +1 on delaying 2.7.13 for a bit. As long as it doesn't muck up >> Benjamin's schedule, the extra time would be helpful (Python 3.6.0 got >> all the focus recently). >> >> >> Raymond >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/doko%40ubuntu.com > From annakoppad at gmail.com Tue Nov 29 09:39:38 2016 From: annakoppad at gmail.com (Annapoornima Koppad) Date: Tue, 29 Nov 2016 20:09:38 +0530 Subject: [Python-Dev] Python private and protected variables and methods Message-ID: Dear All, While teaching Python protected variables, I figured out the oddity about Python. I am using Python 2.7.12+ for my teaching. There is not really a concept of protected variables in Python, is it? For a more detailed explanation of what I am trying to do, please read this article, http://radek.io/2011/07/21/private-protected-and-public-in-python/ It that is the case, Python is not suitable for companies that require secure transactions such as Banking softwares. What solution are possible to overcome data masquerading attempts that might result because of this? Thanks and regards, Annapoornima www.pyladies.co.in -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Nov 29 12:39:01 2016 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 29 Nov 2016 12:39:01 -0500 Subject: [Python-Dev] Python private and protected variables and methods In-Reply-To: References: Message-ID: On 11/29/2016 9:39 AM, Annapoornima Koppad wrote: > While teaching Python protected variables, I figured out the oddity > about Python. > I am using Python 2.7.12+ for my teaching. > There is not really a concept of protected variables in Python, is it? Only the _ convention. > For a more detailed explanation of what I am trying to do, please read > this article, > http://radek.io/2011/07/21/private-protected-and-public-in-python/ > > It that is the case, Python is not suitable for companies that require > secure transactions such as Banking softwares. What solution are > possible to overcome data masquerading attempts that might result > because of this? Hello, Annapoornima pydev list is for development of future Python and Cpython. Questions about using current Python should be directed to python-list or other forums such as Stackoverflow or specialized lists. I know that Python is heavily used in the American finacial sector but not anything about specific security practices. I expect python-list to have readers that do know something. When you post, try to be a bit more specific about particular threat models. -- Terry Jan Reedy From benjamin at python.org Wed Nov 30 02:07:14 2016 From: benjamin at python.org (Benjamin Peterson) Date: Tue, 29 Nov 2016 23:07:14 -0800 Subject: [Python-Dev] Python 2.7.13 release dates In-Reply-To: <41b7f3d0-952a-87a2-6dce-62cba199d46d@ubuntu.com> References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> <1480406500.3889434.802141153.0E325DD2@webmail.messagingengine.com> <41b7f3d0-952a-87a2-6dce-62cba199d46d@ubuntu.com> Message-ID: <1480489634.4189899.803413225.41835A11@webmail.messagingengine.com> Okay, now that we're heard from the other side, and I lacking a concrete reason to delay the release, I'm putting 2.7.13 back at the original dates. On Tue, Nov 29, 2016, at 06:34, Matthias Klose wrote: > same here. > https://wiki.debian.org/DebianStretch shows the final Debian release > freeze for > 2016-02-05, which would be a bit close for a 2.7.13 release. > > On 29.11.2016 15:12, Steve Dower wrote: > > I would much rather keep it in December, as I've already made scheduling decisions around the planned release date and there are fixes in 2.7.13 that I was expecting to be available by the end of the year. One month is highly impactful for me. > > > > Is this schedule change going to remove the month from 2.7.14? Or are we slipping all the rest of the releases (apart from the very last one at the end of 2019, presumably)? I too would like to know the intended use of the extra time. > > > > Top-posted from my Windows Phone > > > > -----Original Message----- > > From: "Benjamin Peterson" > > Sent: ?11/?29/?2016 0:04 > > To: "Raymond Hettinger" ; "Serhiy Storchaka" > > Cc: "Python-Dev at Python. Org" > > Subject: Re: [Python-Dev] Python 2.7.13 release dates > > > > Okay, by popular demand, 2.7.13 now happens in January. > > > > I'm curious what people are planning to do to 2.7 with the extra 5 > > weeks. The 2.7 branch is a place to put occasional conservative bug > > fixes, which we aggregate and release every 6 months. It shouldn't > > really need special attention or become less stable depending on the > > release stage of Python 3. > > > > On Mon, Nov 28, 2016, at 20:50, Raymond Hettinger wrote: > >> > >>> On Nov 28, 2016, at 10:36 AM, Serhiy Storchaka wrote: > >>> > >>> On 28.11.16 09:06, Benjamin Peterson wrote: > >>>> I've have just updated PEP 373 to say that Python 2.7.13 release > >>>> candidate 1 will be released on December 3. The final will follow two > >>>> weeks later on December 17. If there are delays in the process, the > >>>> final will likely to pushed into January. > >>> > >>> Could it be delayed until 3.6.0 released? I paused fixing non-critical and non-documentation bugs while 3.6 in pre-release stage and this could include bugs that affect 2.7. > >>> > >>> In additional, we always receive increased number of bug reports in the first one or two weeks after releasing new Python version. Some of these reports are about regressions introduced by bugfixes. If delay bugfix releases after new version release, we could fix regressions caused by backported bugfixes and make bugfix releases more reliable. > >> > >> +1 on delaying 2.7.13 for a bit. As long as it doesn't muck up > >> Benjamin's schedule, the extra time would be helpful (Python 3.6.0 got > >> all the focus recently). > >> > >> > >> Raymond > >> _______________________________________________ > >> Python-Dev mailing list > >> Python-Dev at python.org > >> https://mail.python.org/mailman/listinfo/python-dev > >> Unsubscribe: > >> https://mail.python.org/mailman/options/python-dev/benjamin%40python.org > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/doko%40ubuntu.com > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/benjamin%40python.org From storchaka at gmail.com Wed Nov 30 04:01:21 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 30 Nov 2016 11:01:21 +0200 Subject: [Python-Dev] cpython: Uniformize argument names of "call" functions In-Reply-To: <20161129175839.27752.62358.C471A8E6@psf.io> References: <20161129175839.27752.62358.C471A8E6@psf.io> Message-ID: On 29.11.16 19:58, victor.stinner wrote: > https://hg.python.org/cpython/rev/7efddbf1aa70 > changeset: 105395:7efddbf1aa70 > user: Victor Stinner > date: Tue Nov 29 18:47:56 2016 +0100 > summary: > Uniformize argument names of "call" functions > > * Callable object: callable, o, callable_object => func > * Object for method calls: o => obj > * Method name: name or nameid => method This change looks wrong to me. "callable" and "callable_object" are better names for functions like PyObject_Call(), since it supports not just functions, but any callables. "name" is appropriate name of the parameter that denotes a method name, not a method object. From victor.stinner at gmail.com Wed Nov 30 04:15:08 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Nov 2016 10:15:08 +0100 Subject: [Python-Dev] cpython: Uniformize argument names of "call" functions In-Reply-To: References: <20161129175839.27752.62358.C471A8E6@psf.io> Message-ID: 2016-11-30 10:01 GMT+01:00 Serhiy Storchaka : >> Uniformize argument names of "call" functions >> >> * Callable object: callable, o, callable_object => func >> * Object for method calls: o => obj >> * Method name: name or nameid => method > > This change looks wrong to me. "callable" and "callable_object" are better > names for functions like PyObject_Call(), since it supports not just > functions, but any callables. "name" is appropriate name of the parameter > that denotes a method name, not a method object. (Oh no, I avoided a review to try to avoid bikeshedding...) I tried to be consistent between argument names and function names. For example, I expect that you have to pass a *function* to PyObject_CallFunction(), and that you have to pass a *method* to PyObject_CallMethod(). -- In third party code, I don't recall having seen a variable called "callable" (or they are very rare?). In the stdlib, "func" is much more common than "callable", raw statistics (default branch): $ grep '\' Lib/*.py|wc -l 318 $ grep '\' Lib/*.py|wc -l 115 In CPython C code, "func" is also more common than "callable" (3.5 branch): $ grep '\' */*.c|wc -l 725 $ grep '\' */*.c|wc -l 126 My concern is not to be accurate in the variable name, just to use a convenient, short and common name to *uniformize* the code. Victor From storchaka at gmail.com Wed Nov 30 05:03:53 2016 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 30 Nov 2016 12:03:53 +0200 Subject: [Python-Dev] cpython: Uniformize argument names of "call" functions In-Reply-To: References: <20161129175839.27752.62358.C471A8E6@psf.io> Message-ID: On 30.11.16 11:15, Victor Stinner wrote: > 2016-11-30 10:01 GMT+01:00 Serhiy Storchaka : >>> Uniformize argument names of "call" functions >>> >>> * Callable object: callable, o, callable_object => func >>> * Object for method calls: o => obj >>> * Method name: name or nameid => method >> >> This change looks wrong to me. "callable" and "callable_object" are better >> names for functions like PyObject_Call(), since it supports not just >> functions, but any callables. "name" is appropriate name of the parameter >> that denotes a method name, not a method object. > > (Oh no, I avoided a review to try to avoid bikeshedding...) Actually it looked as a red flag to me. > I tried to be consistent between argument names and function names. > For example, I expect that you have to pass a *function* to > PyObject_CallFunction(), and that you have to pass a *method* to > PyObject_CallMethod(). You have to pass a *callable object* to PyObject_CallFunction(), and that you have to pass a method *name* to PyObject_CallMethod(). It would be better to roll back the commit. Other changes looks not well justified too. We can start new discussion about uniform names after that. From victor.stinner at gmail.com Wed Nov 30 06:14:41 2016 From: victor.stinner at gmail.com (Victor Stinner) Date: Wed, 30 Nov 2016 12:14:41 +0100 Subject: [Python-Dev] cpython: Uniformize argument names of "call" functions In-Reply-To: References: <20161129175839.27752.62358.C471A8E6@psf.io> Message-ID: I reverted my change and reposted the change as a patch: http://bugs.python.org/issue28838 Victor From solipsis at pitrou.net Wed Nov 30 13:19:40 2016 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 30 Nov 2016 19:19:40 +0100 Subject: [Python-Dev] Python 2.7.13 release dates References: <1480316791.2913744.800826761.72EEE93A@webmail.messagingengine.com> <1480406500.3889434.802141153.0E325DD2@webmail.messagingengine.com> <41b7f3d0-952a-87a2-6dce-62cba199d46d@ubuntu.com> <1480489634.4189899.803413225.41835A11@webmail.messagingengine.com> Message-ID: <20161130191940.4f7fc235@fsol> On Tue, 29 Nov 2016 23:07:14 -0800 Benjamin Peterson wrote: > Okay, now that we're heard from the other side, and I lacking a concrete > reason to delay the release, I'm putting 2.7.13 back at the original > dates. Serhiy may be thinking about https://bugs.python.org/issue28427 Regards Antoine.