From alfred at freebsd.org Sun Jul 1 00:00:33 2018 From: alfred at freebsd.org (Alfred Perlstein) Date: Sat, 30 Jun 2018 21:00:33 -0700 Subject: [Python-Dev] Help preventing SIGPIPE/SIG_DFL anti-pattern. In-Reply-To: <5B38103A.2050002@canterbury.ac.nz> References: <922692be-4089-0162-4c3c-93bad5a2c3d8@freebsd.org> <5B38103A.2050002@canterbury.ac.nz> Message-ID: On 6/30/18 4:20 PM, Greg Ewing wrote: > Alfred Perlstein wrote: >> I am asking if there's a way we can discourage the use of >> "signal(SIGPIPE, SIG_DFL)" unless the user really understands what >> they are doing. > > Maybe there's some way that SIGPIPEs on stdout could be handled > differently by default, so that they exit silently instead of > producing an ugly message. That would remove the source of pain > that's leading people to do this. > Thank you Greg, I can poke around into this, it would be a bit of a challenge as the descriptor which causes BrokenPipeError does not appear to be stored within the exception so differentiating it from other exceptions might be a bit tricky. I will look into this in the coming weeks.? Any tips on accomplishing this?? I was thinking of encoding the fd responsible for causing the error into the exception somehow and then checking to see if it was stdout, then not reporting on it. -Alfred From tim.peters at gmail.com Sun Jul 1 00:32:03 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 30 Jun 2018 23:32:03 -0500 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: [Nick Coghlan] >>> "NAME := EXPR" exists on a different level of complexity, since it >>> adds name binding in arbitrary expressions for the sake of minor >>> performance improvement in code written by developers that are >>> exceptionally averse to the use of vertical screen real estate, > >>> ... [Tim] >> Note that PEP 572 doesn't contain a single word about "performance" (neither > >> that specific word nor any synonym), and I gave only one thought to it when > >> writing Appendix A: "is this going to slow anything down significantly?". > >> The answer was never "yes", which I thought was self-evident, so I never > >> mentioned it. Neither did Chris or Guido. > >> > >> Best I can recall, nobody has argued for it on the grounds of "performance". > >> except in the indirect sense that sometimes it allows a more compact way of > >> reusing an expensive subexpression by giving it a name. Which they already > >> do by giving it a name in a separate statement, so the possible improvement > >> would be in brevity rather than performance. [Nick] > > The PEP specifically cites this example as motivation: The PEP gives many examples. Your original was a strawman mischaracterization of the PEP's _motivations_ (note the plural: you only mentioned "minor performance improvement", and snipped my listing of the major motivations). > > group = re.match(data).group(1) if re.match(data) else None > > > > That code's already perfectly straightforward to read and write as a > > single line, I disagree. In any case of textual repetition, it's a visual pattern-matching puzzle to identify the common substrings (I have to visually scan that line about 3 times to be sure), and then a potentially difficult conceptual puzzle to figure out whether side effects may result in textually identical substrings evaluating to different objects. That's why "refererential transparency" is so highly valued in functional languages ("if subexpressions are spelled the same, they evaluate to the same result, period" - which isn't generally true in Python - to get that enormously helpful (to reasoning) guarantee in Python you have to ensure the subexpression is evaluated exactly once). And as you of all people should be complaining about, textual repetition is also prone to "oops - forgot one!" and "oops! made a typo when changing the second one!" when code is later modified. > so the only reason to quibble about it I gave you three better reasons to quibble about it just above ;-) > is because it's slower than the arguably less clear two-line alternative: > > > > _m = re.match(data) > > group = _m.group(1) if _m else None > I find that much clearer than the one-liner above: the visual pattern matching is easier because the repeated substring is shorter and of much simpler syntactic structure; it guarantees _by construction_ that the two instances of `_m` evaluate to the same object, so there's no possible concern about that (it doesn't even matter if you bound `re` to some "non-standard" object that has nothing to do with Python's `re` module); and any later changes to the single instance of `re.match(data)` don't have to be repeated verbatim elsewhere. It's possible that it runs twice as fast too, but that's the least of my concerns. All of those advantages are retained in the one-liner too if an assignment expression can be used in it. > Thus the PEP's argument is that it wants to allow the faster version > > to remain a one-liner that preserves the overall structure of the > > version that repeats the subexpression: > > > > group = _m.group(1) if _m := re.match(data) else None > > > > That's a performance argument, not a readability one (as if you don't > > care about performance, you can just repeat the subexpression). > How does that differ from the part of what I said that you did retain above? >> sometimes it allows a more compact way of reusing an expensive >> subexpression by giving it a name. Which they already do by giving >> it a name in a separate statement, so the possible improvement would >> be in brevity rather than performance. You already realized the performance gain could be achieved by using two statements. The _additional_ performance gain by using assignment expressions is at best trivial (it may save a LOAD_FAST opcode to fetch the object bound to `_m` for the `if` test). So, no, gaining performance is _not_ the motivation here. You already had a way to make it "run fast'. The motivation is the _brevity_ assignment expressions allow while _retaining_ all of the two-statement form's advantages in easier readability, easier reasoning, reduced redundancy, and performance. As Guido said, in the PEP, of the example you gave here: Guido found several examples where a programmer repeated a subexpression, slowing down the program, in order to save one line of code It couldn't possibly be clearer that Guido thought the programmer's motivation was brevity ("in order to save one line of code"). Guido only happened to mention that they were willing to slow down the code to get that brevity, but, as above, they were also willing to make the code harder to read, reason about, and maintain. With the assignment expression, they don't have to give up any of the latter to get the brevity they mistakenly _think_ ;-) they care most about - and, indeed, they can make it even briefer. I sure don't count it against the PEP that it may trick people overly concerned with brevity into writing code that's clearer and faster too, but that's a tiny indirect part of the PEP's motivation_s_ (note the plural again). -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Jul 1 01:50:48 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 30 Jun 2018 22:50:48 -0700 Subject: [Python-Dev] Help preventing SIGPIPE/SIG_DFL anti-pattern. In-Reply-To: References: <922692be-4089-0162-4c3c-93bad5a2c3d8@freebsd.org> <5B38103A.2050002@canterbury.ac.nz> Message-ID: On Sat, Jun 30, 2018 at 9:02 PM Alfred Perlstein wrote: > > > On 6/30/18 4:20 PM, Greg Ewing wrote: > > Alfred Perlstein wrote: > >> I am asking if there's a way we can discourage the use of > >> "signal(SIGPIPE, SIG_DFL)" unless the user really understands what > >> they are doing. > > > > Maybe there's some way that SIGPIPEs on stdout could be handled > > differently by default, so that they exit silently instead of > > producing an ugly message. That would remove the source of pain > > that's leading people to do this. > > > Thank you Greg, I can poke around into this, it would be a bit of a > challenge as the descriptor which causes BrokenPipeError does not appear > to be stored within the exception so differentiating it from other > exceptions might be a bit tricky. > > I will look into this in the coming weeks. Any tips on accomplishing > this? I was thinking of encoding the fd responsible for causing the > error into the exception somehow and then checking to see if it was > stdout, then not reporting on it. > There's PyErr_SetFromErrnoWithFilenameObject(), which generates an OSError with a filename. We could have a similar call PyErr_SetFromErrnoWithFileDescriptor(), which generates an OSError (or a subclass like BrokenPipeError) with a file descriptor. Just pick a new attribute name to store the fd on, e.g. file_descriptor (I guess it would default to None). Then of course you will have to write the code that calls this instead of plain PyErr_SetFromErrno() for those syscalls where a file descriptor is present. And when Python exits with a BrokenPipeError it could suppress printing the stack trace when the file_descriptor field equals 1. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Jul 1 02:03:31 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 01 Jul 2018 18:03:31 +1200 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: <5B386EB3.8070604@canterbury.ac.nz> Nick Coghlan wrote: > That's a performance argument, not a readability one (as if you don't > care about performance, you can just repeat the subexpression). Repeated subexpressions can be a readability issue too, since you have to examine them to notice they are actually the same. They also provide an opportunity to make the error of not making them the same when they should be, and add the maintenance burden of ensuring they stay the same when changes are made. -- Greg From ncoghlan at gmail.com Sun Jul 1 02:11:41 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 1 Jul 2018 16:11:41 +1000 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: On 1 July 2018 at 14:32, Tim Peters wrote: > [Nick] > >> The PEP specifically cites this example as motivation: > > The PEP gives many examples. Your original was a strawman > mischaracterization of the PEP's _motivations_ (note the plural: you only > mentioned "minor performance improvement", and snipped my listing of the > major motivations). I listed two motivations, not one: 1. Minor performance improvements (the "avoiding repeated subexpressions without use multiple statements" rational) 2. Making certain coding patterns easier to spot (the loop-and-a-half and if-elif chaining cases) Technically, avoid repeated subexpressions without requiring a separate line also falls into the second category. The subsequent interaction with comprehensions and generator expressions is an interesting side effect of extending the basic idea to a fully coherent and self-consistent proposal, not one of the original motivations for it. >> group = re.match(data).group(1) if re.match(data) else None > >> That code's already perfectly straightforward to read and write as a > >> single line, > > I disagree. In any case of textual repetition, it's a visual > pattern-matching puzzle to identify the common substrings (I have to > visually scan that line about 3 times to be sure), and then a potentially > difficult conceptual puzzle to figure out whether side effects may result in > textually identical substrings evaluating to different objects. That's why > "refererential transparency" is so highly valued in functional languages > ("if subexpressions are spelled the same, they evaluate to the same result, > period" - which isn't generally true in Python - to get that enormously > helpful (to reasoning) guarantee in Python you have to ensure the > subexpression is evaluated exactly once). > > And as you of all people should be complaining about, textual repetition is > also prone to "oops - forgot one!" and "oops! made a typo when changing the > second one!" when code is later modified. That's a reasonable readability based argument, but it's not what the PEP currently gives as a motivation for this aspect of the proposal. >> so the only reason to quibble about it > > I gave you three better reasons to quibble about it just above ;-) Then add them to the PEP, as what's currently there really isn't offering a compelling motivation for this aspect of the proposal :) >> is because it's slower than the arguably less clear two-line alternative: > >> _m = re.match(data) > >> group = _m.group(1) if _m else None > > > I find that much clearer than the one-liner above: the visual pattern > matching is easier because the repeated substring is shorter and of much > simpler syntactic structure; it guarantees _by construction_ that the two > instances of `_m` evaluate to the same object, so there's no possible > concern about that (it doesn't even matter if you bound `re` to some > "non-standard" object that has nothing to do with Python's `re` module); and > any later changes to the single instance of `re.match(data)` don't have to > be repeated verbatim elsewhere. It's possible that it runs twice as fast > too, but that's the least of my concerns. I agree with this, but also think the two-line form is a perfectly acceptable way of spelling it, and a perfectly acceptable refactoring of the one-line form with duplicated subexpressions to improve maintainability. > All of those advantages are retained in the one-liner too if an assignment > expression can be used in it. Sure, but the open design question is whether folks that would have written the one-liner with repeated subexpressions are going to be any more likely to use an assignment expression to avoid the repetition without prompting by a more experienced developer than they are to use a separate preceding assignment statement. That assessment of "What is the increased chance that the repeated subexpression will be avoided when the code is first written?" then gets traded off against the overall increase in language complexity arising from allowing name bindings in arbitrary subexpressions. Don't get me wrong, I now agree that the proposal in PEP 572 is the most coherent and self-consistent approach to assignment expressions that we could pursue given the existing scoping semantics of comprehensions and generator expressions. The remaining point of contention is only the "Inevitable cost of change" one: given the level of disruption this will cause in the way that Python gets taught to new users, is it giving a commensurate pay-off in increased semantic expressiveness? My answer to that question remains "No", while your answer is either "Yes" or "I don't see why that should matter" (I'm genuinely unsure which). >>> sometimes it allows a more compact way of reusing an expensive >>> subexpression by giving it a name. Which they already do by giving >>> it a name in a separate statement, so the possible improvement would >>> be in brevity rather than performance. > > You already realized the performance gain could be achieved by using two > statements. The _additional_ performance gain by using assignment > expressions is at best trivial (it may save a LOAD_FAST opcode to fetch the > object bound to `_m` for the `if` test). > > So, no, gaining performance is _not_ the motivation here. You already had a > way to make it "run fast'. The motivation is the _brevity_ assignment > expressions allow while _retaining_ all of the two-statement form's > advantages in easier readability, easier reasoning, reduced redundancy, and > performance. I never said the motivation was to gain performance relative to the two-statement version - I said the motivation given in the PEP is to gain performance relative to the *repeated subexpression* version, *without* making the transition to the already supported two-statement version. > As Guido said, in the PEP, of the example you gave here: > > Guido found several examples where a programmer repeated > a subexpression, slowing down the program, in order to save > one line of code > > > It couldn't possibly be clearer that Guido thought the programmer's > motivation was brevity ("in order to save one line of code"). Guido only > happened to mention that they were willing to slow down the code to get that > brevity, but, as above, they were also willing to make the code harder to > read, reason about, and maintain. With the assignment expression, they > don't have to give up any of the latter to get the brevity they mistakenly > _think_ ;-) they care most about - and, indeed, they can make it even > briefer. The quoted paragraph from the PEP clearly states that the reason the repeated subexpression is considered a problem is because it slows down the program, not because it repeats code. As noted above, the PEP could certainly be updated to point out that repeating subexpressions is problematic for more reasons than just speed, but that isn't what it currently says. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Sun Jul 1 02:22:29 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Jul 2018 16:22:29 +1000 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: <20180701062229.GV14437@ando.pearwood.info> On Sat, Jun 30, 2018 at 11:32:03PM -0500, Tim Peters wrote: [...] > So, no, gaining performance is _not_ the motivation here. You already had > a way to make it "run fast'. The motivation is the _brevity_ assignment > expressions allow while _retaining_ all of the two-statement form's > advantages in easier readability, easier reasoning, reduced redundancy, and > performance. I think the two of you (Tim and Nick) are in violent agreement :-) In fairness to Nick performance is _a_ motivation here, in the sense of "how can we avoid making redundant multiple calls to a function in an expression without splitting it into multiple statements?". If performance were the *only* concern, then I agree with your point that we already have a solution. Simply refactor this: process(expensive_call(arg), expensive_call(arg) + 1) to this: useful_value = expensive_call(arg) process(useful_value, useful_value + 1) But since performance is not the *only* concern, all the other factors you refer to (saving vertical space, DRY, side-effects, etc) count too. Assignment expressions aren't valuable because they give us One Big Win. They're valuable because they give us Many Little Wins, which we (proponents of PEP 572) believe will outweigh the (minor) additional costs in complexity and opportunities for abuse. The closest (in my opinion) assignment expressions comes to a One Big Win is to reduce the need for cascades of if... statements: m = re.match(pattern, text) if m: ... else: m = re.match(other_pattern, text) if m: ... else: m = re.match(third_pattern, text) if m: ... which wastes both vertical and horizontal space. If that were the *only* win, I'd consider dedicated syntax for if/elif statements alone, but it isn't. Little wins include: - use in while statement headers, "while x:= expr" - DRY inside comprehensions, as an alternative to the neat but somewhat obscure idiom: [(x, x+1, x*2) for a in it for x in (func(a),) if x] - running totals and similar inside comprehensions, where we need a way to initialise the total on the first iteration total = 0 [total := total + a for a in it] - and examples such as Tim's use of Heron's Formula: area = sqrt((s := (a+b+c)/2)*(s-a)*(s-b)*(s-c)) This last one is, I think, the only one where the postfix form reads better, but maybe I only say that because I'm more familiar with it: area = sqrt(s*(s-a)*(s-b)*(s-c)) where s = (a+b+c)/2 -- Steve From tim.peters at gmail.com Sun Jul 1 03:36:28 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 1 Jul 2018 02:36:28 -0500 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: I think I'll bow out of this now. It's just too tedious. Like here: [Nick] > I never said the motivation was to gain performance relative to the > two-statement version - I said the motivation given in the PEP is to > gain performance relative to the *repeated subexpression* version, > *without* making the transition to the already supported two-statement > version. This is what you wrote: >> "NAME := EXPR" exists on a different level of complexity, since it >> adds name binding in arbitrary expressions for the sake of minor >> performance improvement in code written by developers that are >> exceptionally averse to the use of vertical screen real estate, I'm not telepathic, so took it for it what said. It didn't say a word about "repeated expressions", nor a word about "two-statement versions" It did say "minor" performance improvements, which sure suggested to me that you had in mind the tiny changes in bytecode that can result from squashing a current two-statement spelling into a one-statement form with an embedded assignment. That fit both "minor performance improvement" and "vertical screen space", which were the only the two clues I had to go on. In the example you eventually gave in a later message, the performance difference was more on the order of a factor 2 (regexp searches can be expensive indeed), which matches nobody's idea of "minor". So the use of "minor" wholly ruled out in my mind that you had _anything_ akin to your eventual example in mind. For the rest, sure, words could be added to the PEP without end. At least some of the points I covered were telegraphically mentioned in my Appendix, although I'll grant that nobody else is telepathic either ;-) For the question: > > The remaining point of contention is only the "Inevitable cost of > > change" one: given the level of disruption this will cause in the way > > that Python gets taught to new users, is it giving a commensurate > > pay-off in increased semantic expressiveness? > > > > My answer to that question remains "No", while your answer is either > > "Yes" or "I don't see why that should matter" (I'm genuinely unsure > > which). I don't know, and I'm not qualified to guess - I don't teach Python to new users for a living. Decades ago I tutored "advanced" engineering undergrads in a variety of science-y subjects, and was routinely surprised by what broke their brains. I have noted that assignment expressions have been part of a great many languages for decades (this isn't cutting edge tech), questions about them are conspicuous by absence on StackOverflow (everyone else seems to teach them effectively), and skimming various online teaching materials for other languages convinced me that none of those professional educators thought it needed more than a page to teach (and to teach them with relatively few words). There's really not much to them: people have to learn what binding means in Python regardless, pretty much starting in the first hour, yes? "Binding" on its own is the hard part. If they don't drop out and stick around for the 10-minute lesson on assignment expressions 3 weeks later, will _that_ finally break their brains? "Wow - it not only binds the name, but ALSO returns the object that was bound?! I just can't take it - please, let's go back to the lesson on metaclasses" just doesn't seem likely to me ;-) At heart, ":=" could make a good case for being the simplest of all Python's operators. It does no computation at all, and you don't even have to worry about a dunder method changing its meaning depending on context. So, ya, when someone claims they'll make Python significantly harder to teach, I'm skeptical of that claim. Which does not mean I believe it's wrong - it means I'm skeptical. I would also be skeptical of a claim that teaching them would be no trouble at all, except nobody has made such a claim ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From dickinsm at gmail.com Sun Jul 1 05:17:02 2018 From: dickinsm at gmail.com (Mark Dickinson) Date: Sun, 1 Jul 2018 10:17:02 +0100 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180622164710.4906f9b0@fsol> Message-ID: On Fri, Jun 22, 2018 at 7:28 PM, Chris Barker via Python-Dev < python-dev at python.org> wrote: > > But once it becomes a more common idiom, students will see it in the wild > pretty early in their path to learning python. So we'll need to start > introducing it earlier than later. > > I think this reflects that the "smaller" a language is, the easier it is > to learn. > For what it's worth, Chris's thoughts are close to my own here. I and several of my colleagues teach week-long Python courses for Enthought. The target audience is mostly scientists and data scientists (many of whom are coming from MATLAB or R or IDL or Excel/VBA or some other development environment, but some of whom are new to programming altogether), and our curriculum is Python, NumPy, SciPy, Pandas, plus additional course-specific bits and pieces (scikit-learn, NLTK, seaborn, statsmodels, GUI-building, Cython, HPC, etc., etc.). There's a constant struggle to keep the Python portion of the course large enough to be coherent and useful, but small enough to allow time for the other topics. To that end, we separate the Python piece of the course into "core topics" that are essential for the later parts, and "advanced topics" that can be covered if time allows, or if we get relevant questions. I can't see a way that the assignment expression wouldn't have to be part of the core topics. async stuff only appears in async code, and it's easy to compartmentalize; in contrast, I'd expect that once the assignment expression took hold we'd be seeing it in a lot of code, independent of the domain. And yes, I too see enough confusion with "is" vs == already, and don't relish the prospect of teaching := in addition to those. That's with my Python-teaching hat on. With my Python-developer hat on, my thoughts are slightly different, but that's off-topic for this thread, and I don't think I have anything to say that hasn't already been said many times by others, so I'll keep quiet about that bit. :-) -- Mark -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Sun Jul 1 08:06:13 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Sun, 1 Jul 2018 13:06:13 +0100 Subject: [Python-Dev] PEP 544 (Protocols): adding a protocol to a class post-hoc In-Reply-To: References: Message-ID: On 30 June 2018 at 23:54, Tin Tvrtkovi? wrote: > [...] > > An attrs class has a special class-level field, __attrs_attrs__, which > holds the attribute definitions. So maybe we can define a protocol: > > class AttrsClass(Protocol): > __attrs_attrs__: ClassVar[Tuple[Attribute, ...]] > > then we could define asdict as (simplified): > > def asdict(inst: AttrsClass) -> Dict[str, Any]: > ... > > and it should work out. My question is how to actually add this protocol > to attrs classes. Now, we currently have an attrs plugin in mypy so maybe > some magic in there could make it happen in this particular case. > Just add a Var with an appropriate name and type to the TypeInfo. This is literary a dozen lines of code, you can ask on mypy tracker or typing Gitter chat for more help. > My second use case is a small library I've developed for work, which > basically wraps attrs and generates and sticks methods on a class for > serialization/deserialization. Consider the following short program, which > does not typecheck on the current mypy. > > class Serializable(Protocol): > def __serialize__(self) -> int: > ... > > def make_serializable(cl: Type) -> Type: > cl = attr.s(cl) > cl.__serialize__ = lambda self: 1 > return cl > > > @make_serializable > class A: > a: int = attr.ib() > > > def serialize(inst: Serializable) -> int: > return inst.__serialize__() > > > serialize(A(1)) > > error: Argument 1 to "serialize" has incompatible type "A"; expected > "Serializable" > error: Too many arguments for "A" > > I have no desire to write a mypy plugin for this library. So I guess what > is needed is a way to annotate the class decorator, telling mypy it's > adding a protocol to a class. It seems to have trouble getting to this > conclusion by itself. > A proper solution for this would be to introduce intersection types, and type your decorator as following: T = TypeVar('T') def make_serializable(cls: Type[T]) -> Type[Intersection[T, Serializable]]: ... However, intersection types are unlikely to appear in mypy this year. In best case they could appear around mid-2019, so you are better with writing a plugin for now. > (The second error implies the attrs plugin doesn't handle wrapping attr.s, > which is unfortunate but a different issue.) > Your decorator is typed as (Type) -> Type, thats it, the function is a black box for mypy (with few special exceptions), if some effect of a function is not declared in its signature, then it is lost forever. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Sun Jul 1 11:35:08 2018 From: mike at selik.org (Michael Selik) Date: Sun, 1 Jul 2018 08:35:08 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: On Sun, Jul 1, 2018 at 12:39 AM Tim Peters wrote: > So, ya, when someone claims [assignment expressions will] make Python > significantly harder to teach, I'm skeptical of that claim. > I don't believe anyone is making that claim. My worry is that assignment expressions will add about 15 to 20 minutes to my class and a slight discomfort. As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional. On Sun, Jul 1, 2018 at 2:19 AM Mark Dickinson wrote: > There's a constant struggle to keep the Python portion of the course large > enough to be coherent and useful, but small enough to allow time for the > other topics. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mcepl at cepl.eu Sun Jul 1 17:48:56 2018 From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Sun, 01 Jul 2018 23:48:56 +0200 Subject: [Python-Dev] Failing tests [Was: Re: Python 3.7.0 is now available! (and so is 3.6.6)] References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: On 2018-06-28, 00:58 GMT, Ned Deily wrote: > On behalf of the Python development community and the Python 3.7 release > team, we are pleased to announce the availability of Python 3.7.0. I am working on updating openSUSE packages to python 3.7, but I have hit quite large number of failing tests (the testsuite obviously passed with 3.6), see https://build.opensuse.org/package/show/home:mcepl:work/python3 (click on the red "failed" label to get logs). I fell into a bout of depression, only to discover that we are not alone in this problem ... Debian doesn't seem to do much better https://is.gd/HKBU4j. Surprisingly, Fedora seems to pass the testsuite https://is.gd/E0KA53; interesting, I will have to investigate which of their many patches did the trick. Anybody has any idea, what's going on, please? Did anybody on the python.org side run test suites on Linux? Best, Mat?j -- https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 The difference between death and taxes is death doesn't get worse every time Congress meets -- Will Rogers From mhroncok at redhat.com Sun Jul 1 18:59:13 2018 From: mhroncok at redhat.com (=?UTF-8?Q?Miro_Hron=c4=8dok?=) Date: Mon, 2 Jul 2018 00:59:13 +0200 Subject: [Python-Dev] Failing tests [Was: Re: Python 3.7.0 is now available! (and so is 3.6.6)] In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: On 1.7.2018 23:48, Mat?j Cepl wrote: > On 2018-06-28, 00:58 GMT, Ned Deily wrote: >> On behalf of the Python development community and the Python 3.7 release >> team, we are pleased to announce the availability of Python 3.7.0. > > I am working on updating openSUSE packages to python 3.7, but > I have hit quite large number of failing tests (the testsuite > obviously passed with 3.6), see > https://build.opensuse.org/package/show/home:mcepl:work/python3 > (click on the red "failed" label to get logs). I fell into > a bout of depression, only to discover that we are not alone in > this problem ... Debian doesn't seem to do much better > https://is.gd/HKBU4j. Surprisingly, Fedora seems to pass the > testsuite https://is.gd/E0KA53; interesting, I will have to > investigate which of their many patches did the trick. Note that we (=Fedora) unfortunately skip some tests. https://src.fedoraproject.org/rpms/python3/blob/master/f/python3.spec#_1051 https://src.fedoraproject.org/rpms/python3/blob/master/f/00160-disable-test_fs_holes-in-rpm-build.patch https://src.fedoraproject.org/rpms/python3/blob/master/f/00163-disable-parts-of-test_socket-in-rpm-build.patch -- Miro Hron?ok -- Phone: +420777974800 IRC: mhroncok From chris.barker at noaa.gov Sun Jul 1 19:05:50 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 1 Jul 2018 16:05:50 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: On Sun, Jul 1, 2018 at 8:35 AM, Michael Selik wrote: > As Mark and Chris said (quoting Mark below), this is just one straw in the > struggle against piling too many things on the haystack. Unlike some > changes to the language, this change of such general use that it won't be > an optional topic. Once widely used, it ain't optional. > Exactly -- and I also emphasis that this would complicate the language in a broad way -- a much bigger deal than adding a self contained new expression or even the nonlocal keyword. But to be clear about my take -- it will make the language that little bit harder to teach, but it will also add a bit of complexity that will effect us non-newbies as well. Is it worth it? maybe. I know I like a better way to express the loop-and-a-half concept in particular. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Sun Jul 1 20:23:16 2018 From: nad at python.org (Ned Deily) Date: Sun, 1 Jul 2018 20:23:16 -0400 Subject: [Python-Dev] Failing tests [Was: Re: Python 3.7.0 is now available! (and so is 3.6.6)] In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: <7802E7B2-40C5-4E45-A3BB-C52AA363232A@python.org> On Jul 1, 2018, at 17:48, Mat?j Cepl wrote: > On 2018-06-28, 00:58 GMT, Ned Deily wrote: >> On behalf of the Python development community and the Python 3.7 release >> team, we are pleased to announce the availability of Python 3.7.0. > > I am working on updating openSUSE packages to python 3.7, but > I have hit quite large number of failing tests (the testsuite > obviously passed with 3.6), see > https://build.opensuse.org/package/show/home:mcepl:work/python3 > (click on the red "failed" label to get logs). I fell into > a bout of depression, only to discover that we are not alone in > this problem ... Debian doesn't seem to do much better > https://is.gd/HKBU4j. Surprisingly, Fedora seems to pass the > testsuite https://is.gd/E0KA53; interesting, I will have to > investigate which of their many patches did the trick. > > Anybody has any idea, what's going on, please? Did anybody on > the python.org side run test suites on Linux? Without doing a detailed analysis of how your build system is set up, I do note you have "--enable-shared" set on ./configure which is often the source of problems. Make sure that you are building from a clean build directory and that you are not building on a system that already has an installed Python 3.7 pre-release at the same prefix location. Of course we run test suites on Linux :) Each time a change is merged our CI tests are run on Linux, we have a fleet of buildbots that test every commit on various flavors of Linux, and prior to release the release manager does a smoke test of building and testing on at least one Linux variant. That's not to say there couldn't be multiple configurations with undetected problems but the number of problems here points to something much more basic. If necessary, please open an issue on bugs.python.org so we can get this resolved quickly. Thanks! -- Ned Deily nad at python.org -- [] From steve at pearwood.info Sun Jul 1 20:25:42 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 2 Jul 2018 10:25:42 +1000 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: <20180702002542.GX14437@ando.pearwood.info> On Sun, Jul 01, 2018 at 08:35:08AM -0700, Michael Selik wrote: > On Sun, Jul 1, 2018 at 12:39 AM Tim Peters wrote: > > > So, ya, when someone claims [assignment expressions will] make Python > > significantly harder to teach, I'm skeptical of that claim. > > > > I don't believe anyone is making that claim. My worry is that assignment > expressions will add about 15 to 20 minutes to my class and a slight > discomfort. How do people who teach other languages deal with this? Assignment expressions are hardly a new-fangled innovation of Python's. They're used in Java, Javascript, Ruby, Julia, R, PHP and of course pretty much the entire C family (C, C++, C# at least). What do teachers of those languages do? R has a similar demographic of users (strong in the sciences, many beginners to programming, growing in popularity). Once R teachers have taught that you can assign values like this: x = 1 + 2 does it take them 15-20 minutes to teach that you can do this as well? y = (x = 1 + 2) + 3 Admittedly R has the advantage that they don't have to teach a distinct assignment syntax and explain *why* it ought to be distinct. But countering that, they have *four* different ways of doing assignment. x <- expression expression -> x x = expression assign('x', expression) (all of which can be used as expressions). > As Mark and Chris said (quoting Mark below), this is just one straw in the > struggle against piling too many things on the haystack. Unlike some > changes to the language, this change of such general use that it won't be > an optional topic. Once widely used, it ain't optional. Without knowing the details of your course, and who they are aimed at, we cannot possibly judge this comment. Decorators are widely used, but surely you don't teach them in a one day introductory class aimed at beginners? Here is the syllabus for a ten week course: https://canvas.uw.edu/courses/1026775/pages/python-100-course-syllabus Note that decorators and even regular expressions don't get touched until week ten. If you can't fit assignment expressions in a ten week course, you're doing something wrong. If you can't fit them in a two hour beginners course, there is so much more that you aren't covering that nobody will notice the lack. -- Steve From mike at selik.org Sun Jul 1 21:09:27 2018 From: mike at selik.org (Michael Selik) Date: Sun, 1 Jul 2018 18:09:27 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: <20180702002542.GX14437@ando.pearwood.info> References: <20180702002542.GX14437@ando.pearwood.info> Message-ID: On Sun, Jul 1, 2018 at 5:28 PM Steven D'Aprano wrote: > On Sun, Jul 01, 2018 at 08:35:08AM -0700, Michael Selik wrote: > > On Sun, Jul 1, 2018 at 12:39 AM Tim Peters wrote: > > > > > So, ya, when someone claims [assignment expressions will] make Python > > > significantly harder to teach, I'm skeptical of that claim. > > > > > > > I don't believe anyone is making that claim. My worry is that assignment > > expressions will add about 15 to 20 minutes to my class and a slight > > discomfort. > > How do people who teach other languages deal with this? > Python may be in a unique situation in the history of programming. It wouldn't surprise me if more people learned Python last year than any other programming language. > Assignment expressions are hardly a new-fangled innovation of Python's. > They're used in Java, Javascript, Ruby, Julia, R, PHP and of course > pretty much the entire C family (C, C++, C# at least). What do > teachers of those languages do? > Assignment expressions are not the issue. The real question is: How do open-source projects balance the addition of new features against the growth of complexity? It's the same as that "Remember the Vasa" thread. [...] R [has] *four* different ways of doing assignment. > I think that's a good explanation of why I teach Python and not R. The first time someone asked me to teach a data science course, Python wasn't the clear winner. In fact, R may have been more popular among statisticians. I picked Python for the same reason it's more popular in the industry -- it's the easiest* to use. * Easiest that gets the job done well. > As Mark and Chris said (quoting Mark below), this is just one straw in the > > struggle against piling too many things on the haystack. Unlike some > > changes to the language, this change of such general use that it won't be > > an optional topic. Once widely used, it ain't optional. > > Without knowing the details of your course, and who they are aimed at, > we cannot possibly judge this comment. I disagree. I think the sentiment holds for a great variety of courses and audiences. > Decorators are widely used, but surely you don't teach them in a one day > introductory class aimed at beginners? > Most of the time, no. Once, yes, because that's what the team needed. I was pretty proud of myself for handling that one. Because I had to teach decorators early, many other important topics were excluded. Here is the syllabus for a ten week course: > https://canvas.uw.edu/courses/1026775/pages/python-100-course-syllabus > > Note that decorators and even regular expressions don't get touched > until week ten. If you can't fit assignment expressions in a ten week > course, you're doing something wrong. If you can't fit them in a two > hour beginners course, there is so much more that you aren't covering > that nobody will notice the lack. > It's not about any one particular topic, but the trade-offs between topics. A 10-week lecture course might be 30 hours of lecture, comparable to a 4-day "bootcamp" style course. I assure you that 4 days doesn't feel long enough when those last few hours are winding down. There's always more to say. -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcidy at gmail.com Sun Jul 1 23:21:19 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Sun, 1 Jul 2018 20:21:19 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: This cynical view on students is shocking! Everyone on this list has been a student or a learner for far longer than an educator, and the perspective from students and learners are far more important than educators to assess this angle regardless. Can anyone adequately explain why this specific modality of learning, a student-in-a-seat based educator, must outweigh all other modalities learners use to increase knowledge and skill, from the perspectives of policy, tool creation, and each of our time spent learning? Shortest story: Teach not to re-use names. Short story: 1) What about the full mosaic of learning vs. this myopic view on seat-based student-educator interaction? 2) What about smart, motivated, diligent and cautious students? 3) What weight should educator opinion be given with respect to providing convenience to professional Python programmers? 4) Who is this Student Stupid von Densemeister anyways? 5) Are assignment expressions convenience and is any danger the pose unmitagatble? 6) Consider adding an "Important Topics not Covered" or "Further Reading" reading section to your class description 7) Creating examples showing this effect is easy, especially when not actually re-using the name in the expression for explanatory purposes. it's the same as creating examples showing how re-use works in comprehensions. Let's stop constructing these fake Students. They only work as appeals to the people we have come across whose lack of understanding has made our life painful. This construction is actively filtering all the good students for the sake of influencing this decision, yet again punishing or discounting the intelligent, quick, and diligent. And what of this underlying premise that educator's should _significantly_ influence language development? Limiting Python's tools to Student Straw-man's ability to learn is just dissonant, they have nothing to do with each other, nor does this cause-effect relationship actually exist. Let's evaluate this reductionist statement: "I understand X, but this other person is not capable of understanding X, therefore X should not exist" Is has there ever been an X for which this is true, let alone the backwardation necessary to fully close the statement? The actual argument is far less reductionist, yet even more ridiculous: "I understand X, this other person may take time to learn X, and may use X wrong, therefore X should not exist" "I understand assignment expressions, but this other class of person may take time to learn assignment expressions, and may use assignment expressions wrong, therefore assignment expressions should not be accepted" Rhetorically I disagree with how teaching is being presented, to the point of near insult (for me lacking a better term). You are saying these statements about _my_ learning path, (though not personally of course.) Each of you occupied a role of student at some point, and each of these statements are being made about your path as well. Do these ring true of your student experience? What about your much broader experience as a _learner_? You think a tool shouldn't exist because it took you time to learn it and you wrote some hard to debug code, and possibly crashed production, got fired, lost your house and your pet snake, and crashed the planet into the sun? Now I yield, I will accept this position: all/some students cannot learn this (or it's too complex to teach), but they must learn this during some class to quickly become effective python developers. How much weight should this position have in this decision? Let's appeal to the learner in us. How much of our learner's path, percentage of total time learning all things python related, has been in a seat listening to someone else, and that's the only place from which we gained the knowledge to meet the educator's objective? This time spent in a class, how does that compare to hours in other learning modalities? Is this percentage not exactly the weight assigned to that position? Are people hired from pure class-room based experience expected to require zero further learning? Are people more valuable based on classroom hours or work hours? As for handling teaching the subject or not, this is easily remedied with how I do it: "Important Topics not Covered", with resources. Anyone here can rightfully claim educator status by having taught another person something related to this language, which includes at-work mentoring, informal discussions, posting/replying on SO, blogging, etc. Are they not being solicited to comment as well? It's possible to answer this question while vehemently disagreeing with the PEP. This focus on people who are being ostensibly paid to teach is myopic. Concretely, it's clear to me that parent-local effects can be dangerously non-obvious when reading and mimicking code without undertsanding. But when? And how to guard against? How about this: teach proper (i.e. not) re-using names. The name will still be ejected to the parent scope, but there won't be any use of it. Teach the explicit declare pattern first (as everyone does anyways), explain to not re-use. Regardless of when re-use is done, it is always (or only) as dangerous as the effect the bound value has anyways, and any re-use has the potential to trigger the same dangerous behaviors. Must we continue this educator assessment of PEP 572? Educators are not gate-keepers, and the only measure of their success is what students learn, and students have a far larger and more fine-grained mosaic now than ever. Seat-based education plays a far smaller role in anyone's learning path than ever before, and even while in the seat they have access to Google. All this yield to tiling the outcome in the favor of the educator by assigning the goal meeting to them, not to the diligence of the student to learn. How unfair to the student. I consider this position purposefully ignoring motivated self-learners of high ability and skill, or just plain old diligent programmers who learned to read specs before using tools. I don't like posting to python-dev because it's not really my realm, but this topic is insanely tilted against PEP572 for the most ridiculous of reasons. I am Pro-572 , so I have decided to join critique of the educator position. I would rather do it on python-Ideas, and I want more types of educators solicited, as well as students and learners. And yes, lets assess your specific lesson plans if you will make claims it's not possible. Do you even teach the difference between assignments and expressions at all? To have this raised in the this stage of this PEP, and on the dev list, illustrates how long it took educators to understand the tool to begin with, as opposed to those who understood even if they disagree. To have a room full of seat-based educators provide feedback on a tool they have had 30m to understand as critical to shaping the language is not defensible in these lower courts. This angle cannot withstand rigorous scrutiny because each premise is false and it rests the majority of students being dense. They aren't, and the vast majority of learners don't need class-room education anyways, so what is this weight being placed on the opinion of these educators? I agree it's important to hear it, but dimishingly. This is not to say that PEP572 should be accepted otherwise. However, this educator angle, raised only now and depending on this platonically dumb student and a non-creative approach to education, is just pure straw-man to distract from the point at hand. And while I have repeated "straw-man" as the critique, I dislike leaning on such a debate-team crutch, and of course employing straw-man doesn't mean the point is invalid. it just means the rhetoric is bad. However, as the underlying premise is a severe minority opinion yet claiming to be broad, and the absolute percentage of solicited opinions is 0% (to the 17th place), I don't see any importance to the position of educators right now, especially since these educators in the thread are complaining about an increase in their personal work, for which it appears they were compensated (this is pretty bad straw-man, sorry). And to repeat, what about the learners? Time spent in seat based education is so insanely small now. The fact that the name is ejected to the parent stop is not terribly difficult, and appropriately factored examples showing the the effect of scope ejection will be easy to construct, even if trivial for explanation purposes. So what, exactly, is the issue with learning it? On Sun, Jul 1, 2018 at 4:35 PM Chris Barker via Python-Dev wrote: > > On Sun, Jul 1, 2018 at 8:35 AM, Michael Selik wrote: >> >> As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional. > > > Exactly -- and I also emphasis that this would complicate the language in a broad way -- a much bigger deal than adding a self contained new expression or even the nonlocal keyword. > > But to be clear about my take -- it will make the language that little bit harder to teach, but it will also add a bit of complexity that will effect us non-newbies as well. > > Is it worth it? maybe. I know I like a better way to express the loop-and-a-half concept in particular. > > -CHB > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/marcidy%40gmail.com From chris.barker at noaa.gov Mon Jul 2 01:59:31 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 1 Jul 2018 22:59:31 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: <20180702002542.GX14437@ando.pearwood.info> References: <20180702002542.GX14437@ando.pearwood.info> Message-ID: On Sun, Jul 1, 2018 at 5:25 PM, Steven D'Aprano wrote: > > > an optional topic. Once widely used, it ain't optional. > > Without knowing the details of your course, and who they are aimed at, > we cannot possibly judge this comment. Decorators are widely used, but > surely you don't teach them in a one day introductory class aimed at > beginners? > > Here is the syllabus for a ten week course: > > https://canvas.uw.edu/courses/1026775/pages/python-100-course-syllabus > That would be mine ;-) > Note that decorators and even regular expressions don't get touched > until week ten. I actually often don't ever get to regex -- why? because they are a topic all of their own, and not unique to Python. And if code uses a regex, it doesn't change anything about any other code anywhere. IN sort, regex's are a library, not a language feature. Which brings us to decorators (and I'm going to add context managers, as its similar). Decorators (and even more so context managers) are used commonly enough that we certainly have to introduce them in a intro class. But there is a really big distinction between having some idea how to use them, and knowing how they work / how to write them yourself. So I introduce: with open(filename) as the_file: do_somethign_with(the_file) early on, with only a hand-wavy explanation of what that with is all about. And when i get to properties, I teach them: @property def a_method(self): .... with also a hand-wavy explanation of what that @ symbol is. and if := catches on, I expect it will be far more common that either of those, especially in the simple script style codes that newbies are going to be reading/writing at first. But in the end, I, at least, am not trying to make the case that assignment expressions are going to be particularly difficult to understand, but that they are a significant complication, and any new feature like that makes the language more complex. That's it. I taught myself Python with version 1.5 in 1998 -- and have been teaching it for I think almost ten years. In that time, the language has grown substantially in complexity, and it does make it harder to teach. We (UWPCE) recently revamped our 3-class program, and had a lot of debate about how early to introduce things -- one of the instructors wanted to leave off lambda and comprehensions 'till the second course in the sequence, as part of functional programming. I I think he was right, if we were teaching it in a more isolated environment, but both of those show up in example code all over the place, so I've found it's necessary to introduce a lot at a high level early: decorators comprehensions lambda context managers *args and **kwargs (off the top of my head) So there is basically no way that we won't have to add assignment expressions early on. As someone else said: it's another straw on the haystack. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Mon Jul 2 02:35:51 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 2 Jul 2018 01:35:51 -0500 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: [Tim] >> ... >> So, ya, when someone claims [assignment expressions will] make >> Python significantly harder to teach, I'm skeptical of that claim. [Michael Selik] > I don't believe anyone is making that claim. I haven't seen it in this specific thread, but the larger discussion has been going on for several months. > My worry is that assignment expressions will add about 15 to 20 > minutes to my class and a slight discomfort. So not intractable - which is my high-order bit ;-) For those who want more bits of precision (perhaps Guido), while quantification is good, it needs context to provide insight. Like, out of how many class hours total? Is 15-20 minutes a little, a lot, par for the course ... compared to other topics? Will it require you to drop other topics? Would you _save_ twice as much class time if we got rid of "is"? ;-) > As Mark and Chris said (quoting Mark below), this is just one straw in > the struggle against piling too many things on the haystack. Unlike > some changes to the language, this change of such general use that > it won't be an optional topic. Once widely used, it ain't optional. If it's accepted, do read the PEP - while the syntax will _allow_ assignment expressions just about everywhere, the recommended examples are all simple & straightforward. "If it's not obviously better, don't use it" is excellent advice. Because it's allowed almost everywhere, I expect that unanticipated "good uses" will pop up, but at the start a high majority of good uses seem to fall into a small number of patterns. Meta: About the Vasa, I'm not concerned. C++ has around 150 people relentlessly writing an endless sequence of enhancement proposals largely aimed at highly esoteric expert applications of an already extraordinarily complex language. Bjarne Stroustrup is right to be concerned about that. His goal is to cut back on the complications striving for theoretical perfection in all conceivable applications, and go back to working on ideas: that can be used by ?ordinary programmers? whose main concern is to ship > great applications on time http://open-std.org/JTC1/SC22/WG21/docs/papers/2018/p0977r0.pdf If C++ hadn't inherited assignment expressions from C from the start(*), I expect that's an idea he'd _want_ to consider now. They're well within the grasp of "ordinary programmers" - if they can master `j = 3`, they're 90% of the way to mastering `j := 3` (although it may be that "good taste" can't be taught at all). (*) Yes, I know about the stuff added to support yet another form of assignment expression in `if` and `switch` headers in C++ 17. That appeared to be more for "theoretical purity". Assignment expressions were always allowed there, but previously only `for` headers allowed _declaring_ a new variable inside the parens. Now `if` and `switch` allow that too. -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Mon Jul 2 03:38:23 2018 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 2 Jul 2018 09:38:23 +0200 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: On 07/02/18 00:59, Miro Hron?ok wrote: > On 1.7.2018 23:48, Mat?j Cepl wrote: >> On 2018-06-28, 00:58 GMT, Ned Deily wrote: >>> On behalf of the Python development community and the Python 3.7 release >>> team, we are pleased to announce the availability of Python 3.7.0. >> >> I am working on updating openSUSE packages to python 3.7, but >> I have hit quite large number of failing tests (the testsuite >> obviously passed with 3.6), see >> https://build.opensuse.org/package/show/home:mcepl:work/python3 >> (click on the red "failed" label to get logs). I fell into >> a bout of depression, only to discover that we are not alone in >> this problem ... Debian doesn't seem to do much better >> https://is.gd/HKBU4j. Surprisingly, Fedora seems to pass the >> testsuite https://is.gd/E0KA53; interesting, I will have to >> investigate which of their many patches did the trick. > > Note that we (=Fedora) unfortunately skip some tests. > > https://src.fedoraproject.org/rpms/python3/blob/master/f/python3.spec#_1051 > > https://src.fedoraproject.org/rpms/python3/blob/master/f/00160-disable-test_fs_holes-in-rpm-build.patch > > > https://src.fedoraproject.org/rpms/python3/blob/master/f/00163-disable-parts-of-test_socket-in-rpm-build.patch [with my Fedora hat on] Fedora* has been building python37 since the alphas, so the final update to rc/stable was smoother. But it also means we aren't solving the same issues as SUSE now, so we won't be able to help all that much :( Do consider trying out alphas/betas in SUSE next time! Anyway, the SUSE tests seem to fail on .pyc files. The main change in that area was [PEP 552], try starting there. AFAIK, SUSE is ahead of Fedora in the reproducible builds area; perhaps that's where the difference is. And while I'm responding here, a bit of reflection and a heads-up: What Fedora as a distro should do better next time is re-build the entire ecosystem with a new Python version. For 3.7 we started doing that too late, and there are way too many projects that weren't prepared for `async` as keyword and PEP 479 (StopIteration handling). If you run into similar problems in SUSE, you might want to take a look at issues tracked under [Fedora bug 1565020]. [PEP 552]: https://www.python.org/dev/peps/pep-0552/ [Fedora bug 1565020]: https://bugzilla.redhat.com/showdependencytree.cgi?id=1565020 * Thanks to Miro Hron?ok for most of the work in Fedora From solipsis at pitrou.net Mon Jul 2 04:31:28 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Jul 2018 10:31:28 +0200 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) References: Message-ID: <20180702103128.46ba3f96@fsol> Is this message some kind of joke or did you just send it to the wrong mailing-list/recipient? On Sun, 1 Jul 2018 20:21:19 -0700 Matt Arcidy wrote: > This cynical view on students is shocking! Everyone on this list has > been a student or a learner for far longer than an educator, and the > perspective from students and learners are far more important than > educators to assess this angle regardless. Can anyone adequately > explain why this specific modality of learning, a student-in-a-seat > based educator, must outweigh all other modalities learners use to > increase knowledge and skill, from the perspectives of policy, tool > creation, and each of our time spent learning? > > Shortest story: > Teach not to re-use names. > > Short story: > 1) What about the full mosaic of learning vs. this myopic view on > seat-based student-educator interaction? > 2) What about smart, motivated, diligent and cautious students? > 3) What weight should educator opinion be given with respect to > providing convenience to professional Python programmers? > 4) Who is this Student Stupid von Densemeister anyways? > 5) Are assignment expressions convenience and is any danger the pose > unmitagatble? > 6) Consider adding an "Important Topics not Covered" or "Further > Reading" reading section to your class description > 7) Creating examples showing this effect is easy, especially when not > actually re-using the name in the expression for explanatory purposes. > it's the same as creating examples showing how re-use works in > comprehensions. > > > Let's stop constructing these fake Students. They only work as > appeals to the people we have come across whose lack of understanding > has made our life painful. This construction is actively filtering > all the good students for the sake of influencing this decision, yet > again punishing or discounting the intelligent, quick, and diligent. > > And what of this underlying premise that educator's should > _significantly_ influence language development? Limiting Python's > tools to Student Straw-man's ability to learn is just dissonant, they > have nothing to do with each other, nor does this cause-effect > relationship actually exist. Let's evaluate this reductionist > statement: > "I understand X, but this other person is not capable of understanding > X, therefore X should not exist" Is has there ever been an X for > which this is true, let alone the backwardation necessary to fully > close the statement? > > The actual argument is far less reductionist, yet even more ridiculous: > "I understand X, this other person may take time to learn X, and may > use X wrong, therefore X should not exist" > "I understand assignment expressions, but this other class of person > may take time to learn assignment expressions, and may use assignment > expressions wrong, therefore assignment expressions should not be > accepted" > > Rhetorically I disagree with how teaching is being presented, to the > point of near insult (for me lacking a better term). You are saying > these statements about _my_ learning path, (though not personally of > course.) Each of you occupied a role of student at some point, and > each of these statements are being made about your path as well. Do > these ring true of your student experience? What about your much > broader experience as a _learner_? You think a tool shouldn't exist > because it took you time to learn it and you wrote some hard to debug > code, and possibly crashed production, got fired, lost your house and > your pet snake, and crashed the planet into the sun? > > Now I yield, I will accept this position: all/some students cannot > learn this (or it's too complex to teach), but they must learn this > during some class to quickly become effective python developers. How > much weight should this position have in this decision? Let's appeal > to the learner in us. How much of our learner's path, percentage of > total time learning all things python related, has been in a seat > listening to someone else, and that's the only place from which we > gained the knowledge to meet the educator's objective? This time > spent in a class, how does that compare to hours in other learning > modalities? Is this percentage not exactly the weight assigned to > that position? Are people hired from pure class-room based experience > expected to require zero further learning? Are people more valuable > based on classroom hours or work hours? > > As for handling teaching the subject or not, this is easily remedied > with how I do it: "Important Topics not Covered", with resources. > > Anyone here can rightfully claim educator status by having taught > another person something related to this language, which includes > at-work mentoring, informal discussions, posting/replying on SO, > blogging, etc. Are they not being solicited to comment as well? It's > possible to answer this question while vehemently disagreeing with the > PEP. This focus on people who are being ostensibly paid to teach is > myopic. > > Concretely, it's clear to me that parent-local effects can be > dangerously non-obvious when reading and mimicking code without > undertsanding. But when? And how to guard against? How about this: > teach proper (i.e. not) re-using names. The name will still be > ejected to the parent scope, but there won't be any use of it. Teach > the explicit declare pattern first (as everyone does anyways), explain > to not re-use. Regardless of when re-use is done, it is always (or > only) as dangerous as the effect the bound value has anyways, and any > re-use has the potential to trigger the same dangerous behaviors. > > Must we continue this educator assessment of PEP 572? Educators are > not gate-keepers, and the only measure of their success is what > students learn, and students have a far larger and more fine-grained > mosaic now than ever. Seat-based education plays a far smaller role > in anyone's learning path than ever before, and even while in the seat > they have access to Google. All this yield to tiling the outcome in > the favor of the educator by assigning the goal meeting to them, not > to the diligence of the student to learn. How unfair to the student. > > I consider this position purposefully ignoring motivated self-learners > of high ability and skill, or just plain old diligent programmers who > learned to read specs before using tools. > > I don't like posting to python-dev because it's not really my realm, > but this topic is insanely tilted against PEP572 for the most > ridiculous of reasons. I am Pro-572 , so I have decided to join > critique of the educator position. I would rather do it on > python-Ideas, and I want more types of educators solicited, as well as > students and learners. And yes, lets assess your specific lesson > plans if you will make claims it's not possible. Do you even teach > the difference between assignments and expressions at all? > > To have this raised in the this stage of this PEP, and on the dev > list, illustrates how long it took educators to understand the tool to > begin with, as opposed to those who understood even if they disagree. > To have a room full of seat-based educators provide feedback on a tool > they have had 30m to understand as critical to shaping the language is > not defensible in these lower courts. This angle cannot withstand > rigorous scrutiny because each premise is false and it rests the > majority of students being dense. They aren't, and the vast majority > of learners don't need class-room education anyways, so what is this > weight being placed on the opinion of these educators? I agree it's > important to hear it, but dimishingly. > > This is not to say that PEP572 should be accepted otherwise. However, > this educator angle, raised only now and depending on this > platonically dumb student and a non-creative approach to education, is > just pure straw-man to distract from the point at hand. And while I > have repeated "straw-man" as the critique, I dislike leaning on such > a debate-team crutch, and of course employing straw-man doesn't mean > the point is invalid. it just means the rhetoric is bad. However, as > the underlying premise is a severe minority opinion yet claiming to be > broad, and the absolute percentage of solicited opinions is 0% (to the > 17th place), I don't see any importance to the position of educators > right now, especially since these educators in the thread are > complaining about an increase in their personal work, for which it > appears they were compensated (this is pretty bad straw-man, sorry). > > And to repeat, what about the learners? Time spent in seat based > education is so insanely small now. The fact that the name is ejected > to the parent stop is not terribly difficult, and appropriately > factored examples showing the the effect of scope ejection will be > easy to construct, even if trivial for explanation purposes. So what, > exactly, is the issue with learning it? > > > > On Sun, Jul 1, 2018 at 4:35 PM Chris Barker via Python-Dev > wrote: > > > > On Sun, Jul 1, 2018 at 8:35 AM, Michael Selik wrote: > >> > >> As Mark and Chris said (quoting Mark below), this is just one straw in the struggle against piling too many things on the haystack. Unlike some changes to the language, this change of such general use that it won't be an optional topic. Once widely used, it ain't optional. > > > > > > Exactly -- and I also emphasis that this would complicate the language in a broad way -- a much bigger deal than adding a self contained new expression or even the nonlocal keyword. > > > > But to be clear about my take -- it will make the language that little bit harder to teach, but it will also add a bit of complexity that will effect us non-newbies as well. > > > > Is it worth it? maybe. I know I like a better way to express the loop-and-a-half concept in particular. > > > > -CHB > > > > > > > > -- > > > > Christopher Barker, Ph.D. > > Oceanographer > > > > Emergency Response Division > > NOAA/NOS/OR&R (206) 526-6959 voice > > 7600 Sand Point Way NE (206) 526-6329 fax > > Seattle, WA 98115 (206) 526-6317 main reception > > > > Chris.Barker at noaa.gov > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/marcidy%40gmail.com From solipsis at pitrou.net Mon Jul 2 04:37:02 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 2 Jul 2018 10:37:02 +0200 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) References: <20180702002542.GX14437@ando.pearwood.info> Message-ID: <20180702103702.30144b17@fsol> On Mon, 2 Jul 2018 10:25:42 +1000 Steven D'Aprano wrote: > > How do people who teach other languages deal with this? > > Assignment expressions are hardly a new-fangled innovation of Python's. > They're used in Java, Javascript, Ruby, Julia, R, PHP and of course > pretty much the entire C family (C, C++, C# at least). Those other languages don't have two different assignment operators, AFAIK. That's the main point of complication PEP 572 introduces, not the fact that assignment can now be used in an expression. > Admittedly R has the advantage that they don't have to teach a distinct > assignment syntax and explain *why* it ought to be distinct. But > countering that, they have *four* different ways of doing assignment. I don't think R is easy to understand. It depends on the demographics. For a software engineer like me, it's pretty hard to wrap my head around R's nonsense. I think R is only easy if you accept to use it in a "tinker aimlessly until my code works" manner. Regards Antoine. From mike at selik.org Mon Jul 2 05:15:04 2018 From: mike at selik.org (Michael Selik) Date: Mon, 2 Jul 2018 02:15:04 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: On Sun, Jul 1, 2018 at 11:36 PM Tim Peters wrote: > [Michael Selik] > > My worry is that assignment expressions will add about 15 to 20 > > minutes to my class and a slight discomfort. > > So not intractable - which is my high-order bit ;-) > > For those who want more bits of precision (perhaps Guido), while > quantification is good, it needs context to provide insight. Like, out of > how many class hours total? > Generally between 20 and 40 hours. Is 15-20 minutes a little, a lot, par for the course ... compared to other > topics? > I guessed 15-20 minutes, because I'm mentally comparing it to things like ternary expressions. Odds and ends that make the code better, but not a major concept that deserves hours. Will it require you to drop other topics? > Yes. It might not seem like much, but every minute counts. I'd probably try to ignore := unless some pesky student brings it up. It's like someone saying, "Hey, I heard that Python can't do threads?!" I always say, "Good question," but internally I'm thinking, "there goes a half hour. What can I cut today?" > Would you _save_ twice as much class time if we got rid of "is"? ;-) > Ha. You joke, but ``is`` takes about 5 minutes. About 5 or 10 minutes more if some clever student notices that ``1 is 1`` and I need to explain Singletons and interpreter optimizations versus language spec. If it's accepted, do read the PEP > I've read it a few times now. I hope I didn't sound like I haven't read it. That'd be embarrassing. Meta: About the Vasa, I'm not concerned. > Matt Arcidy brought up an interesting point, which I'll quote here: "... I don't see any importance to the position of educators right now, especially since these educators in the thread are complaining about an increase in their personal work, for which it appears they were compensated." >From my brief observations, it seems that the nattering nabobs of negativism, such as myself, are mostly educators. I recently started to wonder if I'd care so much about the language if I didn't teach. I suspect that if I didn't worry about teaching new features, Python 4 could be announced tomorrow and I wouldn't really mind. I suppose it is selfish. But I hope that you [Tim], Guido, and the so many others who have poured energy into this project will appreciate that it's not the current users, but the next billion (?!) Pythonistas that will really keep the language going. Maintaining popularity among educators is a big part of that. -------------- next part -------------- An HTML attachment was scrubbed... URL: From mike at selik.org Mon Jul 2 05:34:44 2018 From: mike at selik.org (Michael Selik) Date: Mon, 2 Jul 2018 02:34:44 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: On Sun, Jul 1, 2018 at 8:21 PM Matt Arcidy wrote: > [...] Can anyone adequately explain why this specific modality of > learning, a student-in-a-seat based educator, must outweigh all other > modalities [...]? 1. It doesn't. 2. It's a proxy for the other modes. I hope this was an adequate explanation. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Mon Jul 2 08:13:01 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 2 Jul 2018 14:13:01 +0200 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: 2018-07-02 9:38 GMT+02:00 Petr Viktorin : > On 07/02/18 00:59, Miro Hron?ok wrote: >> Note that we (=Fedora) unfortunately skip some tests. >> (...) > > [with my Fedora hat on] > > Fedora* has been building python37 since the alphas, so the final update to > rc/stable was smoother. > (...) > * Thanks to Miro Hron?ok for most of the work in Fedora (I'm now working in the Python Maintenance team with Petr Viktorin and Miro Hron?ok.) This work is super useful to prepare third party modules for Python 3.7, but also to spot regressions and bugs in Python 3.7. Fedora also helped to detect issues with the latest glibc for example. Examples: konlpy fails on 3.7 because calling Py_Initialize() twice fails with a fatal error https://bugs.python.org/issue33932 Setting LANG=C modifies the --version behavior https://bugs.python.org/issue33824 test_crypt segfaults when using libxcrypt instead of libcrypt https://bugs.python.org/issue32635 NIS module fails to build due to the removal of interfaces related to Sun RPC from glibc. https://bugs.python.org/issue32521 Sadly, one issue has been detected too late and has not been fixed before 3.7.0 final: https://bugs.python.org/issue34008 Victor From mcepl at cepl.eu Mon Jul 2 08:26:09 2018 From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Mon, 02 Jul 2018 14:26:09 +0200 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: <1530534369.18374.2.camel@cepl.eu> On Mon, 2018-07-02 at 14:13 +0200, Victor Stinner wrote: > 2018-07-02 9:38 GMT+02:00 Petr Viktorin : > > Fedora* has been building python37 since the alphas, so the > > final update to > > rc/stable was smoother. > > (...) > > * Thanks to Miro Hron?ok for most of the work in Fedora > > This work is super useful to prepare third party modules for > Python 3.7, but also to spot regressions and bugs in Python > 3.7. Fedora also helped to detect issues with the latest glibc > for example. I hope to do this as well with openSUSE, but I am just getting into the position, and it is still a lot of struggle. Best, Mat?j -- https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 Be pitiful, for every man is fighting a hard battle. -- Ian MacLaren, at least 1898 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: From vstinner at redhat.com Mon Jul 2 09:15:33 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 2 Jul 2018 15:15:33 +0200 Subject: [Python-Dev] Failing tests [Was: Re: Python 3.7.0 is now available! (and so is 3.6.6)] In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: Hi, 2018-07-01 23:48 GMT+02:00 Mat?j Cepl : > I am working on updating openSUSE packages to python 3.7, but > I have hit quite large number of failing tests (the testsuite > obviously passed with 3.6), see > https://build.opensuse.org/package/show/home:mcepl:work/python3 > (click on the red "failed" label to get logs). I created https://bugs.python.org/issue34022 > I fell into > a bout of depression, only to discover that we are not alone in > this problem ... Debian doesn't seem to do much better > https://is.gd/HKBU4j. It seems like it's the same failures. I'm running the Python test suite every day on Fedora and all tests pass. We also run tests multiple times per day on Travis CI (Ubuntu) and buildbots (Debian, Gentoo, RHEL, SLES). Wait, there is a SLES buildbot and all tests pass except of test_gdb.test_threads(). SLES is OpenSUSE no? The Debian buildbot is green on all branches. Ned Deily: > Without doing a detailed analysis of how your build system is set up, I do note you have "--enable-shared" set on ./configure which is often the source of problems. We have 2 buildbot builders testing this configuration, on FreeBSD and Ubuntu. I just ran the full test suite on my Fedora using --enable-shared: all tests pass. --- $ ./configure --enable-shared $ make $ LD_LIBRARY_PATH=$PWD ./python -m test -r -j0 397 tests OK. Total duration: 3 min 9 sec Tests result: SUCCESS --- I also tried after installing Python: all tests pass as well. --- ./configure --enable-shared --prefix=/opt/py37 && make install LD_LIBRARY_PATH=/opt/py37/lib /opt/py37/bin/python3 -m test -r -j0 --- Victor From tjreedy at udel.edu Mon Jul 2 10:14:23 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 2 Jul 2018 10:14:23 -0400 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: On 7/2/2018 3:38 AM, Petr Viktorin wrote: > And while I'm responding here, a bit of reflection and a heads-up: > What Fedora as a distro should do better next time is re-build the > entire ecosystem with a new Python version. For 3.7 we started doing > that too late, and there are way too many projects that weren't prepared > for `async` as keyword and PEP 479 (StopIteration handling). With the 'next version' now being branched off of master at beta 1 rather than at the first release candidate, 'next time' starts about 4 months sooner than it used to. Doing full builds for 3.8 can start any time. It has already had about 5 months of exclusive patches. This suggests that known changes from pending to deprecation to exception should be done soon after the version branching. I don't know if there are any such still needed for 3.8. -- Terry Jan Reedy From vstinner at redhat.com Mon Jul 2 11:22:18 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 2 Jul 2018 17:22:18 +0200 Subject: [Python-Dev] Failing tests [Was: Re: Python 3.7.0 is now available! (and so is 3.6.6)] In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: > I created https://bugs.python.org/issue34022 So I ran test Python test suite of the master branch on Fedora using OpenSUSE configure command: all tests pass. I also run the 6 failing tests of the master branch on Debian Sid using the same configure command than the Debian builder: the 6 tests pass. I also ran the full test suite on Debian Stretch using "./configure": all tests pass. It seems like the package builders have something special, or patches, or the bugs are specific to the 3.7 branch? Victor From marcidy at gmail.com Mon Jul 2 08:59:15 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Mon, 2 Jul 2018 05:59:15 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: On Mon, Jul 2, 2018 at 2:34 AM Michael Selik wrote: > > On Sun, Jul 1, 2018 at 8:21 PM Matt Arcidy wrote: >> >> [...] Can anyone adequately explain why this specific modality of learning, a student-in-a-seat based educator, must outweigh all other modalities [...]? > > > 1. It doesn't. > 2. It's a proxy for the other modes. > > I hope this was an adequate explanation. Absolutely, thank you. We agree it doesn't out weigh other methods. Clearly I disagree about the proxying. From steve at pearwood.info Mon Jul 2 11:34:12 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 3 Jul 2018 01:34:12 +1000 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> Message-ID: <20180702153411.GB14437@ando.pearwood.info> On Wed, Jun 27, 2018 at 07:29:52PM -0500, Tim Peters wrote: [...] > For example, if the name is declared "global" in the outer scope, you'll > get a compile-time error if you try to declare it "nonlocal" in the > contained scope. "parentlocal" adjusts its meaning accordingly, becoming a > synonym for "global" in that specific case. "Parentlocal" is only a thing if we buy into the paradigm that inside comprehensions is a separate "local". And *that* is only true under two circumstances: - if you are utterly immersed in the implementation of comprehensions as invisible, implicit functions; - or if you start from the premise that comprehensions ought to encapsulate not just the loop variable, but anything else as well. But experimenting with locals() inside comprehensions shows that comprehension-scope *isn't* a well-defined thing. It already bleeds out of the comprehension, and so would some (but only some!) assignment expressions. Instead, if we start from the premise that comprehensions (like any other expression) run in the current scope, then there is no need to invent a term "parentlocal". There's just the usual LEGB scopes, plus class (which people usually forget). With no sublocal scopes (a term we never even had prior to this PEP) assignments inside the comprehension are no more special than assignments inside any other expression. They bind in the current scope, same as always, and keep the sensible identity that these two expressions are exactly equivalent in their visible semantics: [x:=0, x:=1, x:=2] [x:=i for i in (0, 1, 2)] including assignments. What about the loop variable? They ARE special, which is completely justified by the Zen: Although practicality beats purity. We can take a series of ever-more-detailed explanations, starting from the highest "bird's eye" view and gradually dropping further into the murky details of the implementation when, and if, required: - assignment within comprehensions is no different from assignment in any other expression, it occurs in the local scope; - loop variables? they're a special case, for good reason, and are encapsulated inside the comprehension; - how? they're hidden in an implicit, invisible scope, same as .0 the implicit, invisible iterator object; - oh, you didn't know about the .0 variable? well forget about it, it's an undocumented implementation detail, just like the invisible, implicit function used by comprehensions; - oh, you didn't know about that either? read the source code. Only the first two levels of explanation are part of Python the language. The rest is CPython implementation. -- Steve From steve at pearwood.info Mon Jul 2 11:36:28 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 3 Jul 2018 01:36:28 +1000 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624055646.GL14437@ando.pearwood.info> <20180624145204.GN14437@ando.pearwood.info> Message-ID: <20180702153627.GC14437@ando.pearwood.info> On Wed, Jun 27, 2018 at 03:31:42PM -0700, Guido van Rossum wrote: > I'd also like to keep the rule prohibiting use of the same name as a > comprehension loop control variable and as an inline assignment target; > this rule would also prohibit shenanigans with nested comprehensions (for > any set of nested comprehensions, any name that's a loop control variable > in any of them cannot be an inline assignment target in any of them). This > would also apply to the "outermost iterable". +1 -- Steve From hobsonlane at gmail.com Mon Jul 2 12:10:46 2018 From: hobsonlane at gmail.com (Hobson Lane) Date: Mon, 2 Jul 2018 09:10:46 -0700 Subject: [Python-Dev] We now have C code coverage! In-Reply-To: References: Message-ID: ` # noqa` works with linters --Hobson On Fri, Jun 29, 2018 at 6:25 AM, Brett Cannon wrote: > > > On Thu, Jun 28, 2018, 21:28 Terry Reedy, wrote: > >> On 6/24/2018 5:03 AM, Ammar Askar wrote: >> >> Is it possible, given that we are not paying for those reports, to >> >> customize the 'exclude_lines' definitions? >> > >> > Do you want to exclude python code or C code? >> >> Python code. >> >> > For Python code, coverage.py also has some comments you can >> > put down to exclude lines: >> > http://coverage.readthedocs.io/en/coverage-4.2/excluding.html >> >> Yes, by default, one can use '# pragma: no cover' and if one uses the >> --branch flag, '# pragma: no branch'. For more 'advanced exclusion', >> one can use the following, normally in .coveragerc. >> [report] >> exclude_lines = ... >> "This is useful if you have often-used constructs to exclude that can be >> matched with a regex. You can exclude them all at once without littering >> your code with exclusion pragmas." >> >> For IDLE's test suite, I use a customized .coveragerc. I strongly >> prefer to not abandon that and litter the code with # pragmas. >> >> In order to make sense of the coverage report and have it be truthful, >> one needs to know what options are being used. >> Is the --branch flag set? >> Is .coveragerc or some other configuration file in use? >> If so, what is the content? >> Do we have any control over the use and content of exclusion settings? >> > > Everything is either covered by the Travis or codecov configuration files > which are both checked into the cpython repo. (I'm on vacation or else I > would provide links to the files themselves.) > > > >> -- >> Terry Jan Reedy >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/ >> brett%40python.org >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > hobsonlane%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Jul 2 14:19:53 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 2 Jul 2018 11:19:53 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: <20180702153411.GB14437@ando.pearwood.info> References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: Thank you all. I will accept the PEP as is. I am happy to accept *clarification* updates to the PEP if people care to submit them as PRs to the peps repo (https://github.com/python/peps), and that could even (to some extent) include summaries of discussion we've had, or outright rejected ideas. But even without any of those I think the PEP is very clear so I will not wait very long (maybe a week). On Mon, Jul 2, 2018 at 8:38 AM Steven D'Aprano wrote: > On Wed, Jun 27, 2018 at 07:29:52PM -0500, Tim Peters wrote: > [...] > > For example, if the name is declared "global" in the outer scope, you'll > > get a compile-time error if you try to declare it "nonlocal" in the > > contained scope. "parentlocal" adjusts its meaning accordingly, > becoming a > > synonym for "global" in that specific case. > > "Parentlocal" is only a thing if we buy into the paradigm that inside > comprehensions is a separate "local". And *that* is only true under > two circumstances: > > - if you are utterly immersed in the implementation of comprehensions > as invisible, implicit functions; > > - or if you start from the premise that comprehensions ought to > encapsulate not just the loop variable, but anything else as well. > > > But experimenting with locals() inside comprehensions shows that > comprehension-scope *isn't* a well-defined thing. It already bleeds out > of the comprehension, and so would some (but only some!) assignment > expressions. > > Instead, if we start from the premise that comprehensions (like any > other expression) run in the current scope, then there is no need to > invent a term "parentlocal". There's just the usual LEGB scopes, plus > class (which people usually forget). > > With no sublocal scopes (a term we never even had prior to this PEP) > assignments inside the comprehension are no more special than > assignments inside any other expression. They bind in the current scope, > same as always, and keep the sensible identity that these two > expressions are exactly equivalent in their visible semantics: > > [x:=0, x:=1, x:=2] > > [x:=i for i in (0, 1, 2)] > > including assignments. > > What about the loop variable? > > They ARE special, which is completely justified by the Zen: > > Although practicality beats purity. > > We can take a series of ever-more-detailed explanations, starting from > the highest "bird's eye" view and gradually dropping further into the > murky details of the implementation when, and if, required: > > - assignment within comprehensions is no different from assignment > in any other expression, it occurs in the local scope; > > - loop variables? they're a special case, for good reason, and are > encapsulated inside the comprehension; > > - how? they're hidden in an implicit, invisible scope, same as .0 > the implicit, invisible iterator object; > > - oh, you didn't know about the .0 variable? well forget about it, > it's an undocumented implementation detail, just like the invisible, > implicit function used by comprehensions; > > - oh, you didn't know about that either? read the source code. > > > Only the first two levels of explanation are part of Python the > language. The rest is CPython implementation. > > > > -- > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Mon Jul 2 15:27:29 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 2 Jul 2018 14:27:29 -0500 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: Message-ID: [Michael Selik] >>> My worry is that assignment expressions will add about 15 to 20 >>> minutes to my class and a slight discomfort. [Tim] >> So not intractable - which is my high-order bit ;-) >> >> For those who want more bits of precision (perhaps Guido), while >> quantification is good, it needs context to provide insight. Like, >> out of how many class hours total? [Michael] > Generally between 20 and 40 hours. Ah - so I take it you're not teaching raw computer beginners, but people who already know how to program in some other language(s)? If so, perhaps you could leverage on that assignment expressions are already present in the most heavily used languages. Depends on the students' backgrounds, of course. >> Is 15-20 minutes a little, a lot, par for the course ... compared to other topics? > I guessed 15-20 minutes, because I'm mentally comparing it to things > like ternary expressions. Odds and ends that make the code better, > but not a major concept that deserves hours. That's where I see it fitting too. While I don't expect it will ever come up, if I were tasked with teaching assignment expressions, I can picture ways of doing it that would take anywhere from one minute to two hours, depending on the target audience's backgrounds, interests, and needs ("two hours" for those fascinated by computer language history). >> Will it require you to drop other topics? > Yes. It might not seem like much, Nope! It was a trick question. If you had answered "no", I would have known you were just making things up ;-) > but every minute counts. I'd probably try to ignore := unless some > pesky student brings it up. It's like someone saying, "Hey, I heard > that Python can't do threads?!" I always say, "Good question," but > internally I'm thinking, "there goes a half hour. What can I cut today?" Absolutely. 40 hours can't possibly cover more than a significant overview of high-order bits. >> Would you _save_ twice as much class time if we got rid of "is"? ;-) > Ha. You joke, but ``is`` takes about 5 minutes. About 5 or 10 minutes > more if some clever student notices that ``1 is 1`` and I need to explain > Singletons and interpreter optimizations versus language spec. That surprised me! Educators have often said "is" was hard to teach, and it's one of the F'est of Python FAQs on StackOverflow. I always figured that's because it's trivial if you have deep understanding of Python's conceptual object model, but appears to be a random Boolean generator if you're just mucking around at a shell without that understanding. Still, something like this sometimes temporarily baffles even experts: >>> [] is [] # OK, `[]` always creates a new list False >>> id([]) == id([]) # or does it??? True >> If it's accepted, do read the PEP > I've read it a few times now. I hope I didn't sound like I haven't > read it. That'd be embarrassing. Heh. No, I just wanted to be sure. It's just impossible to tell from a discussion that's entirely "meta". > ... > From my brief observations, it seems that the nattering nabobs of negativism, > such as myself, are mostly educators. In this specific thread, sure, but I expect that's because it has "educator feedback" in the Subject. There's been plenty of opposition from non-educators in other threads. I like this thread because it's been more civil than most :-) > I recently started to wonder if I'd care so much about the language if I > didn't teach. I suspect that if I didn't worry about teaching new features, > Python 4 could be announced tomorrow and I wouldn't really mind. Sure - and I wouldn't care so much (or, indeed, at all) if Python wasn't my language of choice for most projects for over 20 years now. > I suppose it is selfish. But I hope that you [Tim], Guido, and the so > many others who have poured energy into this project will appreciate > that it's not the current users, but the next billion (?!) Pythonistas that > will really keep the language going. Which, perhaps paradoxically, is why I'm generally in favor of even small (but general) improvements. regardless of short-term costs: so the next billion Pythonistas can benefit. > Maintaining popularity among educators is a big part of that. Oh, I have no idea about that. I'm at a loss for what accounts for longer-term language popularity, so I push for things that appeal to me as a programmer with broad and deep experiences. Everyone who cares about the language _should_ be heard, but their motivations don't even need to overlap. When I started college, I took classes in every language for which there was a class: assembler, FORTRAN, LISP, and SNOBOL4. Three of those survive as niche languages now, and the last is long dead (damned shame, too! SNOBOL4 was brilliantly creative). The last time I had significant contact with academia, Pascal was all the rage (yes, that dates me). Not only was it universally loved by educators, it was in exactly the right place at exactly the right time: the first compilers for PCs were for Pascal variants, and things like Turbo Pascal were hugely (relative to the much smaller user base at the time) popular and influential. Try to get a job teaching Pascal now ;-) So while I don't pretend to know what accounts for longer-term popularity, "evolve or die" captures a necessary condition. Any living language that doesn't continue evolving will die out. Fighting that is like yelling at clouds. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Mon Jul 2 17:46:56 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 2 Jul 2018 22:46:56 +0100 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: On 02/07/2018 19:19, Guido van Rossum wrote: > Thank you all. I will accept the PEP as is. I am happy to accept > *clarification* updates to the PEP if people care to submit them as > PRs to the peps repo (https://github.com/python/peps), and that could > even (to some extent) include summaries of discussion we've had, or > outright rejected ideas. But even without any of those I think the PEP > is very clear so I will not wait very long (maybe a week). > It's late to raise this, but what exactly are the objections to the syntax ??? ??? expr -> name? # or variations such as? expr => name instead of ??? ??? name := expr The PEP mentions that this syntax does not have a problem that "as" does, but does not list any downsides of it. It conforms to left-to-right evaluation, where name:=expr does not. It (I would argue) reduces the asymmetry of the first use of a sub-expression in cases such as ??? [ ( (f(x) -> y)**2, y**3, y**4) for x in iterable ] vs ??? [ ( (y := f(x))**2, y**3, y**4) for x in iterable ] because the first "y" is closer to the way it is used, viz "**2". Regards Rob Cliffe From chris.barker at noaa.gov Tue Jul 3 01:44:36 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Tue, 3 Jul 2018 05:44:36 +0000 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: <20180702153411.GB14437@ando.pearwood.info> References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: > On Jul 2, 2018, at 8:34 AM, Steven D'Aprano wrote: Guido has decided ? and despite my concerns, I?m going to enjoy my new loop-and-a half construct:-) But a comment on this: > comprehension are no more special than > assignments inside any other expression. They bind in the current scope, > same as always, and keep the sensible identity that these two > expressions are exactly equivalent in their visible semantics: > > [x:=0, x:=1, x:=2] > > [x:=i for i in (0, 1, 2)] > > including assignments. Sure ? and I don?t think that?s confusing. However, generator expressions ( why don?t we call them generator comprehensions?) are a different story, as they may be run at some arbitrary time in the future. This hasn?t been an issue (except for the loop variable, which has been addressed) because: 1) Much of the time, the gen_ex is run right away, in-line. 2) There aren?t many ways to manipulate the local namespace in a gen_ex. With assignment expressions, it will be much easier to manipulate the local namespace, so there is room for some real confusion here. So a real local namespace gen_exp (and comprehensions, for consistency) would be nice. However, that ship has pretty much sailed. Will it end up being a common problem? Probably not, because (a) is still the case, and := will be used infrequently, and hopefully with unlikely to clash names. And as for all the other languages that have assignment expressions? Do they have constructs like generator expressions? -CHB From tim.peters at gmail.com Tue Jul 3 01:47:33 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 3 Jul 2018 00:47:33 -0500 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: [Rob Cliffe] > It's late to raise this, By months, yes ;-) > but what exactly are the objections to the syntax > > expr -> name # or variations such as expr => name > > instead of > > name := expr > > > > The PEP mentions that this syntax does not have a problem that "as" > > does, but does not list any downsides of it. My guess: it probably strikes too many as "excessive novelty", These are assignment expressions. Python's assignment statements put the target at the left. Why change that? ":=" is used for assignment in many more other languages than "->" is. Why fight that? > It conforms to left-to-right evaluation, where name:=expr does not. ? Only "expr" is evaluated, so left-to-right seems irrelevant here. The "evaluation" of a simple name as a binding target is a no-op (no code is generated). If you really do see this as a wart anyway, then it's positively a Good Thing that it's exactly the same "wart" as in Python's assignment statements. > It (I would argue) reduces the asymmetry of the first use of a > sub-expression in cases such as > > [ ( (f(x) -> y)**2, y**3, y**4) for x in iterable ] > > vs > > [ ( (y := f(x))**2, y**3, y**4) for x in iterable ] > > because the first "y" is closer to the way it is used, viz "**2". The first form reads a little better to me too, but not a lot better. The problem I have with variations of this example on its own (which comes up surprisingly often with minor changes) is that it's clearer spelled today via [(y**2, y**3, y**4) for y in map(f, iterable)] Spelling that with either form of assignment expression reads significantly worse than that to my eyes But more importantly, it's expected that assignment expressions will be used _most_ often to make some common `if` and `while` patterns briefer. Hardly all. Our eyes are already trained to "look at the far right end" for the value being tested, and, e.g., while data := sock.recv(): preserves that. Especially in code that doesn't _always_ use assignment expressions in such contexts (which is likely all significant blobs of code), it would be visually jarring to have to "sometimes look in the middle instead" to extract the important part of: while sock.recv() -> data: "Look to the left for the name, look to the right for the value" is the rule for assignment statements, assignment expressions, and `for` loop targets. But there's no "QED" here because this isn't a deductive science. The final answer is "because that's what Guido liked best" ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue Jul 3 02:42:50 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 3 Jul 2018 01:42:50 -0500 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: [Chris Barker] > > However, generator expressions ( why don?t we call them generator > > comprehensions?) Because nobody really liked the "iterator comprehensions" or "accumulator displays" they were variously called at the start. https://mail.python.org/pipermail/python-dev/2003-October/039186.html As that explains, "generator expressions" was an attempt to break away from that "comprehensions" was always a dubious term, carried over from set theory where the term focuses on the optional "if" part rather than the more fundamental iterator or computation parts. At the start, for some (forgotten by me) reason it seemed important to make a distinction between "things like this" that were evaluated at once (list, dict, and set comprehensions) and the new-fangled accumulator displays that got evaluated lazily. But the "generator" in "generator comprehensions" would really be enough all by itself to make that clear enough. So if we had it to do over again I'd sigh and accept "generator comprehensions" anyway. It's been an eternal PITA - and especially in the PEP 572 threads! - to keep typing "comprehensions or generator expressions". Then again, if I had the power of Guido's time machine, I'd go back more, and not use "comprehensions" for anything to begin with. Instead we'd have list, dict, set, and generator twizzlers, affectionately called listwiz, dictwiz, setwiz, and gentwiz by the cool kids :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Tue Jul 3 09:10:19 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 3 Jul 2018 14:10:19 +0100 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: On 03/07/2018 06:47, Tim Peters wrote: > [Rob Cliffe] > > It's late to raise this, > > By months, yes ;-) > > > but what exactly are the objections to the syntax > >? ? ? ? ?expr -> name? # or variations such as? expr => name > > instead of > >? ? ? ? ?name := expr > > > > > The PEP mentions that this syntax does not have a problem that "as" > > does, but does not list any downsides of it. > > My guess:? it probably strikes too many as "excessive novelty",? > ?These are assignment expressions.? Python's assignment statements put > the target at the left.? Why change that?? ":=" is used for assignment > in many more other languages than "->" is.? Why fight that? > > > It conforms to left-to-right evaluation, where name:=expr does not. > > ?? Only "expr" is evaluated, so left-to-right seems irrelevant here.? > The "evaluation" of a simple name as a binding target is a no-op (no > code is generated).? If you really do see this as a wart anyway, then > it's positively a Good Thing that it's exactly the same "wart" as in > Python's assignment statements. > > > It (I would argue) reduces the asymmetry of the first use of a > > sub-expression in cases such as > >? ? ?[ ( (f(x) -> y)**2, y**3, y**4) for x in iterable ] > >? vs > >? ? ?[ ( (y := f(x))**2, y**3, y**4) for x in iterable ] > > because the first "y" is closer to the way it is used, viz "**2". > > The first form reads a little better to me too, but not a lot better.? > The problem I have with variations of this example on its own (which > comes up surprisingly often with minor changes) is that it's clearer > spelled today via > > ? ? [(y**2, y**3, y**4) for y in map(f, iterable)] > > Spelling that with either form of assignment expression reads > significantly worse than that to my eyes > > But more importantly, it's expected that assignment expressions will > be used _most_ often to make some common `if` and `while` patterns > briefer.? Hardly all. Our eyes are already trained to "look at the far > right end" for the value being tested, and, e.g., > > ? ? while data := sock.recv(): > > preserves that.? Especially in code that doesn't _always_ use > assignment expressions in such contexts (which is likely all > significant blobs of code), it would be visually jarring to have to > "sometimes look in the middle instead" to extract the important part of: > > ? ? while sockrecv() -> data: > > "Look to the left for the name, look to the right for the value" is > the rule for assignment statements, assignment expressions, and `for` > loop targets. > > But there's no "QED" here because this isn't a deductive science.? The > final answer is "because that's what Guido liked best" ;-) Thanks, Tim, for a thoughtful answer. Don't get me wrong, I feel quite happy with ":=".? Perhaps you have managed to articulate some thoughts that were buried in my subconscious.? It's just that I couldn't come up with any rational objections to "->". Rob Cliffe > > > > Virus-free. www.avg.com > > > > <#DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Tue Jul 3 11:10:37 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 3 Jul 2018 17:10:37 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 Message-ID: <5B3B91ED.4060309@UGent.be> Hello all, in order to make reviewing PEP 576/580 easier and possibly take some ideas from one PEP to the other, let me state the one fundamental difference between these PEPs. There are many details in both PEPs that can still change, so I'm focusing on what I think is the big structural difference. To be clear: I'm referring to the PEP 576 version at https://github.com/markshannon/pep-576/blob/master/README.rst (this really should be merged in the main PEP repo). Both PEPs add a hook for fast calling of C functions. However, they do that on a different level. Let's trace what _PyObject_FastCallKeywords() currently does when acting on an instance of builtin_function_or_method: A. _PyObject_FastCallKeywords() calls B. _PyCFunction_FastCallKeywords() which calls C. _PyMethodDef_RawFastCallKeywords() which calls D. the actual C function (*ml_meth)() PEP 576 hooks the call A->B while PEP 580 hooks the call B->D (getting rid of C). Advantages of the high-level hook (PEP 576): * Much simpler protocol than PEP 580. * More general since B can be anything. * Not being forced to deal with "self". * Slightly faster when you don't care about B. Advantages of the low-level hook (PEP 580): * No need to duplicate the code from B (see the various existing _{FOO}_FastCallKeywords functions). * Enables certain optimizations because other code can make assumptions about what B does. In my personal opinion, the last advantage of PEP 580 is really important: some existing optimizations depend on it and it also allows extending the protocol in a "performance-compatible" way: it's easy to extend the protocol in a way that callers can benefit from it. Anyway, it would be good to have some guidance on how to proceed here. I would really like something like PEP 580 to be accepted and I'm willing to put time and effort into achieving that. Thanks, Jeroen. From chris.barker at noaa.gov Tue Jul 3 16:10:15 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 3 Jul 2018 13:10:15 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: On Mon, Jul 2, 2018 at 11:42 PM, Tim Peters wrote: > "comprehensions" was always a dubious term, carried over from set theory > where the term focuses on the optional "if" part rather than the more > fundamental iterator or computation parts. > I always wondered about that :-) -- I'd say for most of us that aren't familiar with set theory, it's kind of a "sounds something like putting thing together" word and I just left it at that, and learned what they are. > So if we had it to do over again I'd sigh and accept "generator > comprehensions" anyway. It's been an eternal PITA - and especially in the > PEP 572 threads! - to keep typing "comprehensions or generator > expressions". > Well, too late to change the official name, but not too late to start using the term in threads like these -- and other documentation, etc.... I find there is a lot of confusion about the word "generator", as it implies a "thing that generates values on the fly" (like, say the range() object. But then, in Python, a generator is something that gets crated by a generator function, and CAN be an "thing (iterator) that generates things on the fly", but can also be a more generic coroutine, and can be used in nifty ways that really have nothing to do with generating a bunch of value. (like pytest fixtures, for example) So we have generators, iterators, and iterables, and generators can be iterators, but aren't always, and any number of iterators can generate values on the fly, and .... so it's all a bit of a mess to explain to a newbie. Then again, if I had the power of Guido's time machine, I'd go back more, > and not use "comprehensions" for anything to begin with. Instead we'd have > list, dict, set, and generator twizzlers, affectionately called listwiz, > dictwiz, setwiz, and gentwiz by the cool kids :-) > I'd like that! -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Jul 3 17:37:10 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 4 Jul 2018 00:37:10 +0300 Subject: [Python-Dev] Examples for PEP 572 Message-ID: I like programming languages in which all are expressions (including function declarations, branching and loops) and you can use an assignment at any point, but Python is built on other ways, and I like Python too. PEP 572 looks violating several Python design principles. Python looks simple language, and this is its strong side. I believe most Python users are not professional programmers -- they are sysadmins, scientists, hobbyists and kids -- but Python is suitable for them because its clear syntax and encouraging good style of programming. In particularly mutating and non-mutating operations are separated. The assignment expression breaks this. There should be very good reasons for doing this. But it looks to me that all examples for PEP 572 can be written better without using the walrus operator. > results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0] results = [(x, y, x/y) for x in input_data for y in [f(x)] if y > 0] > stuff = [[y := f(x), x/y] for x in range(5)] stuff = [[y, x/y] for x in range(5) for y in [f(x)]] This idiom looks unusual for you? But this is a legal Python syntax, and it is not more unusual than the new walrus operator. This idiom is not commonly used because there is very little need of using above examples in real code. And I'm sure that the walrus operator in comprehension will be very rare unless PEP 572 will encourage writing complicated comprehensions. Most users prefer to write an explicit loop. I want to remember that PEP 572 started from the discussion on Python-ideas which proposed a syntax for writing the following code as a comprehension: smooth_signal = [] average = initial_value for xt in signal: average = (1-decay)*average + decay*xt smooth_signal.append(average) Using the "for in []" idiom this can be written (if you prefer comprehensions) as: smooth_signal = [average for average in [initial_value] for x in signal for average in [(1-decay)*average + decay*x]] Try now to write this using PEP 572. The walrus operator turned to be less suitable for solving the original problem because it doesn't help to initialize the initial value. Examples from PEP 572: > # Loop-and-a-half > while (command := input("> ")) != "quit": > print("You entered:", command) The straightforward way: while True: command = input("> ") if command == "quit": break print("You entered:", command) The clever way: for command in iter(lambda: input("> "), "quit"): print("You entered:", command) > # Capturing regular expression match objects > # See, for instance, Lib/pydoc.py, which uses a multiline spelling > # of this effect > if match := re.search(pat, text): > print("Found:", match.group(0)) > # The same syntax chains nicely into 'elif' statements, unlike the > # equivalent using assignment statements. > elif match := re.search(otherpat, text): > print("Alternate found:", match.group(0)) > elif match := re.search(third, text): > print("Fallback found:", match.group(0)) It may be more efficient to use a single regular expression which consists of multiple or-ed patterns marked as different groups. For example see the cute regex-based tokenizer in gettext.py: > _token_pattern = re.compile(r""" > (?P[ \t]+) | # spaces and horizontal tabs > (?P[0-9]+\b) | # decimal integer > (?Pn\b) | # only n is allowed > (?P[()]) | > (?P[-*/%+?:]|[>, > # <=, >=, ==, !=, &&, ||, > # ? : > # unary and bitwise ops > # not allowed > (?P\w+|.) # invalid token > """, re.VERBOSE|re.DOTALL) > > def _tokenize(plural): > for mo in re.finditer(_token_pattern, plural): > kind = mo.lastgroup > if kind == 'WHITESPACES': > continue > value = mo.group(kind) > if kind == 'INVALID': > raise ValueError('invalid token in plural form: %s' % value) > yield value > yield '' I have not found any code similar to the PEP 572 example in pydoc.py. It has different code: > pattern = re.compile(r'\b((http|ftp)://\S+[\w/]|' > r'RFC[- ]?(\d+)|' > r'PEP[- ]?(\d+)|' > r'(self\.)?(\w+))') ... > start, end = match.span() > results.append(escape(text[here:start])) > > all, scheme, rfc, pep, selfdot, name = match.groups() > if scheme: > url = escape(all).replace('"', '"') > results.append('%s' % (url, url)) > elif rfc: > url = 'http://www.rfc-editor.org/rfc/rfc%d.txt' % int(rfc) > results.append('%s' % (url, escape(all))) > elif pep: ... It doesn't look as a sequence of re.search() calls. It is more clear and efficient, and using the assignment expression will not make it better. > # Reading socket data until an empty string is returned > while data := sock.recv(): > print("Received data:", data) for data in iter(sock.recv, b''): print("Received data:", data) > if pid := os.fork(): > # Parent code > else: > # Child code pid = os.fork() if pid: # Parent code else: # Child code It looks to me that there is no use case for PEP 572. It just makes Python worse. From rosuav at gmail.com Tue Jul 3 17:51:26 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jul 2018 07:51:26 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka wrote: > I believe most Python users are not > professional programmers -- they are sysadmins, scientists, hobbyists and > kids -- [citation needed] > In particularly mutating and > non-mutating operations are separated. The assignment expression breaks > this. [citation needed] In terms of blending mutating and non-mutating operations, augmented assignment is far worse. Contrast: >>> x = 1 >>> y = x >>> x += 1 >>> a = [1] >>> b = a >>> a += [2] Assignment expressions do the exact same thing as assignment statements, but also allow you to keep using that value. There is nothing about mutation. (Unless you believe that assignment *itself* is mutation, in which case Python is definitely the wrong language for you.) ChrisA From tjreedy at udel.edu Tue Jul 3 18:04:35 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 3 Jul 2018 18:04:35 -0400 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: On 7/3/2018 2:42 AM, Tim Peters wrote: > So if we had it to do over again I'd sigh and accept "generator > comprehensions" anyway.? It's been an eternal PITA - and especially in > the PEP 572 threads! - to keep typing "comprehensions or generator > expressions".? Then again, if I had the power of Guido's time machine, > I'd go back more, and not use "comprehensions" for anything to begin > with. Amen. I cannot make 'comprehension' in this context comprehensible without some linguistic twisting. >? Instead we'd have list, dict, set, and generator twizzlers, > affectionately called listwiz, dictwiz, setwiz, and gentwiz by the cool > kids :-) I learned the set notion, such as {n^2: n in N; 1 <= n < 100, n even} # math {n*n for n in range(1,100) if not n%2} # python as 'set builder' notation. If we had followed the math precedent, instead of , we would have set builders, list builders, dict builders, and generator builders. I half seriously think we should consider this for 3.8 for the benefit of future Python programmers as well as ourselves. Comprehensions that can contain assignment expressions are a slightly new thing. -- Terry Jan Reedy From guido at python.org Tue Jul 3 18:08:27 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 3 Jul 2018 15:08:27 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: I smell a new thread. :-) On Tue, Jul 3, 2018 at 3:06 PM Terry Reedy wrote: > On 7/3/2018 2:42 AM, Tim Peters wrote: > > > So if we had it to do over again I'd sigh and accept "generator > > comprehensions" anyway. It's been an eternal PITA - and especially in > > the PEP 572 threads! - to keep typing "comprehensions or generator > > expressions". Then again, if I had the power of Guido's time machine, > > I'd go back more, and not use "comprehensions" for anything to begin > > with. > > Amen. I cannot make 'comprehension' in this context comprehensible > without some linguistic twisting. > > > Instead we'd have list, dict, set, and generator twizzlers, > > affectionately called listwiz, dictwiz, setwiz, and gentwiz by the cool > > kids :-) > > I learned the set notion, such as > {n^2: n in N; 1 <= n < 100, n even} # math > {n*n for n in range(1,100) if not n%2} # python > as 'set builder' notation. > > If we had followed the math precedent, instead of language>, we would have set builders, list builders, dict builders, and > generator builders. > > I half seriously think we should consider this for 3.8 for the benefit > of future Python programmers as well as ourselves. Comprehensions that > can contain assignment expressions are a slightly new thing. > > > -- > Terry Jan Reedy > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Jul 3 18:24:09 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 3 Jul 2018 15:24:09 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Tue, Jul 3, 2018 at 2:51 PM, Chris Angelico wrote: > On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka > wrote: > > I believe most Python users are not > > professional programmers -- they are sysadmins, scientists, hobbyists and > > kids -- > > [citation needed] fair enough, but I think we all agree that *many*, if not most, Python users are "not professional programmers". While on the other hand everyone involved in discussion on python-dev and python-ideas is a serious (If not "professional") programmer. So we do have a bit of a disconnect between much of the user base and folks making decisions about how the language evolves -- which is probably inevitable, Over the years I've been using it (most of its life), Python has evolved to become much less of a "scripting" language, and much more of a "systems" language, and this addition is a (pretty significant) step more in that direction. Serhiy: FWIW, a number of us made the case about the additional complexity of this feature in this discussion -- I think it was a bit side-tracked by the impression that we were only talking about newbies (and also by scope of comprehensions), but the point was made and I assume considered. > > In particularly mutating and > > non-mutating operations are separated. The assignment expression breaks > > this. > I'd call it "local namespace manipulating and non-local namespace manipulating", but yes, that is my concern as well. In terms of blending mutating and non-mutating operations, augmented > assignment is far worse. yeah -- I've always thought it was a shame that augmented assignment worked on immutables :-( -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Jul 3 19:13:20 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 04 Jul 2018 11:13:20 +1200 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: <5B3C0310.2010102@canterbury.ac.nz> Terry Reedy wrote: > If we had followed the math precedent, instead of language>, we would have set builders, list builders, dict builders, and > generator builders. I was intending to suggest something like that back when comprehensions were first being discussed, but people raced ahead and adopted the term "comprehension" before I got the chance. "List builder" and "dict builder" make a lot of sense, but "generator builder" not so much -- it *is* a generator, not something that builds a generator. In fact it doesn't build anything in the sense that the others do. So maybe "generator expression" is the best we could have done. -- Greg From vstinner at redhat.com Tue Jul 3 19:25:28 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 4 Jul 2018 01:25:28 +0200 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: 2018-07-02 20:19 GMT+02:00 Guido van Rossum : > Thank you all. I will accept the PEP as is. (...) I see more and more articles ("on the Internet") saying that Guido van Rossum already accepted the PEP. Is the PEP already accepted or will be accepted? Right now, https://www.python.org/dev/peps/pep-0572/ status is "Draft". Victor From guido at python.org Tue Jul 3 19:46:23 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 3 Jul 2018 16:46:23 -0700 Subject: [Python-Dev] Informal educator feedback on PEP 572 (was Re: 2018 Python Language Summit coverage, last part) In-Reply-To: References: <20180624145204.GN14437@ando.pearwood.info> <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> Message-ID: On Tue, Jul 3, 2018 at 4:25 PM Victor Stinner wrote: > 2018-07-02 20:19 GMT+02:00 Guido van Rossum : > > Thank you all. I will accept the PEP as is. (...) > > I see more and more articles ("on the Internet") saying that Guido van > Rossum already accepted the PEP. Is the PEP already accepted or will > be accepted? > > Right now, https://www.python.org/dev/peps/pep-0572/ status is "Draft". > That's a rather philosophical question. I clearly said "I will" not "I might". And if you're asking whether it's likely that I'll change my mind, no. I would like help with updates to the PEP to summarize some of the discussions and rejected proposals. And I am giving myself a week to "cool off". But I am muting threads that bring up objections that I've heard before (e.g. "design principles"). So those articles aren't wrong. Your patience is appreciated. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Jul 3 20:10:27 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 Jul 2018 10:10:27 +1000 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: <5B3C0310.2010102@canterbury.ac.nz> References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> Message-ID: <20180704001027.GI14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 11:13:20AM +1200, Greg Ewing wrote: > Terry Reedy wrote: > >If we had followed the math precedent, instead of >language>, we would have set builders, list builders, dict builders, and > >generator builders. > > I was intending to suggest something like that back when > comprehensions were first being discussed, but people > raced ahead and adopted the term "comprehension" before > I got the chance. > > "List builder" and "dict builder" make a lot of sense, > but "generator builder" not so much -- it *is* a generator, > not something that builds a generator. In fact it doesn't > build anything in the sense that the others do. So maybe > "generator expression" is the best we could have done. But [expr for x in seq] is a list, just as (expr for ...) is a generator. If you don't believe me, try it: py> type([x for x in (1,)]) py> type(x for x in (1,)) So I think the similarity is complete. Further, if we think of "list builder" as an abbreviation of "list builder syntax", we have: - list builder syntax is syntax which returns a list; - dict builder syntax is syntax which returns a dict; - set builder syntax is syntax which returns a set; - generator builder syntax is syntax which returns a generator. Of course, there are other ways to build lists, such as calling the constructor, or using a list display ("list literal", except it isn't always a literal). But they're not *builder syntax* :-) In hindsight, I think "spam builder (syntax)" would have been better than the rather mysterious technical word "comprehension" and the not very felicitous term "generator expression". -- Steve From songofacandy at gmail.com Tue Jul 3 20:24:30 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Wed, 4 Jul 2018 09:24:30 +0900 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: ? > > In particularly mutating and > > non-mutating operations are separated. The assignment expression breaks > > this. > > [citation needed] > > In terms of blending mutating and non-mutating operations, augmented > assignment is far worse. Contrast: > > >>> x = 1 > >>> y = x > >>> x += 1 > > >>> a = [1] > >>> b = a > >>> a += [2] > > > Assignment expressions do the exact same thing as assignment > statements, but also allow you to keep using that value. There is > nothing about mutation. (Unless you believe that assignment *itself* > is mutation, in which case Python is definitely the wrong language for > you.) > > ?I think Serhiy use "mutation" as "assignment", or "changing variable".? ?And at this point, I'm with Serhiy.? Before PEP 572, assignment is happened on very limited places. When we want to use "variable x", we can check "x isn't changed from value I want" very quickly, without reading full code. For example, with open(some_file) as f: for line in f: line = line.rstrip() # some code here self.some_method(..., # some long arguments (line := line.split())[0], line[1], # oops! ...) # some code here x = {k: f for k in line if (f := k.upper()).startswith('F')} # oops! # some code here Before PEP 572, we can check "f is not changed from `as f`" and "line is not changed from `line = line.rstrip()`" very quickly, without reading expressions in argument list or comprehension. After PEP 572, we need to read all code between place we want to use some variable and these variables are assigned to expected value. In this meaning, ?augmented assignment is far better than assignment expression. It's very easy to find, same to "as x" or "x =". ?So PEP 572 will reduce maintainability of code written ?by ? others ? (1) (1) "others" including "I" in several months ago. Linters helps us sometime, but linter can't help us when others who written the code? didn't use linter and it's difficult to solve every warning from linters. ?This is what I feel how PEP 572 is different from f-string or ternary expression. f-string and ternary expression can do only what expressions can. But PEP 572 expands "what expressions can". I feel PEP 572 breaks border between expression and statement, and it makes readability of dirty code worse. On the other hand, I understand PEP 572 allows clever code simplifies tedious code. It may increase readability of non-dirty code. Regards, -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue Jul 3 21:26:06 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 3 Jul 2018 20:26:06 -0500 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: [INADA Naoki] > ... > On the other hand, I understand PEP 572 allows clever code > simplifies tedious code. It may increase readability of non-dirty code. The latter is the entire intent ,of course. We can't force people to write readable code, but I don't understand the widespread assumption that other programmers are our enemies who have to be preemptively disarmed ;-) Use code review to enforce readable code. If you want a coding standard here, use mine: "if using an assignment expression isn't obviously better (at least a little so), DON'T USE IT". That's the same standard I use for lots of things (e.g., is such-&-such better as a listcomp or as nested loops?). It only requires that you have excellent taste in what "better" means ;-) As I noted in the PEP's Appendix A, I refuse to even write code like i = j = count = nerrors = 0 because it squashes conceptually distinct things into a single statement . I'll always write that as i = j = 0 count = 0 nerrors = 0 instead - or even in 4 lines if `i` and `j` aren't conceptually related. That's how annoyingly pedantic I can be ;-) Yet after staring at lots of code, starting from a neutral position (why have an opinion about anything before examination?), I became a True Believer. I really don't know what Guido likes best about this, but for me it's the large number of objectively small wins in `if` and `while` contexts. They add up. That conclusion surprised me. That there are occasionally bigger wins to be had is pure gravy. But in no case did I count "allows greater cleverness" as a win. The Appendix contains a few examples of "bad" uses too, where cleverness in pursuit of brevity harms clarity. In fact, to this day, I believe those examples derived from abusing assignment expressions in real-life code are more horrifying than any of the examples anyone else _contrived_ to "prove" how bad the feature is. I apparently have more faith that people will use the feature as intended. Not all people, just most. The ones who don't can be beaten into compliance, same as with any other abused feature ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Tue Jul 3 21:31:46 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Wed, 4 Jul 2018 10:31:46 +0900 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3B91ED.4060309@UGent.be> References: <5B3B91ED.4060309@UGent.be> Message-ID: I think both PEPs are relying on FASTCALL calling convention, and can't be accepted until FASTCALL is stable & public. There are enough time before Python 3.8 is released. Let's go step by step. Regards, On Wed, Jul 4, 2018 at 12:10 AM Jeroen Demeyer wrote: > Hello all, > > in order to make reviewing PEP 576/580 easier and possibly take some > ideas from one PEP to the other, let me state the one fundamental > difference between these PEPs. There are many details in both PEPs that > can still change, so I'm focusing on what I think is the big structural > difference. > > To be clear: I'm referring to the PEP 576 version at > https://github.com/markshannon/pep-576/blob/master/README.rst > (this really should be merged in the main PEP repo). > > Both PEPs add a hook for fast calling of C functions. However, they do > that on a different level. Let's trace what _PyObject_FastCallKeywords() > currently does when acting on an instance of builtin_function_or_method: > > A. _PyObject_FastCallKeywords() > calls > B. _PyCFunction_FastCallKeywords() > which calls > C. _PyMethodDef_RawFastCallKeywords() > which calls > D. the actual C function (*ml_meth)() > > PEP 576 hooks the call A->B while PEP 580 hooks the call B->D (getting > rid of C). > > Advantages of the high-level hook (PEP 576): > * Much simpler protocol than PEP 580. > * More general since B can be anything. > * Not being forced to deal with "self". > * Slightly faster when you don't care about B. > > Advantages of the low-level hook (PEP 580): > * No need to duplicate the code from B (see the various existing > _{FOO}_FastCallKeywords functions). > * Enables certain optimizations because other code can make assumptions > about what B does. > > In my personal opinion, the last advantage of PEP 580 is really > important: some existing optimizations depend on it and it also allows > extending the protocol in a "performance-compatible" way: it's easy to > extend the protocol in a way that callers can benefit from it. > > Anyway, it would be good to have some guidance on how to proceed here. I > would really like something like PEP 580 to be accepted and I'm willing > to put time and effort into achieving that. > > > Thanks, > Jeroen. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com > -- INADA Naoki -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Tue Jul 3 21:54:22 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 3 Jul 2018 21:54:22 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On 7/3/2018 5:37 PM, Serhiy Storchaka wrote: > I like programming languages in which all are expressions (including > function declarations, branching and loops) and you can use an > assignment at any point, but Python is built on other ways, and I like > Python too. PEP 572 looks violating several Python design principles. > Python looks simple language, and this is its strong side. I believe > most Python users are not professional programmers -- they are > sysadmins, scientists, hobbyists and kids -- but Python is suitable for > them because its clear syntax and encouraging good style of programming. > In particularly mutating and non-mutating operations are separated. The > assignment expression breaks this. There should be very good reasons for > doing this. But it looks to me that all examples for PEP 572 can be > written better without using the walrus operator. I appreciate you showing alternatives I can use now. Even once implemented, one can not use A.E's until one no longer cares about 3.7 compatibility. Then there will still be a choice. >> results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0] > > ??? results = [(x, y, x/y) for x in input_data for y in [f(x)] if y > 0] Would (f(x),) be faster? import timeit as ti print(ti.timeit('for y in {x}: pass', 'x=1')) print(ti.timeit('for y in [x]: pass', 'x=1')) print(ti.timeit('for y in (x,): pass', 'x=1')) # prints 0.13765254499999996 # seconds per 1_000_000 = microseconds each. 0.10321274000000003 0.09492473300000004 Yes, but not enough to pay for adding ',', and sometimes forgetting. >> stuff = [[y := f(x), x/y] for x in range(5)] > stuff = [[y, x/y] for x in range(5) for y in [f(x)]] Creating an leaky name binding appears to about 5 x faster than iterating a temporary singleton. print(ti.timeit('y=x', 'x=1')) print(ti.timeit('y=x; del y', 'x=1')) # 0.017357778999999907 0.021115051000000107 If one adds 'del y' to make the two equivalent, the chars typed is about the same. To me, the choice amounts to subject reference. Even with y:=x available, I would write the expansion as res = [] for x in range(5): y = f(x) res.append((y, x/y)) rather than use the assignment expression in the tuple. It creates a 'hitch' in thought. > This idiom looks unusual for you? But this is a legal Python syntax, and > it is not more unusual than the new walrus operator. This idiom is not > commonly used because there is very little need of using above examples > in real code. And I'm sure that the walrus operator in comprehension > will be very rare unless PEP 572 will encourage writing complicated > comprehensions. Most users prefer to write an explicit loop. > I want to remember that PEP 572 started from the discussion on > Python-ideas which proposed a syntax for writing the following code as a > comprehension: > > ??? smooth_signal = [] > ??? average = initial_value > ??? for xt in signal: > ??????? average = (1-decay)*average + decay*xt > ??????? smooth_signal.append(average) > > Using the "for in []" idiom this can be written (if you prefer > comprehensions) as: > > ??? smooth_signal = [average > ???????????????????? for average in [initial_value] > ???????????????????? for x in signal > ???????????????????? for average in [(1-decay)*average + decay*x]] > > Try now to write this using PEP 572. The walrus operator turned to be > less suitable for solving the original problem because it doesn't help > to initialize the initial value. > > > Examples from PEP 572: > >> # Loop-and-a-half >> while (command := input("> ")) != "quit": >> ??? print("You entered:", command) > > The straightforward way: > > ??? while True: > ??????? command = input("> ") > ??????? if command == "quit": break > ??????? print("You entered:", command) > > The clever way: > > ??? for command in iter(lambda: input("> "), "quit"): > ??????? print("You entered:", command) The 2-argument form of iter is under-remembered and under-used. The length difference is 8. while (command := input("> ")) != "quit": for command in iter(lambda: input("> "), "quit"): I like the iter version, but the for-loop machinery and extra function call makes a minimal loop half a millisecond slower. import timeit as ti def s(): it = iter(10000*'0' + '1') def w(): it = iter(10000*'0' + '1') while True: command = next(it) if command == '1': break def f(): it = iter(10000*'0' + '1') for command in iter(lambda: next(it), '1'): pass print(ti.timeit('s()', 'from __main__ import s', number=1000)) print(ti.timeit('w()', 'from __main__ import w', number=1000)) print(ti.timeit('f()', 'from __main__ import f', number=1000)) # 0.0009702129999999975 0.9365254250000001 1.5913117949999998 Of course, with added processing of 'command' the time difference disappears. Printing (in IDLE) is an extreme case. def wp(): it = iter(100*'0' + '1') while True: command = next(it) if command == '1': break print('w', command) def fp(): it = iter(100*'0' + '1') for command in iter(lambda: next(it), '1'): print('f', command) print(ti.timeit('wp()', 'from __main__ import wp', number=1)) print(ti.timeit('fp()', 'from __main__ import fp', number=1)) # 0.48 0.47 >> # Capturing regular expression match objects >> # See, for instance, Lib/pydoc.py, which uses a multiline spelling >> # of this effect >> if match := re.search(pat, text): >> ??? print("Found:", match.group(0)) >> # The same syntax chains nicely into 'elif' statements, unlike the >> # equivalent using assignment statements. >> elif match := re.search(otherpat, text): >> ??? print("Alternate found:", match.group(0)) >> elif match := re.search(third, text): >> ??? print("Fallback found:", match.group(0)) > It may be more efficient to use a single regular expression which > consists of multiple or-ed patterns My attempt resulted in a slowdown. Duplicating the dominance of pat over otherpat over third requires, I believe, negative lookahead assertions. --- import re import timeit as ti ##print(ti.timeit('for y in {x}: pass', 'x=1')) ##print(ti.timeit('for y in [x]: pass', 'x=1')) ##print(ti.timeit('for y in (x,): pass', 'x=1')) ## ##print(ti.timeit('y=x', 'x=1')) ##print(ti.timeit('y=x; del y', 'x=1')) pat1 = re.compile('1') pat2 = re.compile('2') pat3 = re.compile('3') pat123 = re.compile('1|2(?!.*1)|3(?!.*(1|2))') # I think most people would prefer to use the 3 simple patterns. def ifel(text): match = re.search(pat1, text) if match: return match.group() match = re.search(pat2, text) if match: return match.group() match = re.search(pat3, text) if match: return match.group() def mach(text): match = re.search(pat123, text) return match.group() print([ifel('321'), ifel('32x'), ifel('3xx')] == ['1', '2', '3']) print([mach('321'), mach('32x'), mach('3xx')] == ['1', '2', '3']) # True, True text = '0'*10000 + '321' print(ti.timeit('ifel(text)', "from __main__ import ifel, text", number=100000)) print(ti.timeit('mach(text)', "from __main__ import mach, text", number=100000)) # 0.77, 7.22 > marked as different groups. When I put parens around 1, 2, 3 in pat123, the 2nd timeit continued until I restarted Shell. Maybe you can do better. > For example see the cute regex-based tokenizer in gettext.py: > >> _token_pattern = re.compile(r""" >> ??????? (?P[ \t]+)??????????????????? | # spaces and >> horizontal tabs >> ??????? (?P[0-9]+\b)?????????????????????? | # decimal integer >> ??????? (?Pn\b)????????????????????????????? | # only n is allowed >> ??????? (?P[()])????????????????????? | >> ??????? (?P[-*/%+?:]|[>> -, <, >, >> ???????????????????????????????????????????????????? # <=, >=, ==, !=, >> &&, ||, >> ???????????????????????????????????????????????????? # ? : >> ???????????????????????????????????????????????????? # unary and >> bitwise ops >> ???????????????????????????????????????????????????? # not allowed >> ??????? (?P\w+|.)?????????????????????????? # invalid token >> ??? """, re.VERBOSE|re.DOTALL) >> >> def _tokenize(plural): >> ??? for mo in re.finditer(_token_pattern, plural): >> ??????? kind = mo.lastgroup >> ??????? if kind == 'WHITESPACES': >> ??????????? continue >> ??????? value = mo.group(kind) >> ??????? if kind == 'INVALID': >> ??????????? raise ValueError('invalid token in plural form: %s' % value) >> ??????? yield value >> ??? yield '' > I have not found any code similar to the PEP 572 example in pydoc.py. It > has different code: > >> pattern = re.compile(r'\b((http|ftp)://\S+[\w/]|' >> ??????????????????????? r'RFC[- ]?(\d+)|' >> ??????????????????????? r'PEP[- ]?(\d+)|' >> ??????????????????????? r'(self\.)?(\w+))') > ... >> start, end = match.span() >> results.append(escape(text[here:start])) >> >> all, scheme, rfc, pep, selfdot, name = match.groups() >> if scheme: >> ??? url = escape(all).replace('"', '"') >> ??? results.append('%s' % (url, url)) >> elif rfc: >> ??? url = 'http://www.rfc-editor.org/rfc/rfc%d.txt' % int(rfc) >> ??? results.append('%s' % (url, escape(all))) >> elif pep: > ... > > It doesn't look as a sequence of re.search() calls. It is more clear and > efficient, and using the assignment expression will not make it better. > >> # Reading socket data until an empty string is returned >> while data := sock.recv(): >> ??? print("Received data:", data) > > ??? for data in iter(sock.recv, b''): > ??????? print("Received data:", data) > >> if pid := os.fork(): >> ??? # Parent code >> else: >> ??? # Child code > > ??? pid = os.fork() > ??? if pid: > ??????? # Parent code > ??? else: > ??????? # Child code > > > It looks to me that there is no use case for PEP 572. It just makes > Python worse. > -- Terry Jan Reedy From steve at pearwood.info Tue Jul 3 22:42:06 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 Jul 2018 12:42:06 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <20180704024206.GL14437@ando.pearwood.info> On Tue, Jul 03, 2018 at 09:54:22PM -0400, Terry Reedy wrote: > > ??? results = [(x, y, x/y) for x in input_data for y in [f(x)] if y > 0] > > Would (f(x),) be faster? There is a deferred feature request to optimize "for x in [item]" as equivalent to "for x in (item,)", to avoid constructing a list: https://bugs.python.org/issue32856 Since it only affects the internal byte-code, not visible semantics (aside from speed and memory) I think that's a neat micro-optimization regardless of whether it is applied to comprehensions or regular for-loops or both. -- Steve From storchaka at gmail.com Wed Jul 4 02:07:46 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 4 Jul 2018 09:07:46 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: 04.07.18 00:51, Chris Angelico ????: > On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka wrote: >> I believe most Python users are not >> professional programmers -- they are sysadmins, scientists, hobbyists and >> kids -- > > [citation needed] I don't understand what citation do you need. >> In particularly mutating and >> non-mutating operations are separated. The assignment expression breaks >> this. > > [citation needed] In Python the assignment (including the augmented assignment) is a statement, del is a statement, function and class declarations are statements, import is a statement. Mutating methods like list.sort() and dict.update() return None to discourage using them in expressions. This a common knowledge, I don't know who's citation you need. From storchaka at gmail.com Wed Jul 4 02:11:16 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 4 Jul 2018 09:11:16 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: 04.07.18 03:24, INADA Naoki ????: > I feel PEP 572 breaks border between expression and statement, and it makes > readability of dirty code worse. Thank you, this is what I meant. > On the other hand, I understand PEP 572 allows clever code simplifies > tedious > code.? It may increase readability of non-dirty code. Unfortunately, I have not seen any example of this yet. From storchaka at gmail.com Wed Jul 4 02:14:57 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 4 Jul 2018 09:14:57 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: 04.07.18 04:26, Tim Peters ????: > I really don't know what Guido likes best about this, but for me it's > the large number of objectively small wins in `if` and `while` > contexts.? ?They add up.? That conclusion surprised me.? That there are > occasionally bigger wins to be had is pure gravy. Could you please show me several examples in real code? I have not seen any win yet. From greg.ewing at canterbury.ac.nz Wed Jul 4 02:54:55 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 04 Jul 2018 18:54:55 +1200 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: <20180704001027.GI14437@ando.pearwood.info> References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> Message-ID: <5B3C6F3F.8050400@canterbury.ac.nz> Steven D'Aprano wrote: > - list builder syntax is syntax which returns a list; > > - dict builder syntax is syntax which returns a dict; > > - set builder syntax is syntax which returns a set; > > - generator builder syntax is syntax which returns a generator. You only get a list/dict/set from the first three after you've run the iterators within it, but with a generator expression, you already have a generator before you've run it. That makes it feel different to me. -- Greg From rosuav at gmail.com Wed Jul 4 03:02:07 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jul 2018 17:02:07 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 4:07 PM, Serhiy Storchaka wrote: > 04.07.18 00:51, Chris Angelico ????: >> >> On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka >> wrote: >>> >>> I believe most Python users are not >>> professional programmers -- they are sysadmins, scientists, hobbyists and >>> kids -- >> >> >> [citation needed] > > > I don't understand what citation do you need. Anything at all that suggests that "most Python users are not professional programmers". Anything beyond just a hunch of yours. >>> In particularly mutating and >>> non-mutating operations are separated. The assignment expression breaks >>> this. >> >> >> [citation needed] > > > In Python the assignment (including the augmented assignment) is a > statement, del is a statement, function and class declarations are > statements, import is a statement. Mutating methods like list.sort() and > dict.update() return None to discourage using them in expressions. This a > common knowledge, I don't know who's citation you need. "Assignment is a statement" -- that's exactly the point under discussion. "del is a statement" -- yes, granted "function and class declarations are statements" -- class, yes, but you have "def" and "lambda" as statement and expression equivalents. "import is a statement" -- but importlib.import_module exists for a reason So you have two statements (del and class), one statement with a function form (import), and one statement with an expression form (def/lambda). I'm going to assume that your term "mutating" there was simply a miswording, and that you're actually talking about *name binding*, which hitherto occurs only in statements. Yes, this is true. And it's the exact point under discussion - that restricting name bindings to statements is unnecessary. ChrisA From tim.peters at gmail.com Wed Jul 4 03:06:57 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 4 Jul 2018 02:06:57 -0500 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: [Tim] >> I really don't know what Guido likes best about this, but for me it's > >> the large number of objectively small wins in `if` and `while` > >> contexts. They add up. That conclusion surprised me. That there are > >> occasionally bigger wins to be had is pure gravy. > [Serhiy Storchaka] > Could you please show me several examples in real code? I > have not seen any win yet. My PEP Appendix was derived entirely from looking at real code. If you don't believe the examples I showed there are wins (and I don't know whether you've seen them, because your original message in this thread only echoed examples from the body of the PEP), then what we each mean by "win" in this context has no intersection, so discussing it would be futile (for both of us). Which is what I expect: the treatment you gave to the examples from the body of the PEP suggests you're determined not to acknowledge any "win", however small. Which is fine by me, but if so it's an extreme position. I don't recall anyone else over the months this has been active who claimed the PEP is 100% loss (not in its current form, or in any of its earlier forms). Even those who hate it passionately have argued instead that downsides outweigh benefits - not that there are no benefits whatsoever. So making such a claim: > It looks to me that there is no use case for PEP 572. It just makes > Python worse. comes across more as trolling than sincere inquiry. It's possible you really can't imagine how anyone could see any benefits here. But then, as above,, our worldviews probably differ too much for communication to be possible - there must be going on a thousand messages in these threads by now, and across all these months you still have no idea why anyone would want this? If so, you're more an unmovable object than I am an irresistible force ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jul 4 03:10:11 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 4 Jul 2018 00:10:11 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Tue, Jul 3, 2018 at 11:07 PM, Serhiy Storchaka wrote: > 04.07.18 00:51, Chris Angelico ????: >> >> On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka >> wrote: >>> >>> I believe most Python users are not >>> professional programmers -- they are sysadmins, scientists, hobbyists and >>> kids -- >> >> >> [citation needed] > > > I don't understand what citation do you need. > >>> In particularly mutating and >>> non-mutating operations are separated. The assignment expression breaks >>> this. >> >> >> [citation needed] > > > In Python the assignment (including the augmented assignment) is a > statement, del is a statement, function and class declarations are > statements, import is a statement. Mutating methods like list.sort() and > dict.update() return None to discourage using them in expressions. This a > common knowledge, I don't know who's citation you need. Right, Python has a *very strong* convention that each line should have at most one side-effect, and that if it does have a side-effect it should be at the outermost level. I think the most striking evidence for this is that during the discussion of PEP 572 we discovered that literally none of us ? including Guido ? even *know* what the order-of-evaluation is inside expressions. In fact PEP 572 now has a whole section talking about the oddities that have turned up here so far, and how to fix them. Which just goes to show that even its proponents don't actually think that anyone uses side-effects inside expressions, because if they did, then they'd consider these changes to be compatibility-breaking changes. Of course the whole point of PEP 572 is to encourage people to embed side-effects inside expressions, so I hope they've caught all the weird cases, because even if we can still change them now we won't be able to after PEP 572 is implemented. Some people make fun of Python's expression/statement dichotomy, because hey don't you know that everything can be an expression, functional languages are awesome hurhur, but I think Python's approach is actually very elegant. Python is unapologetically an imperative language, but even we dirty imperative programmers can agree with the functional fanatics that reasoning about side-effects and sequencing is hard. One-side-effect-per-line is a very elegant way to keep sequencing visible on the page and as easy to reason about as possible. Or as Dijkstra put it: "our intellectual powers are rather geared to master static relations and that our powers to visualize processes evolving in time are relatively poorly developed. For that reason we should do (as wise programmers aware of our limitations) our utmost to shorten the conceptual gap between the static program and the dynamic process, to make the correspondence between the program (spread out in text space) and the process (spread out in time) as trivial as possible." It's very disheartening that not only is PEP 572 apparently going to be accepted, but as far as I can tell neither the text nor its proponents have even addressed this basic issue. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Wed Jul 4 03:25:18 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 4 Jul 2018 00:25:18 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Tue, Jul 3, 2018 at 11:14 PM, Serhiy Storchaka wrote: > 04.07.18 04:26, Tim Peters ????: >> >> I really don't know what Guido likes best about this, but for me it's the >> large number of objectively small wins in `if` and `while` contexts. They >> add up. That conclusion surprised me. That there are occasionally bigger >> wins to be had is pure gravy. > > > Could you please show me several examples in real code? I have not seen any > win yet. The new version of the PEP has several real examples -- there are some "sanitized" ones in this section inspired by Guido's trawling through Dropbox code: https://www.python.org/dev/peps/pep-0572/#the-importance-of-real-code And Tim's appendix has several more, most from his code plus one that Kirill Balunov found in the stdlib: https://www.python.org/dev/peps/pep-0572/#appendix-a-tim-peters-s-findings This is much nicer than the old PEP! Unfortunately I don't find any of these examples convincing myself :-(. Most of them (the Dropbox ones and the first few in Tim's appendix) look like great arguments for pattern matching syntax, which is basically an elegant way of writing chained 'if's and capturing values along the way. And at PyCon Guido was saying he hoped we'd also get a good proposal for pattern matching syntax for 3.8, so for these cases := will hopefully be obsolete before it even ships... The last two in Tim's appendix are the "best" examples, in the sense that they really do depend on the full power of :=. Unfortunately (as Tim acknowledges) the code is fairly incomprehensible to non-Tims regardless of which syntax is used, so to me they aren't really any more convincing than foo/bar examples. But, if I found myself forced to maintain this code, my first step in deciphering it would be to rewrite it to remove the :='s, just to simplify the control flow, so that I could focus on understanding each statement in isolation. The PEP spends a ton of time talking about comprehensions, but still has zero real examples of comprehensions that would be improved by :=. The only cases that seem potentially valuable to me are the ones that are literally the form 'if := ` and 'while := '. (I suspect these are the only cases that I would allow in code that I maintain.) The PEP does briefly discuss the alternative proposal of restricting to just these two cases, but rejects it because it would rule out code like 'if ( := ) '. But those are exactly the cases that I want to rule out, so that seems like a plus to me :-). The 'if as ' syntax would be a simple way to encode exactly these simple non-harmful cases. The PEP rejects it on the grounds that 'as' is already used in a different way by 'except' and 'with'. But... 'as' is *also* used in the *same* way by 'import', so the argument feels disengenuous. Yeah, there'd be an inconsistency, but that inconsistency already exists, and adding 'if ... as' and 'while ... as' wouldn't create any *new* inconsistencies. Guido has asked for PRs to the PEP to flesh out the rejected alternatives and replies-to-common-complaints sections, and I'd be happy to write up something for that, except that despite all the electrons that have been spilled I actually do not know what the PEP authors response would be to the issues that bother me :-(. -n -- Nathaniel J. Smith -- https://vorpus.org From storchaka at gmail.com Wed Jul 4 03:51:50 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 4 Jul 2018 10:51:50 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: 04.07.18 04:54, Terry Reedy ????: > Would (f(x),) be faster? No. Both "for y in [f(x)]" and "for y in (f(x),)" are compiled to the same bytecode. Run your microbenchmarks again, the difference is a small random variation. https://bugs.python.org/issue32925 >>> stuff = [[y := f(x), x/y] for x in range(5)] >> ?stuff = [[y, x/y] for x in range(5) for y in [f(x)]] > > Creating an leaky name binding appears to about 5 x faster than > iterating a temporary singleton. With issue32856 be merged, "for var in [expr]" will be compiled to the same bytecode as just "var = expr". This is a simple optimization, and this is a good kind of changes that increase performance for free, without needing users to use a new syntax. It is not merged yet because I have doubts that the need in the assignment inside comprehensions is worth even such small complication of the compiler (of course PEP 572 adds much more complications, and not only to the code generator). If you need to write the above artificial example as a comprehension, lets just merge issue32856. https://bugs.python.org/issue32856 > The 2-argument form of iter is under-remembered and under-used.? The > length difference is 8. > ??? while (command := input("> ")) != "quit": > for command in iter(lambda: input("> "), "quit"): This doesn't look like a good rationale for such large change as PEP 572. > I like the iter version, but the for-loop machinery and extra function > call makes a minimal loop half a millisecond slower. > > import timeit as ti > > def s(): > ??? it = iter(10000*'0' + '1') > > def w(): > ??? it = iter(10000*'0' + '1') > ??? while True: > ??????? command = next(it) > ??????? if command == '1': break > > def f(): > ??? it = iter(10000*'0' + '1') > ??? for command in iter(lambda: next(it), '1'): pass > > print(ti.timeit('s()', 'from __main__ import s', number=1000)) > print(ti.timeit('w()', 'from __main__ import w', number=1000)) > print(ti.timeit('f()', 'from __main__ import f', number=1000)) > # > 0.0009702129999999975 > 0.9365254250000001 > 1.5913117949999998 Using partial() makes it faster: from functools import partial def p(): it = iter(10000*'0' + '1') for command in iter(partial(next, it), '1'): pass print(ti.timeit('s()', 'from __main__ import s', number=1000)) print(ti.timeit('w()', 'from __main__ import w', number=1000)) print(ti.timeit('f()', 'from __main__ import f', number=1000)) print(ti.timeit('p()', 'from __main__ import p', number=1000)) # 0.0016302559961332008 0.7507075049943523 1.3297416319983313 0.6211225209990516 > Of course, with added processing of 'command' the time difference > disappears. Yes, and this is why I didn't bother about a tiny overhead of a lambda. You can use partial() in a tight performance critical loop. It is even faster than a bare while loop. >>> # Capturing regular expression match objects >>> # See, for instance, Lib/pydoc.py, which uses a multiline spelling >>> # of this effect >>> if match := re.search(pat, text): >>> ??? print("Found:", match.group(0)) >>> # The same syntax chains nicely into 'elif' statements, unlike the >>> # equivalent using assignment statements. >>> elif match := re.search(otherpat, text): >>> ??? print("Alternate found:", match.group(0)) >>> elif match := re.search(third, text): >>> ??? print("Fallback found:", match.group(0)) > >> It may be more efficient to use a single regular expression which >> consists of multiple or-ed patterns > > My attempt resulted in a slowdown.? Duplicating the dominance of pat > over otherpat over third requires, I believe, negative lookahead > assertions. I have to admit that *this* example will not get a benefit from rewriting with a single regular expression. I was fooled by the misleading reference to pydoc.py. The code in pydoc.py doesn't have anything common with this example, it searches the first occurrence of the set of patterns in a loop, while the example searches patterns sequentially and only once. The code similar to pydoc.py is common, but I would want to see a real code that corresponds the example in PEP 572. From storchaka at gmail.com Wed Jul 4 03:58:48 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 4 Jul 2018 10:58:48 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704024206.GL14437@ando.pearwood.info> References: <20180704024206.GL14437@ando.pearwood.info> Message-ID: 04.07.18 05:42, Steven D'Aprano ????: > On Tue, Jul 03, 2018 at 09:54:22PM -0400, Terry Reedy wrote: >>> ??? results = [(x, y, x/y) for x in input_data for y in [f(x)] if y > 0] >> >> Would (f(x),) be faster? > > There is a deferred feature request to optimize "for x in [item]" as > equivalent to "for x in (item,)", to avoid constructing a list: > > https://bugs.python.org/issue32856 No, this optimization was already made in issue32925. Issue32856 is a different optimization -- it makes such constructions be as fast as a simple assignment. It allows to use an assignment in comprehensions without introducing a new syntax and with zero overhead. From jeanpierreda at gmail.com Wed Jul 4 04:03:02 2018 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 4 Jul 2018 01:03:02 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 12:26 AM Nathaniel Smith wrote: > The only cases that seem potentially valuable to me are the ones that > are literally the form 'if := ` and 'while := > '. (I suspect these are the only cases that I would allow in > code that I maintain.) The PEP does briefly discuss the alternative > proposal of restricting to just these two cases, but rejects it > because it would rule out code like 'if ( := ) > '. But those are exactly the cases that I want to > rule out, so that seems like a plus to me :-). The PEP doesn't talk about it, but FWIW, Go and C++17 if statements allow you to put arbitrary simple statements in the suite header, and by doing that, solves all the issues you and the PEP mentioned. In Python it'd look like this: if x = expr(); x < 0: do_stuff() (and presumably, the same for while: "while line = f.readline(); line: ...") -- Devin From solipsis at pitrou.net Wed Jul 4 04:13:53 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 4 Jul 2018 10:13:53 +0200 Subject: [Python-Dev] Examples for PEP 572 References: Message-ID: <20180704101353.79b76172@fsol> On Tue, 3 Jul 2018 15:24:09 -0700 Chris Barker via Python-Dev wrote: > > fair enough, but I think we all agree that *many*, if not most, Python > users are "not professional programmers". While on the other hand everyone > involved in discussion on python-dev and python-ideas is a serious (If not > "professional") programmer. > > So we do have a bit of a disconnect between much of the user base and folks > making decisions about how the language evolves -- which is probably > inevitable, > > Over the years I've been using it (most of its life), Python has evolved to > become much less of a "scripting" language, and much more of a "systems" > language, and this addition is a (pretty significant) step more in that > direction. This change doesn't even make Python a better fit as a systems programming language. Rust's distinguishing feature, after all, isn't that it has an assignment operator. Regards Antoine. From storchaka at gmail.com Wed Jul 4 04:54:34 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 4 Jul 2018 11:54:34 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: 04.07.18 10:06, Tim Peters ????: > [Tim] > >> I really don't know what Guido likes best about this, but for me it's > > >> the large number of objectively small wins in `if` and `while` > > >> contexts.? ?They add up.? That conclusion surprised me.? That there are > > >> occasionally bigger wins to be had is pure gravy. > > > [Serhiy Storchaka] > > Could you please show me several examples in real code? I > > have not seen any win yet. > > My PEP Appendix was derived entirely from looking at real code.? If you > don't believe the examples I showed there are wins (and I don't know > whether you've seen them, because your original message in this thread > only echoed examples from the body of the PEP), then what we each mean > by "win" in this context has no intersection, so discussing it would be > futile (for both of us). Sorry, this PEP was rewritten so many times that I missed your Appendix. > while total != (total := total + term): > term *= mx2 / (i*(i+1)) > i += 2 > return total This code looks clever that the original while loop with a break in a middle. I like clever code. But it needs more mental efforts for understanding it. I admit that this is a good example. There is a tiny problem with it (and with rewriting a while loop as a for loop, as I like). Often the body contains not a single break. In this case the large part of cleverness is disappeared. :-( > if result := solution(xs, n): > # use result It looks equally clear with the original code. This is not enough for introducing a new syntax. > if reductor := dispatch_table.get(cls): > rv = reductor(x) > elif reductor := getattr(x, "__reduce_ex__", None): > rv = reductor(4) > elif reductor := getattr(x, "__reduce__", None): > rv = reductor() > else: > raise Error("un(shallow)copyable object of type %s" % cls) I was going to rewrite this code as reductor = dispatch_table.get(cls) if reductor: rv = reductor(x) else: rv = x.__reduce_ex__(4) There were reasons for the current complex code in Python 2, but now classic classes are gone, and every class has the __reduce_ex__ method which by default calls __reduce__ which by default is inherited from object. With that simplification the benefit of using ":=" in this example looks less impressed. > if (diff := x - x_base) and (g := gcd(diff, n)) > 1: > return g > while a > (d := x // a**(n-1)): > a = ((n-1)*a + d) // n > return a I would have a fun writing such code. As well as I believe you had a fun writing it. But I suppose the original code is simpler for a casual reader, and I will refrain from using such code in a project maintained by other people (in particular in the Python stdlib). > Which is what I expect:? the treatment you gave to the examples from the > body of the PEP suggests you're determined not to acknowledge any "win", > however small. I have to admit that *there are* examples that can have a small win. I wondering why your examples are not used in the PEP body instead of examples which play *against* PEP 572. Yet a win too small to me for justifying such syntax change. I know that I can not convince you or Guido. From vstinner at redhat.com Wed Jul 4 05:17:03 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 4 Jul 2018 11:17:03 +0200 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: <20180704024206.GL14437@ando.pearwood.info> Message-ID: 2018-07-04 9:58 GMT+02:00 Serhiy Storchaka :> 04.07.18 05:42, Steven D'Aprano ????: >> There is a deferred feature request to optimize "for x in [item]" as >> equivalent to "for x in (item,)", to avoid constructing a list: >> >> https://bugs.python.org/issue32856 > > > No, this optimization was already made in issue32925. Oh, I missed that optimization: it's even optimized at the AST level, cool! Good job INADA Naoki for the new AST optimizer and Serhiy for this optimization! $ ./python Python 3.8.0a0 (heads/master:97ae32c92e, Jul 4 2018, 09:37:54) >>> import dis; dis.dis('x in [y]') 1 0 LOAD_NAME 0 (x) 2 LOAD_NAME 1 (y) 4 BUILD_TUPLE 1 6 COMPARE_OP 6 (in) 8 RETURN_VALUE Note: "x in [1]" was already optimized as "x in (1,)" since at least Python 2.7 (in the bytecode peephole optimizer, not at the AST level). Victor From tismer at stackless.com Wed Jul 4 05:15:27 2018 From: tismer at stackless.com (Christian Tismer) Date: Wed, 4 Jul 2018 11:15:27 +0200 Subject: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow In-Reply-To: <20180425034330.GP11616@ando.pearwood.info> References: <20180425034330.GP11616@ando.pearwood.info> Message-ID: On 25.04.18 05:43, Steven D'Aprano wrote: > On Tue, Apr 24, 2018 at 08:10:49PM -0500, Tim Peters wrote: > >> Binding expressions are debugger-friendly in that they _don't_ just >> vanish without a trace. It's their purpose to _capture_ the values of >> the expressions they name. Indeed, you may want to add them all over >> the place inside expressions, never intending to use the names, just >> so that you can see otherwise-ephemeral intra-expression results in >> your debugger ;-) > > That's a fantastic point and I'm surprised nobody has thought of it > until now (that I've seen). > > Chris, if you're still reading this and aren't yet heartedly sick and > tired of the PEP *wink* this ought to go in as another motivating point. Yay, that's like a dream, really fantastic. So sorry that I was way too deep in development in spring and did not read earlier about that PEP. I was actually a bit reluctant about "yet another way to prove Python no longer simple" and now even that Pascal-ish look! :-) But this argument has completely sold me. Marvellous! -- Christian Tismer-Sperling :^) tismer at stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://pyside.org 14482 Potsdam : GPG key -> 0xE7301150FB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: OpenPGP digital signature URL: From rosuav at gmail.com Wed Jul 4 06:07:36 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jul 2018 20:07:36 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 7:59 PM, St?fane Fermigier wrote: > > > On Tue, Jul 3, 2018 at 11:52 PM Chris Angelico wrote: >> >> On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka >> wrote: >> > I believe most Python users are not >> > professional programmers -- they are sysadmins, scientists, hobbyists >> > and >> > kids -- >> >> [citation needed] > > > Let's focus on France: > > 1) there are ~800 000 person aged 15. (Source: > https://www.insee.fr/fr/statistiques/1892088?sommaire=1912926 ) > > 2) 40% of these are pupils in the "fili?re g?n?rale" (Source: > https://fr.wikipedia.org/wiki/Baccalaur%C3%A9at_en_France#Statistiques ). > > 3) Since 2017, Python is the language used to teach algorithmics to these > pupils. > > => We can conclude that there are at least 320 000 pupils learning Python > this year, and safely assume that most of them are not "professional > programmers". > > Note also that this number is accretive (i.e.: 320 000 this year, 640 000 > next year, etc.). > > 4) The annual turnover for the IT sector in France (including: software > vendor, service and consulting) was 54 billions euros in 2017. This probably > translates to around or less than 600 000 IT professionals. How many of them > are developers ? I have no idea, but I'm sure it's less that 50%. How many > of them are developing professionally in Python ? I have no idea, I'd guess > less than 10%, but let's assume 20%. > > => This gives less than 60 000 professional Python programmers in France. > Probably much less. > > Conclusion: there are probably more than 5 times more non professional > Python programmers in France (counting only the high-school pupils, not > other categories such as their teachers, but also scientist, sysadmins or > hobbyist) than professional programmers. > > Next year it will be (more than) 1 to 10. The year after that, 1 to 15, etc. Even assuming your figures to be 100% accurate, I don't think you can accept that scaling. Are you claiming that every high school student (a) continues to use Python forever, and (b) continues to use it at a non-professional level? I find that highly unlikely. I'm pretty dubious that these figures will correspond to the rest of the world, where you can't expect that every single high school student is using Python. ChrisA From arj.python at gmail.com Wed Jul 4 07:21:29 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Wed, 4 Jul 2018 15:21:29 +0400 Subject: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow In-Reply-To: References: Message-ID: was going to tell instead of := maybe => better := too close to other langs Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ Of the proposed syntaxes, I dislike identifer := expression less, but > I'd still rather not see it added. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/arj.python%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jul 4 07:35:40 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 Jul 2018 21:35:40 +1000 Subject: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow In-Reply-To: References: Message-ID: <20180704113540.GP14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 03:21:29PM +0400, Abdur-Rahmaan Janhangeer wrote: > was going to tell > > instead of := maybe => better > > := too close to other langs The fact that := will be familiar to many people (especially if they know Go, Pascal or Eiffel etc) is a point in its favour. https://en.wikipedia.org/wiki/Assignment_%28computer_science%29#Notation The => arrow puts the arguments around the "wrong" way compared to regular assignment: name = expression expression => name Although personally I like that order, many people did not and I think Guido ruled it out very early in the discussion. Also it may be too easily confused with <= or >= or the -> syntax from annotations. -- Steve From arj.python at gmail.com Wed Jul 4 07:50:06 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Wed, 4 Jul 2018 15:50:06 +0400 Subject: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow In-Reply-To: <20180704113540.GP14437@ando.pearwood.info> References: <20180704113540.GP14437@ando.pearwood.info> Message-ID: agree for => but how many people use pascal eiffel etc? (go has a chance) that's a reminder of an old, fading epoch, bland IDEs, hard-to-crunch fonts BDL Guido once remarked in a pycon talk that today agencies would've charged you a high price to tell you what the word python might tickle in the subconscious of the common user, we should maybe write on what ":=" tickles in the mind of most programmers Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ Mauritius The fact that := will be familiar to many people (especially if they > know Go, Pascal or Eiffel etc) is a point in its favour. > > https://en.wikipedia.org/wiki/Assignment_%28computer_science%29#Notation > > The => arrow puts the arguments around the "wrong" way compared to > regular assignment: > > name = expression > expression => name > > Although personally I like that order, many people did not and I think > Guido ruled it out very early in the discussion. > > Also it may be too easily confused with <= or >= or the -> syntax from > annotations. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Jul 4 08:00:26 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Jul 2018 22:00:26 +1000 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: On 2 July 2018 at 17:38, Petr Viktorin wrote: > Anyway, the SUSE tests seem to fail on .pyc files. The main change in that > area was [PEP 552], try starting there. AFAIK, SUSE is ahead of Fedora in > the reproducible builds area; perhaps that's where the difference is. In particular, if a build system sets SOURCE_DATE_EPOCH without specifying a pyc format for py_compile or compileall, Python 3.7 will give you checked hashes by default: https://docs.python.org/3/library/py_compile.html?highlight=source_date_epoch#py_compile.compile Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Wed Jul 4 08:05:01 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Jul 2018 22:05:01 +1000 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: On 4 July 2018 at 22:00, Nick Coghlan wrote: > On 2 July 2018 at 17:38, Petr Viktorin wrote: >> Anyway, the SUSE tests seem to fail on .pyc files. The main change in that >> area was [PEP 552], try starting there. AFAIK, SUSE is ahead of Fedora in >> the reproducible builds area; perhaps that's where the difference is. > > In particular, if a build system sets SOURCE_DATE_EPOCH without > specifying a pyc format for py_compile or compileall, Python 3.7 will > give you checked hashes by default: > https://docs.python.org/3/library/py_compile.html?highlight=source_date_epoch#py_compile.compile Running the following locally fails for me: $ SOURCE_DATE_EPOCH=`date` ./python -m test test_py_compile test_compileall So my guess would be that this is a test suite error where we're not handling the "running in a reproducible build environment with SOURCE_DATE_EPOCH already set" case. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vstinner at redhat.com Wed Jul 4 08:29:08 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 4 Jul 2018 14:29:08 +0200 Subject: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow In-Reply-To: References: Message-ID: The PEP 572 has been approved, it's no longer worth it to discuss it ;-) Victor 2018-07-04 13:21 GMT+02:00 Abdur-Rahmaan Janhangeer : > was going to tell > > instead of := maybe => better > > := too close to other langs > > Abdur-Rahmaan Janhangeer > https://github.com/Abdur-rahmaanJ > >> Of the proposed syntaxes, I dislike identifer := expression less, but >> >> I'd still rather not see it added. >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/arj.python%40gmail.com From vstinner at redhat.com Wed Jul 4 08:30:05 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 4 Jul 2018 14:30:05 +0200 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: Yes, see my issue https://bugs.python.org/issue34022 to discuss how to fix tests. Victor 2018-07-04 14:05 GMT+02:00 Nick Coghlan : > On 4 July 2018 at 22:00, Nick Coghlan wrote: >> On 2 July 2018 at 17:38, Petr Viktorin wrote: >>> Anyway, the SUSE tests seem to fail on .pyc files. The main change in that >>> area was [PEP 552], try starting there. AFAIK, SUSE is ahead of Fedora in >>> the reproducible builds area; perhaps that's where the difference is. >> >> In particular, if a build system sets SOURCE_DATE_EPOCH without >> specifying a pyc format for py_compile or compileall, Python 3.7 will >> give you checked hashes by default: >> https://docs.python.org/3/library/py_compile.html?highlight=source_date_epoch#py_compile.compile > > Running the following locally fails for me: > > $ SOURCE_DATE_EPOCH=`date` ./python -m test test_py_compile test_compileall > > So my guess would be that this is a test suite error where we're not > handling the "running in a reproducible build environment with > SOURCE_DATE_EPOCH already set" case. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com From ncoghlan at gmail.com Wed Jul 4 08:34:02 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 4 Jul 2018 22:34:02 +1000 Subject: [Python-Dev] Checking that PEP 558 (defined semantics for locals()) handles assignment expressions Message-ID: With PEP 572's formal acceptance now expected to be just a matter of time, I'll limit any further personal expressions of opinion on the change and the process taken to achieve it to other venues :) However, there's a design aspect I do need to address, which is to make sure that PEP 558 (my proposal to actually nail down an implementation independent specification for how we expect locals() to behave at runtime) accurately covers the way runtimes and debuggers are expected to behave when comprehensions and generator expressions are executed at different scopes. My current view is that it's going to be OK to expose the differing behaviours at module and function scope directly: * if you debug a module level comprehension or genexp, the target name will be flagged on the code object as a dynamically resolved name reference, so a debugger should handle it the same was as it would handle any other "global NAME" declaration (and it will appear only in globals(), not in locals()) * similarly for a function comprehension or genexp, the name will show up like any other "nonlocal NAME" (appears in locals() rather than globals(), and is affected by the proposed changes in the interaction between trace functions and the frame.f_locals attribute) I think that's an acceptable outcome, since it's a natural consequence of PEP 572 defining its comprehension and generator expression handling in terms of existing global and nonlocal scoping semantics. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vano at mail.mipt.ru Wed Jul 4 08:36:18 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 4 Jul 2018 15:36:18 +0300 Subject: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow In-Reply-To: References: Message-ID: <3638eff2-3ae6-d03b-90a1-c1ff750bc231@mail.mipt.ru> On 04.07.2018 15:29, Victor Stinner wrote: > The PEP 572 has been approved, it's no longer worth it to discuss it ;-) > > Victor As of now, https://www.python.org/dev/peps/pep-0572/ is marked as "draft". > 2018-07-04 13:21 GMT+02:00 Abdur-Rahmaan Janhangeer : >> was going to tell >> >> instead of := maybe => better >> >> := too close to other langs >> >> Abdur-Rahmaan Janhangeer >> https://github.com/Abdur-rahmaanJ >> >>> Of the proposed syntaxes, I dislike identifer := expression less, but >>> >>> I'd still rather not see it added. >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe: >>> https://mail.python.org/mailman/options/python-dev/arj.python%40gmail.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From brett at python.org Wed Jul 4 08:43:04 2018 From: brett at python.org (Brett Cannon) Date: Wed, 4 Jul 2018 09:43:04 -0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Tue, Jul 3, 2018, 22:27 Tim Peters, wrote: > [INADA Naoki] > > ... > > On the other hand, I understand PEP 572 allows clever code > > simplifies tedious code. It may increase readability of non-dirty > code. > > The latter is the entire intent ,of course. We can't force people to > write readable code, but I don't understand the widespread assumption that > other programmers are our enemies who have to be preemptively disarmed ;-) > > Use code review to enforce readable code. If you want a coding standard > here, use mine: "if using an assignment expression isn't obviously better > (at least a little so), DON'T USE IT". That's the same standard I use for > lots of things (e.g., is such-&-such better as a listcomp or as nested > loops?). It only requires that you have excellent taste in what "better" > means ;-) > I think this is a very key point that the "this is bad" crowd is overlooking. Even if this syntax turns out to not be that useful, abusing the walrus operator can be fixed with a comment of "hard to follow; please simplify" without teaching any new concepts or idioms (and which happens with constructs most of us would agree are useful and well-designed). IOW when I'm doing code reviews for folks this is not going to be what I stress about or spill the most characters trying to explain how to fix. > As I noted in the PEP's Appendix A, I refuse to even write code like > > i = j = count = nerrors = 0 > > because it squashes conceptually distinct things into a single statement > . I'll always write that as > > i = j = 0 > count = 0 > nerrors = 0 > > instead - or even in 4 lines if `i` and `j` aren't conceptually related. > > That's how annoyingly pedantic I can be ;-) Yet after staring at lots of > code, starting from a neutral position (why have an opinion about anything > before examination?), I became a True Believer. > > I really don't know what Guido likes best about this, but for me it's the > large number of objectively small wins in `if` and `while` contexts. They > add up. That conclusion surprised me. That there are occasionally bigger > wins to be had is pure gravy. > Another point is we live in a dictatorship by choice, and yet some people seem very upset our dictator dictated in the end. ? Rather than being upset that Guido chose not to heed some advice that someone personally thought was crucial, I'm going to rely on the point that all the salient points were made (albeit sometimes in sensationalist terms in what seems like a higher-than-normal frequency), and trust Guido's design taste which I have appreciated for my 18 years as a Python user. -Brett > But in no case did I count "allows greater cleverness" as a win. The > Appendix contains a few examples of "bad" uses too, where cleverness in > pursuit of brevity harms clarity. In fact, to this day, I believe those > examples derived from abusing assignment expressions in real-life code are > more horrifying than any of the examples anyone else _contrived_ to "prove" > how bad the feature is. > > I apparently have more faith that people will use the feature as > intended. Not all people, just most. The ones who don't can be beaten > into compliance, same as with any other abused feature ;-) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jul 4 08:56:55 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 Jul 2018 22:56:55 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <20180704125654.GQ14437@ando.pearwood.info> On Tue, Jul 03, 2018 at 03:24:09PM -0700, Chris Barker via Python-Dev wrote: > Over the years I've been using it (most of its life), Python has evolved to > become much less of a "scripting" language, and much more of a "systems" > language, and this addition is a (pretty significant) step more in that > direction. As I understand it, the most commonly accepted definitions are: Systems language: low-level languages designed to compile down to efficient machine code, suitable for writing operating systems, device drivers, and code for embedded systems. Application language: high-level language intended to insulate the programmer from the gory details of memory management, suitable for writing user-space applications. Scripting language: any interpreted high-level language the speaker doesn't like *wink* https://en.wikipedia.org/wiki/System_programming_language See also Ousterhout's (false) dichotomy: https://en.wikipedia.org/wiki/Ousterhout's_dichotomy Python is certainly not a systems language in the sense of C, Ada or Rust. It could be classified as an application language, like Java. And it still remains an awesome glue language for C and Fortran code. -- Steve From steve at pearwood.info Wed Jul 4 09:31:25 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 Jul 2018 23:31:25 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <20180704133125.GR14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 01:03:02AM -0700, Devin Jeanpierre wrote: > The PEP doesn't talk about it, but FWIW, Go and C++17 if statements > allow you to put arbitrary simple statements in the suite header, and > by doing that, solves all the issues you and the PEP mentioned. Whether they solve "all the issues" discussed in the PEP is doubtful, since they're entirely limited to just if statements. It amuses me that the Go designers -- presumably Rob Pike -- decided on a rule "no assignment expressions", and then immediately broke it "well okay, assignment expressions in if statements". At least Guido waited 20+ years to change his mind :-) > In Python it'd look like this: > > if x = expr(); x < 0: > do_stuff() That's even more doubtful, since the semicolon in Python already has a meaning of separating statements on a line. That could be equivalent to: if x = expr() x < 0: do_stuff() which would clearly have to be a syntax error. *Two* syntax errors. Python's parser is restricted to LL(1) by design, so treating semicolons in "if" statements as a special case might not even be possible. But even if it is possible, not only would this special case break the Zen, but it would be UGLY too. -- Steve From steve at pearwood.info Wed Jul 4 09:35:39 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 4 Jul 2018 23:35:39 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <20180704133539.GS14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 05:02:07PM +1000, Chris Angelico wrote: > On Wed, Jul 4, 2018 at 4:07 PM, Serhiy Storchaka wrote: > "Assignment is a statement" -- that's exactly the point under discussion. Not any more it isn't. We've now gone from discussion to bitter recriminations *wink* > "del is a statement" -- yes, granted > > "function and class declarations are statements" -- class, yes, but > you have "def" and "lambda" as statement and expression equivalents. Even class can be re-written as a call to type(), if you need to. It's probably not practical to do so in anything but the simplest cases, but it is there. -- Steve From mcepl at cepl.eu Wed Jul 4 09:46:31 2018 From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Wed, 04 Jul 2018 15:46:31 +0200 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: <1530711991.8216.3.camel@cepl.eu> On Wed, 2018-07-04 at 22:05 +1000, Nick Coghlan wrote: > Running the following locally fails for me: > > ????$ SOURCE_DATE_EPOCH=`date` ./python -m test test_py_compile > test_compileall Just if this is taken literally, it is wrong. According to https: //reproducible-builds.org/specs/source-date-epoch/ it should be $ SOURCE_DATE_EPOCH=`date +%s` ./python -m test test_py_compile \ test_compileall Mat?j -- https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 And religious texts are a bit like software standards, the interpretation is always the tricky and complicated bit. -- Alan Cox -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 836 bytes Desc: This is a digitally signed message part URL: From mertz at gnosis.cx Wed Jul 4 09:52:40 2018 From: mertz at gnosis.cx (David Mertz) Date: Wed, 4 Jul 2018 09:52:40 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 3:02 AM Chris Angelico wrote: > "Assignment is a statement" -- that's exactly the point under discussion. > "del is a statement" -- yes, granted > "function and class declarations are statements" -- class, yes, but > you have "def" and "lambda" as statement and expression equivalents. > "import is a statement" -- but importlib.import_module exists for a reason I'm going to assume that your term "mutating" there was simply a > miswording, and that you're actually talking about *name binding*, > which hitherto occurs only in statements. Yes, this is true. > Nope, not actually: >>> del foo >>> print(globals().update({'foo':42}), foo) None 42 -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Jul 4 09:54:44 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 4 Jul 2018 23:54:44 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 11:52 PM, David Mertz wrote: > On Wed, Jul 4, 2018 at 3:02 AM Chris Angelico wrote: >> >> "Assignment is a statement" -- that's exactly the point under discussion. >> "del is a statement" -- yes, granted >> "function and class declarations are statements" -- class, yes, but >> you have "def" and "lambda" as statement and expression equivalents. >> "import is a statement" -- but importlib.import_module exists for a reason >> >> I'm going to assume that your term "mutating" there was simply a >> miswording, and that you're actually talking about *name binding*, >> which hitherto occurs only in statements. Yes, this is true. > > > Nope, not actually: > >>>> del foo >>>> print(globals().update({'foo':42}), foo) > None 42 > Try it inside a function though. ChrisA From mertz at gnosis.cx Wed Jul 4 10:20:35 2018 From: mertz at gnosis.cx (David Mertz) Date: Wed, 4 Jul 2018 10:20:35 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: Hmmm... I admit I didn't expect quite this behavior. I'm don't actually understand why it's doing what it does. >>> def myfun(): ... print(globals().update({'foo', 43}), foo) ... >>> myfun() Traceback (most recent call last): File "", line 1, in File "", line 2, in myfun TypeError: cannot convert dictionary update sequence element #0 to a sequence That said, this is a silly game either way. And even though you CAN (sometimes) bind in an expression pre-572, that's one of those perverse corners that one shouldn't actually use. On Wed, Jul 4, 2018 at 9:58 AM Chris Angelico wrote: > On Wed, Jul 4, 2018 at 11:52 PM, David Mertz wrote: > > On Wed, Jul 4, 2018 at 3:02 AM Chris Angelico wrote: > >> > >> "Assignment is a statement" -- that's exactly the point under > discussion. > >> "del is a statement" -- yes, granted > >> "function and class declarations are statements" -- class, yes, but > >> you have "def" and "lambda" as statement and expression equivalents. > >> "import is a statement" -- but importlib.import_module exists for a > reason > >> > >> I'm going to assume that your term "mutating" there was simply a > >> miswording, and that you're actually talking about *name binding*, > >> which hitherto occurs only in statements. Yes, this is true. > > > > > > Nope, not actually: > > > >>>> del foo > >>>> print(globals().update({'foo':42}), foo) > > None 42 > > > > Try it inside a function though. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed Jul 4 10:27:23 2018 From: steve.dower at python.org (Steve Dower) Date: Wed, 4 Jul 2018 07:27:23 -0700 Subject: [Python-Dev] PEP 572 semantics Message-ID: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> Now that it's a done deal, I am closely reviewing the semantics section of PEP 572. (I had expected one more posting of the final PEP, but it seems the acceptance came somewhere in a thread that was already muted.) Since there has been no final posting that I'm aware of, I'm referring to https://www.python.org/dev/peps/pep-0572/ as of about an hour before posting this (hopefully it doesn't take me that long). To be clear, I am *only* looking at the "Syntax and semantics" section. So if something has been written down elsewhere in the PEP, please take my questions as a request to have it referenced from this section. I also gave up on the discussion by the third python-dev thread - if there were things decided that you think I'm stupid for not knowing, it probably means they never made it into the PEP. = Syntax and Semantics Could we include the changes necessary to https://docs.python.org/3/reference/grammar.html in order to specify where these expressions are valid? And ideally check that they work. This may expose new exceptional cases, but will also clarify some of the existing ones, especially for those of us who write Python parsers. == Exceptional cases Are the cases in the "Exceptional cases" section supposed to raise SyntaxError on compilation? That seems obvious, but no harm in stating it. (FWIW, I'd vote to ban the "bad" cases in style guides or by forcing parentheses, rather than syntactically. And for anyone who wonders why that's different from my position on slashes in f-strings, it's because I don't think we can ever resolve these cases but I hope that one day we can fix f-string slashes :) ) == Scope of the target The PEP uses the phrase "an assignment expression occurs in a comprehension" - what does this mean? Does it occur when/where it is compiled, instantiated, or executed? This is important because where it occurs determines which scope will be modified. For sanity sake, I want to assume that it means compiled, but now what happens when that scope is gone? >>> def f(): ... return (a := i for i in range(5)) ... >>> list(f()) [0, 1, 2, 3, 4] # or a new error because the scope has gone? >>> a ??? I'll push back real hard on doing the assignment in the scope where the generator is executed: >>> def do_secure_op(name, numbers): ... authorised = check_authorised(name) ... if not all(numbers): ... raise ValueError() ... if not authorised: ... raise SecurityError() ... print('You made it!') ... >>> do_secure_op('whatever', (authorised := i for i in [1, 2, 3])) You made it! >>> authorised NameError: name 'authorised' is undefined >From the any()/all() examples, it seems clear that the target scope for the assignment has to be referenced from the generator scope (but not for other comprehension types, which can simply do one transfer of the assigned name after fully evaluating all the contents). Will this reference keep the frame object alive for as long as the generator exists? Can it be a weak reference? Are assignments just going to be silently ignored when the frame they should assign to is gone? I'd like to see these clarified in the main text. When an assignment is "expressly invalid" due to avoiding "edge cases", does this mean we should raise a SyntaxError? Or a runtime error? I'm not sure how easily these can be detected by our current compiler (or runtime, for that matter), but in the other tools that I work on it isn't going to be a trivial check. Also, I'm not clear at all on why [i := i+1 for i in range(5)] is a problem? Similarly for the other examples here. There's nothing wrong with `for i in range(5): i = i+1`, so why forbid this? == Relative precedence "may be used directly in a positional function call argument" - why not use the same syntax as generator expressions? Require parentheses unless it's the only argument. It seems like that's still got a TODO on it from one of the examples, so consider this a vote for matching generator-as-argument syntax. == Differences between assignment expressions I'm pretty sure the equivalent of "x = y = z = 0" would be "z := (y := (x := 0))". Not that it matters when there are no side-effects of assignment (unless we decide to raise at runtime for invalid assignments), but it could become a point of confusion for people in the future to see it listed like this. Assignment expressions always evaluate from innermost to outermost. Gramatically, "Single assignment targets *other than* NAME are not supported" would be more precise. And for specification's sake, does "not supported" mean "is a syntax error"? The "equivalent needs extra parentheses" examples add two sets of extra parentheses. Are both required? Or just the innermost set? --- Apologies for the lack of context. I've gone back and added the section headings for as I read through this section. Cheers, Steve From solipsis at pitrou.net Wed Jul 4 10:41:18 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 4 Jul 2018 16:41:18 +0200 Subject: [Python-Dev] Examples for PEP 572 References: Message-ID: <20180704164118.0e050c65@fsol> On Wed, 4 Jul 2018 09:43:04 -0300 Brett Cannon wrote: > > I think this is a very key point that the "this is bad" crowd is > overlooking. Even if this syntax turns out to not be that useful, abusing > the walrus operator can be fixed with a comment of "hard to follow; please > simplify" without teaching any new concepts or idioms The same could be said of any language mis-feature. Do we want to claim that questionable semantics in Javascript and PHP are not a problem, because the bad constructs can simply be turned away in code review? That sounds like a modern re-phrasing of the old argument, """C is not dangerous in itself, it's only the fault of incompetent programmers""". Just replace "incompetent programmers" with "complacent reviewers"... > Another point is we live in a dictatorship by choice, and yet some people > seem very upset our dictator dictated in the end. Not sure what you mean with "by choice". When I arrived here, I certainly wasn't asked whether I wanted the place to be a dictatorship or not ;-) Granted, I did choose to come here, but not because of a personal appeal for dictatorship. One could claim that the qualities of Python are due to it being a dictatorship. I think it's impossible to answer that question rigorously, and all we're left with is our personal feelings and biases on the subject. Regards Antoine. From guido at python.org Wed Jul 4 11:22:22 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 4 Jul 2018 08:22:22 -0700 Subject: [Python-Dev] Checking that PEP 558 (defined semantics for locals()) handles assignment expressions In-Reply-To: References: Message-ID: Correct, there shouldn't be any additional corner cases for your PEP due to inline assignments. We're not introducing new scopes nor other runtime mechanisms; the target of an inline assignment is either a global or a cell (nonlocal) defined at a specific outer level. What I wish we had (quite independent from PEP 572) is a way in a debugger to evaluate a comprehension that references a local in the current stack frame. E.g. here's something I had in a pdb session yesterday: (Pdb) p [getattr(context.context, x) for x in dir(context.context)] *** NameError: name 'context' is not defined (Pdb) global cc; cc = context (Pdb) p [getattr(cc.context, x) for x in dir(cc.context)] [, ............] (Pdb) The first attempt failed because the outer `context` was a local variable in some function, and pdb uses eval() to evaluate expressions. On Wed, Jul 4, 2018 at 5:36 AM Nick Coghlan wrote: > With PEP 572's formal acceptance now expected to be just a matter of > time, I'll limit any further personal expressions of opinion on the > change and the process taken to achieve it to other venues :) > > However, there's a design aspect I do need to address, which is to > make sure that PEP 558 (my proposal to actually nail down an > implementation independent specification for how we expect locals() to > behave at runtime) accurately covers the way runtimes and debuggers > are expected to behave when comprehensions and generator expressions > are executed at different scopes. > > My current view is that it's going to be OK to expose the differing > behaviours at module and function scope directly: > > * if you debug a module level comprehension or genexp, the target name > will be flagged on the code object as a dynamically resolved name > reference, so a debugger should handle it the same was as it would > handle any other "global NAME" declaration (and it will appear only in > globals(), not in locals()) > * similarly for a function comprehension or genexp, the name will show > up like any other "nonlocal NAME" (appears in locals() rather than > globals(), and is affected by the proposed changes in the interaction > between trace functions and the frame.f_locals attribute) > > I think that's an acceptable outcome, since it's a natural consequence > of PEP 572 defining its comprehension and generator expression > handling in terms of existing global and nonlocal scoping semantics. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Jul 4 11:23:39 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 4 Jul 2018 08:23:39 -0700 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> Message-ID: Thanks for thinking about the details! I want to answer all of these but right now I have some social obligations so it may be a few days. I expect the outcome of this investigation to result in an improved draft for PEP 572. On Wed, Jul 4, 2018 at 7:29 AM Steve Dower wrote: > Now that it's a done deal, I am closely reviewing the semantics section > of PEP 572. (I had expected one more posting of the final PEP, but it > seems the acceptance came somewhere in a thread that was already muted.) > > Since there has been no final posting that I'm aware of, I'm referring > to https://www.python.org/dev/peps/pep-0572/ as of about an hour before > posting this (hopefully it doesn't take me that long). > > To be clear, I am *only* looking at the "Syntax and semantics" section. > So if something has been written down elsewhere in the PEP, please take > my questions as a request to have it referenced from this section. I > also gave up on the discussion by the third python-dev thread - if there > were things decided that you think I'm stupid for not knowing, it > probably means they never made it into the PEP. > > = Syntax and Semantics > > Could we include the changes necessary to > https://docs.python.org/3/reference/grammar.html in order to specify > where these expressions are valid? And ideally check that they work. > This may expose new exceptional cases, but will also clarify some of the > existing ones, especially for those of us who write Python parsers. > > == Exceptional cases > > Are the cases in the "Exceptional cases" section supposed to raise > SyntaxError on compilation? That seems obvious, but no harm in stating > it. (FWIW, I'd vote to ban the "bad" cases in style guides or by forcing > parentheses, rather than syntactically. And for anyone who wonders why > that's different from my position on slashes in f-strings, it's because > I don't think we can ever resolve these cases but I hope that one day we > can fix f-string slashes :) ) > > == Scope of the target > > The PEP uses the phrase "an assignment expression occurs in a > comprehension" - what does this mean? Does it occur when/where it is > compiled, instantiated, or executed? This is important because where it > occurs determines which scope will be modified. For sanity sake, I want > to assume that it means compiled, but now what happens when that scope > is gone? > > >>> def f(): > ... return (a := i for i in range(5)) > ... > >>> list(f()) > [0, 1, 2, 3, 4] # or a new error because the scope has gone? > >>> a > ??? > > I'll push back real hard on doing the assignment in the scope where the > generator is executed: > > >>> def do_secure_op(name, numbers): > ... authorised = check_authorised(name) > ... if not all(numbers): > ... raise ValueError() > ... if not authorised: > ... raise SecurityError() > ... print('You made it!') > ... > >>> do_secure_op('whatever', (authorised := i for i in [1, 2, 3])) > You made it! > >>> authorised > NameError: name 'authorised' is undefined > > From the any()/all() examples, it seems clear that the target scope for > the assignment has to be referenced from the generator scope (but not > for other comprehension types, which can simply do one transfer of the > assigned name after fully evaluating all the contents). Will this > reference keep the frame object alive for as long as the generator > exists? Can it be a weak reference? Are assignments just going to be > silently ignored when the frame they should assign to is gone? I'd like > to see these clarified in the main text. > > > When an assignment is "expressly invalid" due to avoiding "edge cases", > does this mean we should raise a SyntaxError? Or a runtime error? I'm > not sure how easily these can be detected by our current compiler (or > runtime, for that matter), but in the other tools that I work on it > isn't going to be a trivial check. > > Also, I'm not clear at all on why [i := i+1 for i in range(5)] is a > problem? Similarly for the other examples here. There's nothing wrong > with `for i in range(5): i = i+1`, so why forbid this? > > == Relative precedence > > "may be used directly in a positional function call argument" - why not > use the same syntax as generator expressions? Require parentheses unless > it's the only argument. It seems like that's still got a TODO on it from > one of the examples, so consider this a vote for matching > generator-as-argument syntax. > > > == Differences between assignment expressions > > I'm pretty sure the equivalent of "x = y = z = 0" would be "z := (y := > (x := 0))". Not that it matters when there are no side-effects of > assignment (unless we decide to raise at runtime for invalid > assignments), but it could become a point of confusion for people in the > future to see it listed like this. Assignment expressions always > evaluate from innermost to outermost. > > Gramatically, "Single assignment targets *other than* NAME are not > supported" would be more precise. And for specification's sake, does > "not supported" mean "is a syntax error"? > > The "equivalent needs extra parentheses" examples add two sets of extra > parentheses. Are both required? Or just the innermost set? > > --- > > Apologies for the lack of context. I've gone back and added the section > headings for as I read through this section. > > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jul 4 11:56:11 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 01:56:11 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <20180704155610.GT14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 12:10:11AM -0700, Nathaniel Smith wrote: > Right, Python has a *very strong* convention that each line should > have at most one side-effect, import math, fractions, decimal (PEP 8 be damned, sometimes, especially in the REPL, this is much better than three separate imports.) values = [mapping.pop(key) for key in (1, 3, 6, 15)] Admittedly, that's an awfully specific case. But would you really re-write that as: values = [mapping.pop(1)] values.append(mapping.pop(3) values.append(mapping.pop(6) values.append(mapping.pop(15) just to keep the "one side-effect per line" rule? I wouldn't. Anyway, since "one side-effect per line" is just a convention, there's absolutely no reason why it cannot remain a convention. Don't do this: values = (x := expensive_func(1, 2))+(y := expensive_func(3, 4)) + x*y unless you really have to. It's just common sense. Conventions are good. If they'e enough to stop people writing: mylist = mylist.sort() or mylist.reverse() or mylist they'll be good enough to stop people stuffing every line full of assignment expressions. > and that if it does have a side-effect > it should be at the outermost level. I don't understand what that means. Do you mean the global scope? > I think the most striking evidence for this is that during the > discussion of PEP 572 we discovered that literally none of us ? > including Guido ? even *know* what the order-of-evaluation is inside > expressions. I'm not sure that "literally none of us" is quite true, after all the code is deterministic and well-defined and surely whoever maintains it probably understands it, but even if it were true, I don't see the connection between "we don't know the order of operations" and "Python has a rule no more than one-side effect per line". Seems a pretty tenuous ccomclusion to draw. > In fact PEP 572 now has a whole section talking about the > oddities that have turned up here so far, and how to fix them. Which > just goes to show that even its proponents don't actually think that > anyone uses side-effects inside expressions, because if they did, then > they'd consider these changes to be compatibility-breaking changes. If you're talking about fixing the scoping issues inside classes, that's no longer part of the PEP (although the current version hasn't been updated to reflect that). The weirdness of class scopes already exist. This might, at worst, make it more visible, but it isn't going to make the problem worse. If you're talking about something else, I can't think of what it might be. Can you explain? > Some people make fun of Python's expression/statement dichotomy, > because hey don't you know that everything can be an expression, > functional languages are awesome hurhur, Oh, you mean functional languages like Rust, Javascript, and Ruby? (Or at least, *almost* everything is an expression.) The functional programming paradigm is a lot more than just "everything is an expression", and very much *non*-functional languages like Javascript can be designed to treat everything as an expression. -- Steve From steve at pearwood.info Wed Jul 4 12:02:47 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 02:02:47 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <20180704160247.GU14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 10:20:35AM -0400, David Mertz wrote: > Hmmm... I admit I didn't expect quite this behavior. I'm don't actually > understand why it's doing what it does. > > >>> def myfun(): > ... print(globals().update({'foo', 43}), foo) Try it with a dict {'foo': 43} instead of a set :-) > ... > >>> myfun() > Traceback (most recent call last): > File "", line 1, in > File "", line 2, in myfun > TypeError: cannot convert dictionary update sequence element #0 to a > sequence I think Chris meant to try it inside a function using locals() rather than globals. > That said, this is a silly game either way. And even though you CAN > (sometimes) bind in an expression pre-572, that's one of those perverse > corners that one shouldn't actually use. Still, it is sometimes useful to explore scoping issues by using globals() and/or locals() to see what happens. But I would really hesitate to use them in production unless I was really careful. -- Steve From J.Demeyer at UGent.be Wed Jul 4 11:59:42 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 4 Jul 2018 17:59:42 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> Message-ID: <5B3CEEEE.30204@UGent.be> On 2018-07-04 03:31, INADA Naoki wrote: > I think both PEPs are relying on FASTCALL calling convention, > and can't be accepted until FASTCALL is stable & public. First of all, the fact that FASTCALL has not been made public should not prevent from discussing those PEPs and even making a (provisional?) decision on them. I don't think that the precise API of FASTCALL really matters that much. More importantly, I don't think that you can separate making FASTCALL public from PEP 576/580. As you noted in [1], making FASTCALL public means more than just documenting METH_FASTCALL. In particular, a new API should be added for calling objects using the FASTCALL convention. Here I mean both an abstract API for arbitrary callables as well as a specific API for certain classes. Since PEP 580 (and possibly also PEP 576) proposes changes to the implementation of FASTCALL, it makes sense to design the public API for FASTCALL after it is clear which of those PEPs (if any) is accepted. If we fix the FASTCALL API now, it might not be optimal when either PEP 576 or PEP 580 is accepted. Jeroen. [1] https://mail.python.org/pipermail/python-dev/2018-June/153956.html From steve at pearwood.info Wed Jul 4 12:17:56 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 02:17:56 +1000 Subject: [Python-Dev] PEP 572: Write vs Read, Understand and Control Flow In-Reply-To: <3638eff2-3ae6-d03b-90a1-c1ff750bc231@mail.mipt.ru> References: <3638eff2-3ae6-d03b-90a1-c1ff750bc231@mail.mipt.ru> Message-ID: <20180704161756.GV14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 03:36:18PM +0300, Ivan Pozdeev via Python-Dev wrote: > On 04.07.2018 15:29, Victor Stinner wrote: > >The PEP 572 has been approved, it's no longer worth it to discuss it ;-) > > > >Victor > > As of now, https://www.python.org/dev/peps/pep-0572/ is marked as "draft". https://mail.python.org/pipermail/python-dev/2018-July/154231.html https://mail.python.org/pipermail/python-dev/2018-July/154247.html The state of the published PEP has unfortunately lagged behind the state of the discussion, due to the overwhelming number of posts. -- Steve From mertz at gnosis.cx Wed Jul 4 12:31:11 2018 From: mertz at gnosis.cx (David Mertz) Date: Wed, 4 Jul 2018 12:31:11 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704160247.GU14437@ando.pearwood.info> References: <20180704160247.GU14437@ando.pearwood.info> Message-ID: On Wed, Jul 4, 2018 at 12:13 PM Steven D'Aprano wrote: > On Wed, Jul 04, 2018 at 10:20:35AM -0400, David Mertz wrote: > > Hmmm... I admit I didn't expect quite this behavior. I'm don't actually > > understand why it's doing what it does. > > > > >>> def myfun(): > > ... print(globals().update({'foo', 43}), foo) > > Try it with a dict {'foo': 43} instead of a set :-) > I think Chris meant to try it inside a function using locals() rather > than globals. Ah... you say "tomato" I say "raspberry"... set, dict... Yeah, typo fixed, it makes more sense. Still, even a `globals().update()` expression is a binding operation. But it works perfectly fine with locals() also :-): >>> def myfun(): ... print(locals().update({'foo': 44}), locals().get('foo')) ... >>> myfun() None 44 -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Wed Jul 4 13:13:15 2018 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 4 Jul 2018 10:13:15 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704133125.GR14437@ando.pearwood.info> References: <20180704133125.GR14437@ando.pearwood.info> Message-ID: On Wed, Jul 4, 2018 at 6:36 AM Steven D'Aprano wrote: > > On Wed, Jul 04, 2018 at 01:03:02AM -0700, Devin Jeanpierre wrote: > > > The PEP doesn't talk about it, but FWIW, Go and C++17 if statements > > allow you to put arbitrary simple statements in the suite header, and > > by doing that, solves all the issues you and the PEP mentioned. > > Whether they solve "all the issues" discussed in the PEP is doubtful, > since they're entirely limited to just if statements. The only issue that PEP 572 has with limiting to if statements is: "Disadvantages: Answers only a fraction of possible use-cases, even in if/whilestatements." The Go/C++17 syntax resolves that (while still being limited to if statements and while statements -- the primary use cases AFAIK, and the only ones Nathaniel agrees with). > > In Python it'd look like this: > > > > if x = expr(); x < 0: > > do_stuff() > > > That's even more doubtful, since the semicolon in Python already has a > meaning of separating statements on a line. That could be equivalent to: > > if x = expr() > x < 0: > do_stuff() > > which would clearly have to be a syntax error. *Two* syntax errors. > > Python's parser is restricted to LL(1) by design, so treating semicolons > in "if" statements as a special case might not even be possible. > But even if it is possible, not only would this special case > break the Zen, but it would be UGLY too. It isn't a special case, and it's still LL(1). Once you encounter an "if", you parse n simple statements separated by a ";", until you reach a colon. Where LL(1) runs into trouble is that last statement must be an Expr statement, but that can be checked after parsing (e.g. during compilation). I don't understand why you think this is "clearly" a syntax error. (So is ":=", if you restrict yourself to the current Python syntax.) As for ugly... well, compare: while (kv := x.pop())[0] is not None: foo(kv[1]) while k, v = x.pop(); k is not None: foo(v) if (v := d[k]) == _EOF: return None if v = d[k]; v == _EOF: return None I like ":=" when it is the only thing in the expression: "if a:= b:" etc. Every other time I would prefer C++/Go's syntax. -- Devin From tim.peters at gmail.com Wed Jul 4 13:21:19 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 4 Jul 2018 12:21:19 -0500 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> Message-ID: Just a quickie: [Steve Dower] > > The PEP uses the phrase "an assignment expression > occurs in a comprehension" - what does this mean? It's about static analysis of the source code, at compile-time, to establish scopes. So "occurs in" means your eyeballs see an assignment expression in a comprehension. > Does it occur when/where it is compiled, instantiated, or executed? This > is important because where it occurs determines which scope will be > modified. For sanity sake, I want to assume that it means compiled, Probably ;-) I just don't know _exactly_ what those distinctions mean to you. > but now what happens when that scope is gone? Nothing new. There's no new concept of "scope" here, which is why the PEP doesn't say a whole lot about scope. > > >>> def f(): > > ... return (a := i for i in range(5)) > > ... Same as now, `i` is local to the synthetic nested function created for the genexp. The scope of `a` is determined by pretending the assignment occurred in the block containing the outermost (textually - static analysis) comprehension. In this case, `a = anything` before the `return` would establish that `a` is local to `f`, so that's the answer: `a` is local to `f`. If `a` had been declared global in `f`, then `a` in the genexp would be the same global `a`. And similarly if `a` had been declared nonlocal.in `f`. In all cases the scope resolution is inherited from the closest containing non-comprehension/genexp block, with the twist if that if a name is unknown in that block, the name is established as being local to that block. So your example is actually the subtlest case. > >>> list(f()) > > [0, 1, 2, 3, 4] # or a new error because the scope has gone? Function scopes in Python have "indefinite extent", and nothing about that changes. So, ya, that's the output - same as today if you changed your example to delete the "a :=" part. Internally, f's local `a` was left bound to 4, but there's no way to see that here because the genexp has been run to exhaustion and reference-counting has presumably thrown everything away by now. >>> a > ??? Same thing typing `a` would result in if you had never typed `list(f())`. Here's a variation: def f(): > yield (a := i for i in range(5)) yield a Then: >>> g = f() >>> list(next(g)) [0, 1, 2, 3, 4] >>> next(g) 4 > I'll push back real hard on doing the assignment in the scope > where the generator is executed: > > > > >>> def do_secure_op(name, numbers): > > ... authorised = check_authorised(name) This instance of `authorized` is local to `do_secure_op`. > ... if not all(numbers): > > ... raise ValueError() > > ... if not authorised: > > ... raise SecurityError() > > ... print('You made it!') > > ... > > >>> do_secure_op('whatever', (authorised := i for i in [1, 2, 3])) And this instance of `authorized` is a global (because the genexp appears in top-level code, so its containing block is the module). The two instances have nothing to do with each other. > You made it! Yup - you did! > >>> authorised > > NameError: name 'authorised' is undefined It would display 3 instead. > From the any()/all() examples, it seems clear that the target scope for > > the assignment has to be referenced from the generator scope (but not > > for other comprehension types, which can simply do one transfer of the > > assigned name after fully evaluating all the contents). I don't think that follows. It _may_ in some cases. For example, def f(): i = None # not necessary, just making i's scope obvious def g(ignore): return i+1 return [g(i := j) for j in range(3)] _While_ the list comprehension is executing, it needs to rebind f's `i` on each iteration so that the call to `g()` on each iteration can see `i`'s then-current value. > Will this reference keep the frame object alive for as > long as the generator exists? Can it be a weak reference? > Are assignments just going to be silently ignored when > the frame they should assign to is gone? I'd like to see these > clarified in the main text. Here you're potentially asking questions about how closures work in Python (in the most-likely case that an embedded assignment-statement target resolves to an enclosing function-local scope), but the PEP doesn't change anything about how closures already work. Closures are implemented via "cell objects", one per name, which already supply both "read" and "write" access to both the owning and referencing scopes. def f(): a = 42 return (a+1 for i in range(3)) That works fine today, and a cell object is used in the synthesized genexp function to get read access to f's local `a`. But references to `a` in `f` _also_ use that cell object. - the thing that lives in f's frame isn't really the binding for `a`, but a reference to the cell object that _holds_ a's current binding. The cell object doesn't care whether f's frame goes away (except that the cell object's refcount drops by one when f's frame vanishes). Nothing about that changes if the synthesized genexp wants write access instead. While a gentle introduction to how closures are implemented in Python would be a cool thing, this PEP is the last place to include one ;-) It may help to realize that there's nothing here that can't be done today by explicitly writing nested functions with appropriate scope declarations, in a straightforward way. There's nothing _inherently_ new. Huh! Not so much "a quickie" after all :-( So I'll stop here for now. Thank you for the critical reading and feedback! -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Wed Jul 4 13:33:17 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 4 Jul 2018 20:33:17 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> On 04.07.2018 11:54, Serhiy Storchaka wrote: > 04.07.18 10:06, Tim Peters ????: >> [Tim] >> ?>> I really don't know what Guido likes best about this, but for me >> it's >> >> ?>> the large number of objectively small wins in `if` and `while` >> >> ?>> contexts.? ?They add up.? That conclusion surprised me.? That >> there are >> >> ?>> occasionally bigger wins to be had is pure gravy. >> >> >> [Serhiy Storchaka] >> ?> Could you please show me several examples in real code? I >> ?> have not seen any win yet. >> >> My PEP Appendix was derived entirely from looking at real code. If >> you don't believe the examples I showed there are wins (and I don't >> know whether you've seen them, because your original message in this >> thread only echoed examples from the body of the PEP), then what we >> each mean by "win" in this context has no intersection, so discussing >> it would be futile (for both of us). > > Sorry, this PEP was rewritten so many times that I missed your Appendix. > >> while total != (total := total + term): >> ??? term *= mx2 / (i*(i+1)) >> ??? i += 2 >> return total > > This code looks clever that the original while loop with a break in a > middle. I like clever code. But it needs more mental efforts for > understanding it. > > I admit that this is a good example. > > There is a tiny problem with it (and with rewriting a while loop as a > for loop, as I like). Often the body contains not a single break. In > this case the large part of cleverness is disappeared. :-( It took me a few minutes to figure out that this construct actually checks term == 0. So, this example abuses the construct to do something it's not designed to do: perform an unrelated operation before checking the condition. (Cue attempts to squeeze ever mode code here.) I would fail it in review. This "clever" code is exactly what Perl burned itself on and what Python, being its antithesis, was specifically designed to avoid. > >> if result := solution(xs, n): >> ??? # use result > > It looks equally clear with the original code. This is not enough for > introducing a new syntax. > >> if reductor := dispatch_table.get(cls): >> ??? rv = reductor(x) >> elif reductor := getattr(x, "__reduce_ex__", None): >> ??? rv = reductor(4) >> elif reductor := getattr(x, "__reduce__", None): >> ??? rv = reductor() >> else: >> ??? raise Error("un(shallow)copyable object of type %s" % cls) > > I was going to rewrite this code as > > ??? reductor = dispatch_table.get(cls) > ??? if reductor: > ??????? rv = reductor(x) > ??? else: > ??????? rv = x.__reduce_ex__(4) > > There were reasons for the current complex code in Python 2, but now > classic classes are gone, and every class has the __reduce_ex__ method > which by default calls __reduce__ which by default is inherited from > object. With that simplification the benefit of using ":=" in this > example looks less impressed. > >> if (diff := x - x_base) and (g := gcd(diff, n)) > 1: >> ??? return g > >> while a > (d := x // a**(n-1)): >> ??? a = ((n-1)*a + d) // n >> return a > > I would have a fun writing such code. As well as I believe you had a > fun writing it. But I suppose the original code is simpler for a > casual reader, and I will refrain from using such code in a project > maintained by other people (in particular in the Python stdlib). > >> Which is what I expect:? the treatment you gave to the examples from >> the body of the PEP suggests you're determined not to acknowledge any >> "win", however small. > > I have to admit that *there are* examples that can have a small win. I > wondering why your examples are not used in the PEP body instead of > examples which play *against* PEP 572. > > Yet a win too small to me for justifying such syntax change. I know > that I can not convince you or Guido. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From yselivanov.ml at gmail.com Wed Jul 4 13:50:48 2018 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 4 Jul 2018 13:50:48 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> References: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> Message-ID: On Wed, Jul 4, 2018 at 1:35 PM Ivan Pozdeev via Python-Dev wrote: > > On 04.07.2018 11:54, Serhiy Storchaka wrote: > >> while total != (total := total + term): > >> term *= mx2 / (i*(i+1)) > >> i += 2 > >> return total > > > > This code looks clever that the original while loop with a break in a > > middle. I like clever code. But it needs more mental efforts for > > understanding it. > > > > I admit that this is a good example. > > > > There is a tiny problem with it (and with rewriting a while loop as a > > for loop, as I like). Often the body contains not a single break. In > > this case the large part of cleverness is disappeared. :-( > > It took me a few minutes to figure out that this construct actually > checks term == 0. Wow, I gave up on this example before figuring this out (and I also stared at it for a good couple of minutes). Now it makes sense. It's funny that this super convoluted snippet is shown as a good example for PEP 572. Although almost all PEP 572 examples are questionable. Yury From tim.peters at gmail.com Wed Jul 4 13:58:39 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 4 Jul 2018 12:58:39 -0500 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> References: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> Message-ID: [Serhiy Storchaka] > > Sorry, this PEP was rewritten so many times that I missed your > [Tim's] Appendix. > > > >> while total != (total := total + term): > >> term *= mx2 / (i*(i+1)) > >> i += 2 > >> return total > > > > This code looks clever that the original while loop with a break in a > > middle. I like clever code. But it needs more mental efforts for > > understanding it. > > > > I admit that this is a good example. > > > > There is a tiny problem with it (and with rewriting a while loop as a > > for loop, as I like). Often the body contains not a single break. In > > this case the large part of cleverness is disappeared. :-( [Ivan Pozdeev] > It took me a few minutes to figure out that this construct actually > > checks term == 0. > > > > So, this example abuses the construct to do something it's not designed > > to do: perform an unrelated operation before checking the condition. > > (Cue attempts to squeeze ever mode code here.) I would fail it in review. > > > > This "clever" code is exactly what Perl burned itself on and what > > Python, being its antithesis, was specifically designed to avoid. So you didn't read the PEP Appendix at all, and Serhiy did but apparently skipped reading what the PEP _said_ about that example. It was clearly identified as abuse: a case in which using assignment expressions made the code significantly WORSE. I gave examples of both "wins" and "losses" while staring at real code - I wasn't searching for "proof" that a pre-determined conclusion was justified. So I wholly agree with you (Ivan) about that example - and that it struck Serhiy as a good example convinces me more than before that there's no overlap in the ways Serhiy and I view this part of the world ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jul 4 14:02:15 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 04:02:15 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: <20180704133125.GR14437@ando.pearwood.info> Message-ID: <20180704180215.GW14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 10:13:15AM -0700, Devin Jeanpierre wrote: > > > In Python it'd look like this: > > > > > > if x = expr(); x < 0: > > > do_stuff() [...] > > Python's parser is restricted to LL(1) by design, so treating semicolons > > in "if" statements as a special case might not even be possible. > > But even if it is possible, not only would this special case > > break the Zen, but it would be UGLY too. > > It isn't a special case, and it's still LL(1). Once you encounter an > "if", you parse n simple statements separated by a ";", until you > reach a colon. I'll take your word about the LL(1). Did you actually mean arbitrary simple statements? if import math; mylist.sort(); print("WTF am I reading?"); True: pass A more reasonable suggestion would be to only allow *assignments*, not arbitrary simple statements. But that's another special case: - everywhere else where a semicolon is legal, it can separate arbitrary simple statements; - except inside an if clause, where only assignments are allowed. > I don't understand why you think this is "clearly" a syntax error. Taking the principle that semicolons separate statements, you have an if statement if condition: with one or more semicolon-separated statements between the "if" and the condition: if statement; statement; condition: If we stick to the rule that semicolons separate statements, that means we have: if statement # SyntaxError statement # okay condition: # SyntaxError If we don't want that, we need a new rule to treat semicolons differently inside if statements that they're treated elsewhere. If we applied this rule "allow statements separated by semicolons" everywhere, we'd get this: # I trust you don't want anything even close to these from x=1; sys import y=2; argv data x=1; y=2; = sorted(data) raise x=1; y=2; Exception as err: So we have to treat semicolons inside if statements differently from semicolons everywhere else. That what I meant by saying it would be a special case. The beauty of assignment expressions is that they AREN'T a special case. Being an expression, they're legal anywhere any other expression is allowed, and illegal anywhere where expressions aren't allowed. -- Steve From tim.peters at gmail.com Wed Jul 4 14:16:00 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 4 Jul 2018 13:16:00 -0500 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> Message-ID: [Yury Selivanov] > Wow, I gave up on this example before figuring this out (and I also > > stared at it for a good couple of minutes). Now it makes sense. It's > > funny that this super convoluted snippet is shown as a good example > > for PEP 572. Although almost all PEP 572 examples are questionable. And another who didn't actually read the PEP Appendix. See my reply just before this one: yes, the Appendix gave that as a good example, but as a good example of assignment-expression ABUSE. The opposite of something desirable. I've never insisted there's only one side to this, and when staring at code was equally interested in cases where assignment expressions would hurt as where they would help. I ended up giving more examples where they would help, because after writing up the first two bad examples in the Appendix figured it was clear enough that "it's a bad idea except in cases where it _obviously_ helps". Same way, e.g., as when list comprehensions were new, I focused much more on cases where they might help after convincing myself that a great many nested loops building lists were much better left _as_ nested loops. So I looked instead for real-code cases where they would obviously help, and found plenty. And I'm really glad Python added listcomps too, despite the possibility of gross abuse ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jul 4 14:21:46 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 04:21:46 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> References: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> Message-ID: <20180704182145.GX14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 08:33:17PM +0300, Ivan Pozdeev via Python-Dev wrote: > >>while total != (total := total + term): > >>??? term *= mx2 / (i*(i+1)) > >>??? i += 2 > >>return total [...] > It took me a few minutes to figure out that this construct actually > checks term == 0. That's badly wrong. Hint: it's floating point code. total and term are floats, so things like this are possible: py> total = 9.5e95 py> term = 1.2 py> total + term == total True As you can see, term is nothing even close to zero. > So, this example abuses the construct to do something it's not designed > to do: perform an unrelated operation before checking the condition. Well, that's one opinion. > (Cue attempts to squeeze ever mode code here.) I would fail it in review. Okay. -- Steve From rob.cliffe at btinternet.com Wed Jul 4 14:20:51 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 4 Jul 2018 19:20:51 +0100 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <5c7b6b5e-8ed7-5d2f-babd-2c44958e103d@btinternet.com> On 04/07/2018 02:54, Terry Reedy wrote: > > The 2-argument form of iter is under-remembered and under-used. The > length difference is 8. > ??? while (command := input("> ")) != "quit": > ??? for command in iter(lambda: input("> "), "quit"): A general principle that Chris Angelico has repeatedly mention applies here:? If you want (as I instinctively would) ??? while (command := input("> ")).lower() != "quit": you can't express that in your iter version. Regards Rob Cliffe From skrah at bytereef.org Wed Jul 4 11:48:21 2018 From: skrah at bytereef.org (Stefan Krah) Date: Wed, 4 Jul 2018 17:48:21 +0200 Subject: [Python-Dev] Tone it down on Twitter? Message-ID: <20180704154821.GA7081@bytereef.org> Apparently I have made it into "club of three who don't care much about opinions of others" for the crime of a single +0.5 for PEP-572 without participating in the discussion at all (neither did Jason). https://twitter.com/SerhiyStorchaka/status/1014274554452209665 This is a new high for Twitter gossip. Well done. Perhaps in the next vote the politbureau can indicate the intended outcome beforehand so we know how to vote. Thanks, Stefan Krah From kirillbalunov at gmail.com Wed Jul 4 13:58:36 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Wed, 4 Jul 2018 20:58:36 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: Forwarding the reply to the list. Accidentally pressed the wrong button and sent this message personally to Serhiy. Sorry :) ??, 4 ???. 2018 ?. ? 11:57, Serhiy Storchaka : > > if reductor := dispatch_table.get(cls): > > rv = reductor(x) > > elif reductor := getattr(x, "__reduce_ex__", None): > > rv = reductor(4) > > elif reductor := getattr(x, "__reduce__", None): > > rv = reductor() > > else: > > raise Error("un(shallow)copyable object of type %s" % cls) > > I was going to rewrite this code as > > reductor = dispatch_table.get(cls) > if reductor: > rv = reductor(x) > else: > rv = x.__reduce_ex__(4) > > There were reasons for the current complex code in Python 2, but now > classic classes are gone, and every class has the __reduce_ex__ method > which by default calls __reduce__ which by default is inherited from > object. With that simplification the benefit of using ":=" in this > example looks less impressed. > > Yes you are right with this particular snippet (in its current version, it is an echo from Python 2). But I think you have missed the general idea: The logic of this code is flat (like a switch) but the visual structure is pretty nested. That is why (maybe just for me) there is a mental imbalance in perception. And this type of code is rather common. Therefore, for me, the main gain from using the assignment expression in these patterns of code is that the visual syntax emphasizes the semantics. To be honest, I do not see any benefit of usage of assignment expression outside `if` and `while` headers, but if others do see let it be so. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Wed Jul 4 05:59:47 2018 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Wed, 4 Jul 2018 11:59:47 +0200 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Tue, Jul 3, 2018 at 11:52 PM Chris Angelico wrote: > On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka > wrote: > > I believe most Python users are not > > professional programmers -- they are sysadmins, scientists, hobbyists and > > kids -- > > [citation needed] > Let's focus on France: 1) there are ~800 000 person aged 15. (Source: https://www.insee.fr/fr/statistiques/1892088?sommaire=1912926 ) 2) 40% of these are pupils in the "fili?re g?n?rale" (Source: https://fr.wikipedia.org/wiki/Baccalaur%C3%A9at_en_France#Statistiques ). 3) Since 2017, Python is the language used to teach algorithmics to these pupils. => We can conclude that there are at least 320 000 pupils learning Python this year, and safely assume that most of them are not "professional programmers". Note also that this number is accretive (i.e.: 320 000 this year, 640 000 next year, etc.). 4) The annual turnover for the IT sector in France (including: software vendor, service and consulting) was 54 billions euros in 2017. This probably translates to around or less than 600 000 IT professionals. How many of them are developers ? I have no idea, but I'm sure it's less that 50%. How many of them are developing professionally in Python ? I have no idea, I'd guess less than 10%, but let's assume 20%. => This gives less than 60 000 professional Python programmers in France. Probably much less. Conclusion: there are probably more than 5 times more non professional Python programmers in France (counting only the high-school pupils, not other categories such as their teachers, but also scientist, sysadmins or hobbyist) than professional programmers. Next year it will be (more than) 1 to 10. The year after that, 1 to 15, etc. S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Wed Jul 4 06:36:39 2018 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Wed, 4 Jul 2018 12:36:39 +0200 Subject: [Python-Dev] Fwd: Examples for PEP 572 In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 12:09 PM Chris Angelico wrote: > > Even assuming your figures to be 100% accurate, I don't think you can > accept that scaling. Are you claiming that every high school student > (a) continues to use Python forever, and (b) continues to use it at a > non-professional level? I find that highly unlikely. > What I'm claiming is that ~300 000 pupils in "seconde" (~ 10th grade in the US, cf. https://jeretiens.net/conversion-des-niveaux-scolaires-france-grande-bretagne-etats-unis/amp/ ) are supposed to start learning (some basics of) Python this year during their math classes, next year theses 300k pupils will be in 11th grade and still be doing some Python, and next year they will be in 12th and pass their baccalaur?at (~ "high school diploma"). That's 1 millions pupils actively learning some Python in 2020, not counting other categories of students ("classes pr?pa", engineering school, etc.). A majority of them will probably stop being involved with Python later, and a small percentage will become "professional Python programmers". I'm pretty dubious that these figures will correspond to the rest of > the world, It's probably similar to every country where Python is the recommended or mandated language for teaching algorithmics or the basics of CS in school. I have unfortunately no idea if there are any other countries in a similar situation. And dissimilar to countries where CS is not taught in schools, or another language is used (Scratch or other block-languages are usually popular). (Fun fact: before Python, the language that used to be taught in high school in France, though only to a minority of students, used to be OCaml, a language that was created by french academics in the 80s and 90s. Python won because it's simpler and more mainstream.) S. > where you can't expect that every single high school > student is using Python. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/sfermigier%2Blists%40gmail.com > -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From sf at fermigier.com Wed Jul 4 06:40:05 2018 From: sf at fermigier.com (=?UTF-8?Q?St=C3=A9fane_Fermigier?=) Date: Wed, 4 Jul 2018 12:40:05 +0200 Subject: [Python-Dev] Examples for PEP 572 References: Message-ID: On Wed, Jul 4, 2018 at 12:36 PM St?fane Fermigier wrote: > > And dissimilar to countries where CS is not taught in schools, or another > language is used (Scratch or other block-languages are usually popular). > Just found: https://www.ibtimes.co.uk/coding-uk-classroom-python-overtakes-french-most-popular-language-primary-schools-1517491 :) as a pythonista. :( as a frenchman Also, not sure how that translates into raw numbers. S. -- Stefane Fermigier - http://fermigier.com/ - http://twitter.com/sfermigier - http://linkedin.com/in/sfermigier Founder & CEO, Abilian - Enterprise Social Software - http://www.abilian.com/ Chairman, Free&OSS Group @ Systematic Cluster - http://www.gt-logiciel-libre.org/ Co-Chairman, National Council for Free & Open Source Software (CNLL) - http://cnll.fr/ Founder & Organiser, PyParis & PyData Paris - http://pyparis.org/ & http://pydata.fr/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Wed Jul 4 14:31:00 2018 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Wed, 4 Jul 2018 14:31:00 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> Message-ID: On Wed, Jul 4, 2018 at 2:16 PM Tim Peters wrote: > > [Yury Selivanov] > > Wow, I gave up on this example before figuring this out (and I also > > > stared at it for a good couple of minutes). Now it makes sense. It's > > > funny that this super convoluted snippet is shown as a good example > > > for PEP 572. Although almost all PEP 572 examples are questionable. > > And another who didn't actually read the PEP Appendix. See my reply just before this one: yes, the Appendix gave that as a good example, but as a good example of assignment-expression ABUSE. The opposite of something desirable. > > I've never insisted there's only one side to this, and when staring at code was equally interested in cases where assignment expressions would hurt as where they would help. I ended up giving more examples where they would help, because after writing up the first two bad examples in the Appendix figured it was clear enough that "it's a bad idea except in cases where it _obviously_ helps". > > Same way, e.g., as when list comprehensions were new, I focused much more on cases where they might help after convincing myself that a great many nested loops building lists were much better left _as_ nested loops. So I looked instead for real-code cases where they would obviously help, and found plenty. > > And I'm really glad Python added listcomps too, despite the possibility of gross abuse ;-) Thank you for the clarification, Tim. I agree, list comprehensions can indeed be abused and we all see that happening occasionally. However, assignment expressions make it easy (almost compelling) to push more logic even to simple comprehensions (with one "for" / one "if"). You probably understand why most core devs are irritated with the PEP: the majority thinks that the potential of its abuse can't be compared to any other Python syntax. It's sad that the PEP doesn't really address that except saying "This is a tool, and it is up to the programmer to use it where it makes sense, and not use it where superior constructs can be used." Well, to those of use who routinely review code written by people who aren't proficient Python coders (read most people working at Google, Facebook, Dropbox, Microsoft, or really any company) this argument is very weak. In other words: I'm looking forward to reviewing clever code written by clever people. Spending extra time arguing if a snippet is readable or not should be fun and productive, right? :) Disabling := as a company-wide policy will probably be the only way out. Yury From solipsis at pitrou.net Wed Jul 4 14:34:33 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 4 Jul 2018 20:34:33 +0200 Subject: [Python-Dev] Tone it down on Twitter? References: <20180704154821.GA7081@bytereef.org> Message-ID: <20180704203433.57a1a0e2@fsol> I was going to answer that I agree the language there is unwarranted, but then it came to me that the best way to keep things civil is perhaps not to discuss Twitter-emitted opinions back on python-dev. Regards Antoine. On Wed, 4 Jul 2018 17:48:21 +0200 Stefan Krah wrote: > Apparently I have made it into "club of three who don't care much about > opinions of others" for the crime of a single +0.5 for PEP-572 without > participating in the discussion at all (neither did Jason). > > https://twitter.com/SerhiyStorchaka/status/1014274554452209665 > > This is a new high for Twitter gossip. Well done. Perhaps in the next vote > the politbureau can indicate the intended outcome beforehand so we know how > to vote. > > > > Thanks, > > Stefan Krah > > > From srkunze at mail.de Wed Jul 4 14:32:32 2018 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 4 Jul 2018 20:32:32 +0200 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: Sorry for adding yet another mail. :-( On 04.07.2018 10:54, Serhiy Storchaka wrote: > Sorry, this PEP was rewritten so many times that I missed your Appendix. > >> while total != (total := total + term): >> ??? term *= mx2 / (i*(i+1)) >> ??? i += 2 >> return total > This very example here caught my eye. Isn't total not always equal to total? What would "regular" Python have looked like? Regards, Sven From njs at pobox.com Wed Jul 4 14:41:38 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 4 Jul 2018 11:41:38 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704155610.GT14437@ando.pearwood.info> References: <20180704155610.GT14437@ando.pearwood.info> Message-ID: On Wed, Jul 4, 2018, 09:09 Steven D'Aprano wrote: > On Wed, Jul 04, 2018 at 12:10:11AM -0700, Nathaniel Smith wrote: > > > Right, Python has a *very strong* convention that each line should > > have at most one side-effect, > > import math, fractions, decimal > > (PEP 8 be damned, sometimes, especially in the REPL, this is much > better than three separate imports.) > Sure, the nice thing about "very strong convention" versus "strict rule" is that there's a bit of lee-way at the edges. I'm pretty sure PEP 572 would not be accepted if it's proponents were expecting it to be forbidden by PEP 8 though. > values = [mapping.pop(key) for key in (1, 3, 6, 15)] > > Admittedly, that's an awfully specific case. But would you really > re-write that as: > > values = [mapping.pop(1)] > values.append(mapping.pop(3) > values.append(mapping.pop(6) > values.append(mapping.pop(15) > > just to keep the "one side-effect per line" rule? I wouldn't. > I don't think I've ever seen something like this in real life, but sure, I'd say it's pushing the edge of good taste, but maybe just squeaking under the line. A more common edge case for my rule would be: x = d.pop() where one could argue that this is doing two things: (1) removing an item from a container, and (2) storing it in a variable. But that's ok for me argument, because my argument is about how people represent the code in their mind so they can reason about it, and "take a value from *here* and put it *there*" is still simple enough to feel like a single operation, and there's no possibility of confusion about what order things happen in. "Import these three modules" is similar. This is also why I'm ok with if x := fn(): But my mind stack-overflows when trying to read if (x := fn()) is None: The first is a complex operation, that both triggers a jump and mutates a variable, which is two side-effects. But the two side-effects are closely linked and cannot affect each other, so I can think of it as one "save and jump" operation. In the second case, there's clearly 3 separate phases that I have to simulate in my head: first the mutation of 'x', then switching back to expression evaluation to do the comparison with None, and then the 'if'. And of course it gets much worse in the PEPs motivating examples, where a variable is created and used within the same expression. This is a whole mode of reasoning about code that python has never required before (in practice), and even if you think := is fabulous you should still at least be able to acknowledge that this is a huge shift. > Anyway, since "one side-effect per line" is just a convention, there's > absolutely no reason why it cannot remain a convention. Don't do this: > > values = (x := expensive_func(1, 2))+(y := expensive_func(3, 4)) + x*y > > unless you really have to. It's just common sense. > I think this really underestimates the importance of this kind of convention in how people learn and use the language, and how the language and it's conventions influence each other. If list.sort returned self, then people would absolutely write things like mylist.sort().reverse() And then we'd have lint rules and arguments about it and we'd have to waste precious brain power deciphering this in other people's code and arguing about it in code reviews. By not returning 'self', the language clearly takes a position that you shouldn't do this, and cuts off all that discussion. The := operator is exactly the opposite: it's *only purpose* is to break this convention, which is a pretty strong statement that now we are supposed to embed side effects inside expressions, and be prepared to read code that does this. > Conventions are good. If they'e enough to stop people writing: > > mylist = mylist.sort() or mylist.reverse() or mylist > > they'll be good enough to stop people stuffing every line full of > assignment expressions. > > > > > and that if it does have a side-effect > > it should be at the outermost level. > > I don't understand what that means. Do you mean the global scope? > I mean something like "the top node of the statement's AST". A normal statement: print([a]) Technically only one side-effect in this line, but it's in a weird place: [print(a)] > > I think the most striking evidence for this is that during the > > discussion of PEP 572 we discovered that literally none of us ? > > including Guido ? even *know* what the order-of-evaluation is inside > > expressions. > > I'm not sure that "literally none of us" is quite true, after all the > code is deterministic and well-defined and surely whoever maintains it > probably understands it, but even if it were true, I don't see the > connection between "we don't know the order of operations" and "Python > has a rule no more than one-side effect per line". Seems a pretty > tenuous ccomclusion to draw. > Side-effects and sequencing are intimately linked. When you have side-effects, you have to think about their order; contrariwise, in pure code, order of evaluation is a meaningless concept. Therefore, Python programmers have to know the order of evaluation inside expressions exactly to the extent that they embed side-effecting operations inside of expressions. > > > In fact PEP 572 now has a whole section talking about the > > oddities that have turned up here so far, and how to fix them. Which > > just goes to show that even its proponents don't actually think that > > anyone uses side-effects inside expressions, because if they did, then > > they'd consider these changes to be compatibility-breaking changes. > > If you're talking about fixing the scoping issues inside classes, that's > no longer part of the PEP (although the current version hasn't been > updated to reflect that). The weirdness of class scopes already exist. > This might, at worst, make it more visible, but it isn't going to make > the problem worse. > > If you're talking about something else, I can't think of what it might > be. Can you explain? > I'm talking about things like the PEP's discussion of what order keys and values should be evaluated in dict displays. Previously, no one knew or cared about this order (outside from, yes, a tiny handful of people maintaining this bit of the compiler). Now that we're adding a language construct whose whole purpose is to embed side-effects inside expressions, this suddenly becomes something we all have to actually think about. > > > Some people make fun of Python's expression/statement dichotomy, > > because hey don't you know that everything can be an expression, > > functional languages are awesome hurhur, > > Oh, you mean functional languages like Rust, Javascript, and Ruby? > > (Or at least, *almost* everything is an expression.) > > The functional programming paradigm is a lot more than just "everything > is an expression", and very much *non*-functional languages like > Javascript can be designed to treat everything as an expression. > Sure, obviously languages can be designed that way. And there's a tradition which argues that all languages *should* be designed that way, which I'm hand-wavily attributing to "functional folks" ? it goes back to Lisp, ultimately, and in functional languages having a distinction between statements and expressions *is* weird and artificial. JS, Ruby, and Rust aren't functional languages, but in this respect they were very much influenced by this functional tradition. Python quite intentionally diverges. Why? Why bother distinguishing statements and expressions at all? Python doesn't normally introduce concepts like this for no reason. Is JS just a more elegant and thoughtfully designed language than Python? Had Guido just never heard of Lisp? Obviously not. Python has a statement/expression dichotomy exactly because Python is imperative, and the basic program structure is "do this, and then do this, and then do this", where each "do this" is one side-effecting operation. In this context it's super helpful to have a really clear, privileged syntactic encoding of the things you "do" (statements) and their relative ordering (suites). -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Wed Jul 4 14:50:53 2018 From: steve.dower at python.org (Steve Dower) Date: Wed, 4 Jul 2018 11:50:53 -0700 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> Message-ID: On 04Jul2018 1021, Tim Peters wrote: > Same as now, `i` is local to the synthetic nested function created for > the genexp.? The scope of `a` is determined by pretending the assignment > occurred in the block containing the outermost (textually - static > analysis) comprehension.? In this case, `a = anything` before the > `return` would establish that `a` is local to `f`, so that's the > answer:? `a` is local to `f`.? If `a` had been declared global in `f`, > then `a` in the genexp would be the same global `a`.? And similarly if > `a` had been declared nonlocal.in `f`. > > In all cases the scope resolution is inherited from the closest > containing non-comprehension/genexp block, with the twist if that if a > name is unknown in that block, the name is established as being local to > that block.? So your example is actually the subtlest case. Okay, so as far as the specification goes, saying "assignment expressions in comprehensions get or create a cell variable in the defining scope and update its value" satisfies me just fine (or some other wording that more closely mirrors the actual behaviour - all my work here is on my own compiler, not the actual CPython one, and I don't know that they're identical). I don't think this should be left assumed by the PEP. If it's likely to be a restriction on other implementations to say "cell variable", then say "For example, in CPython, ..." > > From the any()/all() examples, it seems clear that the target scope for > > the assignment has to be referenced from the generator scope (but not > > for other comprehension types, which can simply do one transfer of the > > assigned name after fully evaluating all the contents). > > I don't think that follows.? It _may_ in some cases.? For example, > [SNIP] > _While_ the list comprehension is executing, it needs to rebind f's `i` > on each iteration so that the call to `g()` on each iteration can see > `i`'s then-current value. Good point. My statement above captures this nuance, as far as I'm concerned. (Same for the frame lifetime discussion, which I snipped). Cheers, Steve From breamoreboy at gmail.com Wed Jul 4 14:51:50 2018 From: breamoreboy at gmail.com (Mark Lawrence) Date: Wed, 4 Jul 2018 19:51:50 +0100 Subject: [Python-Dev] Tone it down on Twitter? In-Reply-To: <20180704154821.GA7081@bytereef.org> References: <20180704154821.GA7081@bytereef.org> Message-ID: On 04/07/18 16:48, Stefan Krah wrote: > > Apparently I have made it into "club of three who don't care much about > opinions of others" for the crime of a single +0.5 for PEP-572 without > participating in the discussion at all (neither did Jason). > > https://twitter.com/SerhiyStorchaka/status/1014274554452209665 > > This is a new high for Twitter gossip. Well done. Perhaps in the next vote > the politbureau can indicate the intended outcome beforehand so we know how > to vote. > > > > Thanks, > > Stefan Krah > > > Quite frankly I don't give a hoot what gets said on twitter, as it appears to be reserved for morons like the current president of the USA. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From levkivskyi at gmail.com Wed Jul 4 15:13:58 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Wed, 4 Jul 2018 20:13:58 +0100 Subject: [Python-Dev] Tone it down on Twitter? In-Reply-To: <20180704154821.GA7081@bytereef.org> References: <20180704154821.GA7081@bytereef.org> Message-ID: Just to clarify, the club of three in _my_ twit are the three authors of the PEP. Also, how about you not telling me what to say? -- Ivan On 4 July 2018 at 16:48, Stefan Krah wrote: > > Apparently I have made it into "club of three who don't care much about > opinions of others" for the crime of a single +0.5 for PEP-572 without > participating in the discussion at all (neither did Jason). > > https://twitter.com/SerhiyStorchaka/status/1014274554452209665 > > This is a new high for Twitter gossip. Well done. Perhaps in the next > vote > the politbureau can indicate the intended outcome beforehand so we know how > to vote. > > > > Thanks, > > Stefan Krah > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > levkivskyi%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jul 4 15:18:32 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 05:18:32 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <20180704191832.GZ14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 08:32:32PM +0200, Sven R. Kunze wrote: > >>while total != (total := total + term): > >>??? term *= mx2 / (i*(i+1)) > >>??? i += 2 > >>return total > > This very example here caught my eye. > > Isn't total not always equal to total? What would "regular" Python have > looked like? Read the Appendix to the PEP: https://github.com/python/peps/blob/master/pep-0572.rst And no, total is not always not equal to total. When total and term are sufficiently different, total+term underflows to just total, and the loop exits. py> total = 1.5e30 py> term = 12.5 py> total + term != total False I read it as: while total != updated total: do stuff and find it easier to follow than having to juggle the extra book-keeping "old" variable in the original code. YMMV. -- Steve From tjreedy at udel.edu Wed Jul 4 15:24:08 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 4 Jul 2018 15:24:08 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704133539.GS14437@ando.pearwood.info> References: <20180704133539.GS14437@ando.pearwood.info> Message-ID: On 7/4/2018 9:35 AM, Steven D'Aprano wrote: > On Wed, Jul 04, 2018 at 05:02:07PM +1000, Chris Angelico wrote: >> On Wed, Jul 4, 2018 at 4:07 PM, Serhiy Storchaka wrote: > >> "Assignment is a statement" -- that's exactly the point under discussion. I believe that this is Chris quoting and commenting on Serhiy having said 'assigment is a statement' > Not any more it isn't. We've now gone from discussion to bitter > recriminations *wink* I don't see any recrimination in what either said. >> "del is a statement" -- yes, granted >> >> "function and class declarations are statements" -- class, yes, but >> you have "def" and "lambda" as statement and expression equivalents. > > Even class can be re-written as a call to type(), if you need to. It's > probably not practical to do so in anything but the simplest cases, but > it is there. Serhiy's viewpoint is a legitimate one. A major difference between def, class, and import statements and equivalent lambda, type, and __import__ expressions is that the latter do not name-bind the resulting object. This will continue to be true. There is, however, precedent for subverting "assignment is a statement". >>> class C(): pass >>> c = C() >>> c.a = 1 >>> setattr(c, 'b', 2) >>> c.b 2 However, the target name, when given explicitly, is quoted. This preserves the general rule that non-keyword identifiers in expressions are immediately evaluated. The current exceptions are the 'and', 'or', and 'if-else' constructs that embed flow control in expressions. But if a name in such expressions is not ignored, it is evaluated normally. The purpose of setattr is to allow the target attribute name to be any variable or expression that evaluates to a string, so that one can do the following. >>> d = 'c' >>> setattr(c, d+'2', 3) >>> c.c2 3 or even >>> setattr(c, input('aname: '), 33) aname: qr >>> c.qr 33 (An allowed side-effect is the possibility of adding attributes, such as '2e', that can only be recovered with getattr.) The same comments apply to globals().update({'foo':42}), pointed out by David Merz in another post. (This specific form does not work does not work within functions, as pointed out by Chris Angelico. But others do.) A major difference between a while loop statement and an equivalent tail recursion call (expression) is that while loops do explicit assignment in the current namespace while recursive calls do implicit assignment in a new (and unneeded) execution frame. The targets are the unquoted parameter names in the function header. I regard the the easy use of unevaluated unquoted names as a justification for having statements in addition to expressions. An alternate assignment-within-expressions proposal would be to follow the setattr precedent. Add a builtin function 'set' with paramenters 'name' and 'value'. It would have to be built-in because Python code cannot directly access function local namespaces. The problem, aside from extra typing, is that efficient implementation of functions requires that all local names be known at compile time. But a name added by "set(input('name: '), )" is impossible to know. Making the assignment target be an unquoted unevaluated non-keyword name immediately followed by ':=' solves this. But it either introduces a new and different type of exception to 'names in expressions are evaluated', or one must regard 'assignment expression' as a new statement-expression hybrid. Advocates of assignment expressions should not really be surprised that this disquiets some people. -- Terry Jan Reedy From steve at pearwood.info Wed Jul 4 15:34:11 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 05:34:11 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: <20180704133539.GS14437@ando.pearwood.info> Message-ID: <20180704193411.GA14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 03:24:08PM -0400, Terry Reedy wrote: > On 7/4/2018 9:35 AM, Steven D'Aprano wrote: > >On Wed, Jul 04, 2018 at 05:02:07PM +1000, Chris Angelico wrote: > >>On Wed, Jul 4, 2018 at 4:07 PM, Serhiy Storchaka > >>wrote: > > > >>"Assignment is a statement" -- that's exactly the point under discussion. > > I believe that this is Chris quoting and commenting on Serhiy having > said 'assigment is a statement' > > >Not any more it isn't. We've now gone from discussion to bitter > >recriminations *wink* > > I don't see any recrimination in what either said. Haven't you been reading the rest of the thread? The Twitter storm? The ranting on Reddit that this is the end of the world and Python is doomed? I wasn't referring specifically to Chris or Serhiy's comments, but about the general over-reaction, here and elsewhere. -- Steve From jeanpierreda at gmail.com Wed Jul 4 16:00:41 2018 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Wed, 4 Jul 2018 13:00:41 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704180215.GW14437@ando.pearwood.info> References: <20180704133125.GR14437@ando.pearwood.info> <20180704180215.GW14437@ando.pearwood.info> Message-ID: On Wed, Jul 4, 2018 at 11:04 AM Steven D'Aprano wrote: > Did you actually mean arbitrary simple statements? > > if import math; mylist.sort(); print("WTF am I reading?"); True: > pass Yes. To quote PEP 572: "This is a tool, and it is up to the programmer to use it where it makes sense, and not use it where superior constructs can be used." > A more reasonable suggestion would be to only allow *assignments*, not > arbitrary simple statements. But that's another special case: I don't agree that it is more reasonable, for exactly the reasons you describe it to be surprising. > with one or more semicolon-separated statements between the "if" and the > condition: > > if statement; statement; condition: > > If we stick to the rule that semicolons separate statements, that means > we have: > > > if statement # SyntaxError > statement # okay > condition: # SyntaxError > > > If we don't want that, we need a new rule to treat semicolons > differently inside if statements that they're treated elsewhere. Yes. This is analogous to complaining that [1, 2, 3] should be a syntax error because clearly this is a tuple with three elements: "[1", "2", and "3]". In as far as it's a new parsing rule, it is a "special case" indeed. > If we applied this rule "allow statements separated by semicolons" > everywhere, we'd get this: [snip] Nobody said anything about allowing semicolon-delimited statements in arbitrary places in the grammar. -- Devin From srkunze at mail.de Wed Jul 4 16:07:44 2018 From: srkunze at mail.de (Sven R. Kunze) Date: Wed, 4 Jul 2018 22:07:44 +0200 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704191832.GZ14437@ando.pearwood.info> References: <20180704191832.GZ14437@ando.pearwood.info> Message-ID: <1f6a9bc8-176a-fe64-ae8a-e2b8bafdfd14@mail.de> On 04.07.2018 21:18, Steven D'Aprano wrote: > Read the Appendix to the PEP: > https://github.com/python/peps/blob/master/pep-0572.rst Yes, I did after I realized where that example came from. But my point was actually to understand the evaluation order because Uncle Timmy won't be around to explain when new code appears. > And no, total is not always not equal to total. Err, yeah. Double negation rules. ;-) > I read it as: > > while total != updated total: > do stuff > > and find it easier to follow than having to juggle the extra > book-keeping "old" variable in the original code. updated_total rocks. Reminds me of those pattern, we usually use in those cases. What just confused me is the evaluation order. It seems to me that it's like left-to-right first and then assignment expression. Using some math-style-inspired markers (execution-irrelevant) would be cool: while total != (total' := total + term): ??? do stuff total and total' can be different at the same whereas total is total (at least in my mental model). But it seems I need to adapt here. Regards, Sven From tim.peters at gmail.com Wed Jul 4 18:18:19 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 4 Jul 2018 17:18:19 -0500 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> Message-ID: [Steve Dower] > Okay, so as far as the specification goes, saying "assignment > > expressions in comprehensions get or create a cell variable in the > > defining scope and update its value" satisfies me just fine (or some > > other wording that more closely mirrors the actual behaviour - all my > > work here is on my own compiler, not the actual CPython one, and I don't > > know that they're identical). > > > > I don't think this should be left assumed by the PEP. If it's likely to > > be a restriction on other implementations to say "cell variable", then > > say "For example, in CPython, ..." The problem: nothing about Python's scoping is changed by this PEP, so saying anything related to the implementation of scoping risks giving a misleading impression that something about scoping _is_ being changed. The PEP certainly needs to define the intended scope of assignment-expression targets in all cases. It already tried, with perhaps too few words. But it strikes me as counterproductive to go on to say anything about how scopes are implemented. If, e.g., the PEP is clear that in def f(): return (a := i for i in range(5)) `a` is local to `f` and `i` is local to a directly nested scope, what's left to the imagination? Well, the implementations of scopes and closures. None of which the PEP changes. The semantics of `i` haven't changed from what we already have, neither the visibility nor extent of the generator expression object returned. The only new thing is specifying the scope of `a`, where "local to f" means exactly the same thing as for any other name local to a function today. So far as the PEP semantics go, it doesn't even matter whether an implementation _does_ implement some form of closure as such. It just has to provide the visible semantics of *lexically* nested scopes with indefinite extent, by whatever means it likes best. That's what "local to f" means (and has meant all along - well, since lexically nested scopes were first introduced). Would it help to say "and by 'local to f' we mean the same thing we meant before this PEP was written"? Probably not ;-) If someone doesn't know what "name N belongs to scope S" means, that needs to get a clear answer, but it's a fundamental question that's quite independent of this particular PEP. That said, I bet it would be helpful to give some examples that show how CPython intends to implement this stuff, via giving explicit nested workalike Python functions decorated with scope declarations. The concreteness of that is helpful. The eternal problem with that is finding a way to stop readers from taking the "workalike functions" too literally (e.g., "but i tried it and it doesn't generate exactly the same byte code!"). -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Wed Jul 4 18:51:37 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 00:51:37 +0200 Subject: [Python-Dev] Assignment expression and coding style: the while True case Message-ID: Hi, Let's say that the PEP 572 (assignment expression) is going to be approved. Let's move on and see how it can be used in the Python stdlib. I propose to start the discussion about "coding style" (where are assignment expressions appropriate or not?) with the "while True" case. I wrote a WIP pull request to use assignment expressions in "while True": https://github.com/python/cpython/pull/8095/files In short, replace: while True: x = expr if not x: break ... with: while (x := expr): ... My question is now: for which "while True" patterns are the assignment expression appropriate? There identified different patterns. == Pattern 1, straighforward == while True: line = input.readline() if not line: break ... IMHO here assingment expression is appropriate here. The code remains straighfoward to read. while (line := input.readline()): ... == Pattern 2, condition == Condition more complex than just "not line": while True: q = c//n if n <= q: break ... replaced with: while (q := c//n) < n: ... IMHO it's still acceptable to use assignement expression... Maybe only for basic conditions? (see above) == Pattern 3, double condition == while True: s = self.__read(1) if not s or s == NUL: break .... replaced with: while (s := self.__read(1)) and s != NUL: ... Honestly, here, I don't know if it's appropriate... At the first look, "s != NUL" is surprising, since "s" is not defined before the while, it's only defined in the first *test* (defining a variable inside a test is *currently* uncommon in Python). == Pattern 4, while (...): pass == Sometimes, the loop body is replaced by "pass". while True: tarinfo = self.next() if tarinfo is None: break replaced with: while (tarinfo := self.next()) is not None: pass It reminds me the *surprising* "while (func());" or "while (func()) {}" in C (sorry for theorical C example, I'm talking about C loops with an empty body). Maybe it's acceptable here, I'm not sure. Note: such loop is rare (see my PR). == Pattern 5, two variables == while True: m = match() if not m: break j = m.end() if i == j: break ... replaced with: while (m := match()) and (j := m.end()) == i: ... Maybe we reached here the maximum acceptable complexity of a single Python line? :-) == Other cases == I chose to not use assignment expressions for the following while loops. (A) while True: name, token = _getname(g) if not name: break ... "x, y := ..." is invalid. It can be tricked using "while (x_y := ...)[0]: x, y = x_y; ...". IMHO it's not worth it. (B) while True: coeff = _dlog10(c, e, places) # assert len(str(abs(coeff)))-p >= 1 if coeff % (5*10**(len(str(abs(coeff)))-p-1)): break places += 3 NOT replaced with: while not (coeff := _dlog10(c, e, places)) % (5*10**(len(str(abs(coeff)))-p-1)): places += 3 ^-- Tim Peters, I'm looking at you :-) coeff is defined and then "immediately" used in "y" expression of x%y... Yeah, it's valid code, but it looks too magic to me... (C) while True: chunk = self.raw.read() if chunk in empty_values: nodata_val = chunk break ... "nodata_val = chunk" cannot be put into the "chunk := self.raw.read()" assignment expression combined with a test. At least, I don't see how. (D) while 1: u1 = random() if not 1e-7 < u1 < .9999999: continue ... Again, I don't see how to use assignment expression here. Victor From vano at mail.mipt.ru Wed Jul 4 19:10:46 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 02:10:46 +0300 Subject: [Python-Dev] Don't assign to a variable used later in the expression In-Reply-To: References: Message-ID: <77d12d3f-6220-c644-f481-d08c1a0e8c93@mail.mipt.ru> On 04.07.2018 10:10, Nathaniel Smith wrote: > On Tue, Jul 3, 2018 at 11:07 PM, Serhiy Storchaka wrote: >> 04.07.18 00:51, Chris Angelico ????: >>> On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka >>> wrote: >>>> I believe most Python users are not >>>> professional programmers -- they are sysadmins, scientists, hobbyists and >>>> kids -- >>> [citation needed] >> I don't understand what citation do you need. >> >>>> In particularly mutating and >>>> non-mutating operations are separated. The assignment expression breaks >>>> this. >>> [citation needed] >> In Python the assignment (including the augmented assignment) is a >> statement, del is a statement, function and class declarations are >> statements, import is a statement. Mutating methods like list.sort() and >> dict.update() return None to discourage using them in expressions. This a >> common knowledge, I don't know who's citation you need. > Right, Python has a *very strong* convention that each line should > have at most one side-effect, and that if it does have a side-effect > it should be at the outermost level. > > I think the most striking evidence for this is that during the > discussion of PEP 572 we discovered that literally none of us ? > including Guido ? even *know* what the order-of-evaluation is inside > expressions. In fact PEP 572 now has a whole section talking about the > oddities that have turned up here so far, and how to fix them. Which > just goes to show that even its proponents don't actually think that > anyone uses side-effects inside expressions, because if they did, then > they'd consider these changes to be compatibility-breaking changes. Of > course the whole point of PEP 572 is to encourage people to embed > side-effects inside expressions, so I hope they've caught all the > weird cases, because even if we can still change them now we won't be > able to after PEP 572 is implemented. I may have a fix to this: Do not recommend assigning to the variable that is used later in the expression. And to facilitate that, do not make any strong guarantees about evaluation order -- making any such attempt a gamble. I immediately saw those "total := total + item" as odd but couldn't quite point out why. Now I see: it ignores the whole augmented assignment machinery thing, which will make people demand that next. Making this a discouraged case will diminish valid use cases and lower the need for that. > Some people make fun of Python's expression/statement dichotomy, > because hey don't you know that everything can be an expression, > functional languages are awesome hurhur, but I think Python's approach > is actually very elegant. Python is unapologetically an imperative > language, but even we dirty imperative programmers can agree with the > functional fanatics that reasoning about side-effects and sequencing > is hard. One-side-effect-per-line is a very elegant way to keep > sequencing visible on the page and as easy to reason about as > possible. > > Or as Dijkstra put it: "our intellectual powers are rather geared to > master static relations and that our powers to visualize processes > evolving in time are relatively poorly developed. For that reason we > should do (as wise programmers aware of our limitations) our utmost to > shorten the conceptual gap between the static program and the dynamic > process, to make the correspondence between the program (spread out in > text space) and the process (spread out in time) as trivial as > possible." > > It's very disheartening that not only is PEP 572 apparently going to > be accepted, but as far as I can tell neither the text nor its > proponents have even addressed this basic issue. > > -n > -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Jul 4 19:15:41 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 4 Jul 2018 16:15:41 -0700 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 3:51 PM, Victor Stinner wrote: > My question is now: for which "while True" patterns are the assignment > expression appropriate? There identified different patterns. > > > == Pattern 1, straighforward == > > while True: > line = input.readline() > if not line: > break > ... > > IMHO here assingment expression is appropriate here. The code remains > straighfoward to read. > > while (line := input.readline()): > ... There are some obvious caveats here ? no-one has real experience with := yet, so any opinions right now are informed guesswork that will probably change some if/when we get more experience with it. Also, the stdlib is a big place, and it seems inevitable that maintainers of different modules will have different preferences. So I don't think it makes sense to try to define some Official Universal Rule about when := is appropriate and when it isn't. That said, FWIW, my current feeling is that this simplest case is the only one where I would use :=; for your other examples I'd stick with the loop-and-a-half style. -n -- Nathaniel J. Smith -- https://vorpus.org From njs at pobox.com Wed Jul 4 19:29:12 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 4 Jul 2018 16:29:12 -0700 Subject: [Python-Dev] Don't assign to a variable used later in the expression In-Reply-To: <77d12d3f-6220-c644-f481-d08c1a0e8c93@mail.mipt.ru> References: <77d12d3f-6220-c644-f481-d08c1a0e8c93@mail.mipt.ru> Message-ID: On Wed, Jul 4, 2018 at 4:10 PM, Ivan Pozdeev via Python-Dev wrote: > On 04.07.2018 10:10, Nathaniel Smith wrote: >> Right, Python has a *very strong* convention that each line should >> have at most one side-effect, and that if it does have a side-effect >> it should be at the outermost level. > >> I think the most striking evidence for this is that during the >> discussion of PEP 572 we discovered that literally none of us ? >> including Guido ? even *know* what the order-of-evaluation is inside >> expressions. In fact PEP 572 now has a whole section talking about the >> oddities that have turned up here so far, and how to fix them. Which >> just goes to show that even its proponents don't actually think that >> anyone uses side-effects inside expressions, because if they did, then >> they'd consider these changes to be compatibility-breaking changes. Of >> course the whole point of PEP 572 is to encourage people to embed >> side-effects inside expressions, so I hope they've caught all the >> weird cases, because even if we can still change them now we won't be >> able to after PEP 572 is implemented. > > I may have a fix to this: > > Do not recommend assigning to the variable that is used later in the > expression. This would rule out all the comprehension use cases. I'd be fine with that personally. Reading through the PEP again I see that there are more examples of them than I previously realized, inside the semantics discussion and... well, this may be a personal thing but for me they'd all be better described as "incomprehensions". But, nonetheless, the comprehension use cases are supposed to be a core motivation for the whole PEP. Also, some of the main arguments for why a full-fledged := is better than the more limited alternative proposals rely on using a variable on the same line where it's assigned (e.g. Tim's gcd example). So I don't see this recommendation getting any official traction within PEP 572 or PEP 8. Of course you're free to use whatever style rules you prefer locally ? python-dev has nothing to do with that. -n -- Nathaniel J. Smith -- https://vorpus.org From python at mrabarnett.plus.com Wed Jul 4 19:49:52 2018 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 5 Jul 2018 00:49:52 +0100 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: On 2018-07-04 23:51, Victor Stinner wrote: [snip] > (C) > > while True: > chunk = self.raw.read() > if chunk in empty_values: > nodata_val = chunk > break > ... > > "nodata_val = chunk" cannot be put into the "chunk := self.raw.read()" > assignment expression combined with a test. At least, I don't see how. > If that's the only 'break' in the loop, then you know that 'chunk' will have an 'empty' value after the loop, so you can change it to: while True: chunk = self.raw.read() if chunk in empty_values: break ... nodata_val = chunk which then leads to: while (chunk := self.raw.read()) not in empty_values: ... nodata_val = chunk [snip] From vstinner at redhat.com Wed Jul 4 20:03:23 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 02:03:23 +0200 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: On the 3360 for loops of the stdlib (*), I only found 2 loops which would benefit of assignment expressions. It's not easy to find loops which: - build a list, - are simple enough to be expressed as list comprehension, - use a condition (if), - use an expression different than just a variable name as the list value (value appended to the list). diff --git a/Lib/mailbox.py b/Lib/mailbox.py index 056251dce0..dc61a3a8b6 100644 --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -1341,9 +1341,9 @@ class Babyl(_singlefileMailbox): if len(stops) < len(starts): stops.append(line_pos - len(linesep)) starts.append(next_pos) - labels = [label.strip() for label - in self._file.readline()[1:].split(b',') - if label.strip()] + labels = [slabel for label + in self._file.readline()[1:].split(b',') + if (slabel := label.strip())] label_lists.append(labels) elif line == b'\037' or line == b'\037' + linesep: if len(stops) < len(starts): diff --git a/Lib/nntplib.py b/Lib/nntplib.py index 5961a28ab7..a5d13e35be 100644 --- a/Lib/nntplib.py +++ b/Lib/nntplib.py @@ -843,11 +843,9 @@ class _NNTPBase: DeprecationWarning, 2) line_pat = re.compile('^([^ \t]+)[ \t]+(.*)$') resp, raw_lines = self._longcmdstring('XGTITLE ' + group, file) - lines = [] - for raw_line in raw_lines: - match = line_pat.search(raw_line.strip()) - if match: - lines.append(match.group(1, 2)) + lines = [match.group(1, 2) + for raw_line in raw_lines + if (match := line_pat.search(raw_line.strip()))] return resp, lines def xpath(self, id): (*) Command used to count the number of for loops in the stdlib: $ grep '\bfor\b' Lib/*py Lib/{asyncio,logging,multiprocessing}/*.py|grep -v '"""'|grep -v "'''"|grep -v '\.py: *#'|wc -l 3360 Victor From python-dev at mgmiller.net Wed Jul 4 19:52:54 2018 From: python-dev at mgmiller.net (Mike Miller) Date: Wed, 4 Jul 2018 16:52:54 -0700 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" Message-ID: Recently on Python-Dev: On 2018-07-03 15:24, Chris Barker wrote: > On Tue, Jul 3, 2018 at 2:51 PM, Chris Angelico On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka > > > I believe most Python users are not > > professional programmers -- they are sysadmins, scientists, hobbyists > > and kids -- > > [citation needed] > > fair enough, but I think we all agree that *many*, if not most, Python users > are "not professional programmers". While on the other hand everyone involved > in discussion on python-dev and python-ideas is a serious (If not > "professional") programmer. Python Audience - wants clarity: Not sure I'd say that most users are not professionals, but one major strength of Python is its suitability as a teaching language, which enlarges the community every year. Additionally, I have noticed a dichotomy between prolific "C programmers" who've supported this PEP and many Python programmers who don't want it. While C-devs use this construct all the time, their stereotypical Python counterpart is often looking for simplicity and clarity instead. That's why we're here, folks. Value - good: Several use cases are handled well by PEP 572. However it has been noted that complexity must be capped voluntarily relatively early?or the cure soon becomes worse than the disease. Frequency - not much: The use cases for assignment-expressions are not exceedingly common, coming up here and there. Their omission has been a very mild burden and we've done without for a quarter century. Believe the authors agreed that it won't be used too often and won't typically be mis- or overused. New Syntax - a high burden: For years I've read on these lists that syntax changes must clear a high threshold of the (Value*Frequency)/Burden (or VF/B) ratio. Likewise, a few folks have compared PEP 572 to 498 (f-strings) which some former detractors have come to appreciate. Don't believe this comparison applies well, since string interpolation is useful a hundred times a day, more concise, clear, and runs faster than previous functionality. Threshold was easily cleared there. Conclusion: An incongruous/partially redundant new syntax to perform existing functionality more concisely feels too low on the VF/B ratio IMHO. Value is good though mixed, frequency is low, and burden is higher than we'd like, resulting in "meh" and binary reactions. Indeed many modern languages omit this feature specifically in an effort to reduce complexity, ironically citing the success of Python in support. Less is more. Compromise: Fortunately there is a compromise design that is chosen often these days in new languages---restricting these assignments to if/while (potentially comp/gen) statements. We can also reuse the existing "EXPR as NAME" syntax that already exists and is widely enjoyed. This compromise design: 1 Handles the most common cases (of a group of infrequent cases) 0 Doesn't handle more obscure cases. 1 No new syntax (through reuse) 1 Looks Pythonic as hell 1 Difficult to misuse, complexity capped Score: 4/5 PEP 572: 1 Handles the most common cases (of a group of infrequent cases) 1 Handles even more obscure cases. 0 New syntax 0 Denser look: more colons, parens, expression last 0 Some potential for misuse, complexity uncapped Score: 2/5 Thanks for reading, happy independence, -Mike From vstinner at redhat.com Wed Jul 4 20:11:43 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 02:11:43 +0200 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: The code comes from Lib/_pyio.py. Simplified code: --- nodata_val = b"" ... if n is None or n == -1: ... current_size = 0 while True: chunk = self.raw.read() if chunk in empty_values: nodata_val = chunk break current_size += len(chunk) chunks.append(chunk) return b"".join(chunks) or nodata_val ... while avail < n: chunk = self.raw.read(wanted) if chunk in empty_values: nodata_val = chunk break avail += len(chunk) chunks.append(chunk) ... return out[:n] if out else nodata_val --- It seems like "nodata_val = " assignment can be moved out of the first loop, but cannot be moved for the second loop (since the second loop has no iteration if "avail >= n"). Yeah, maybe for this specific file, assignment expressions could be used for the (C) case and would be worth it. Victor 2018-07-05 1:49 GMT+02:00 MRAB : > On 2018-07-04 23:51, Victor Stinner wrote: > [snip] >> >> (C) >> >> while True: >> chunk = self.raw.read() >> if chunk in empty_values: >> nodata_val = chunk >> break >> ... >> >> "nodata_val = chunk" cannot be put into the "chunk := self.raw.read()" >> assignment expression combined with a test. At least, I don't see how. >> > If that's the only 'break' in the loop, then you know that 'chunk' will have > an 'empty' value after the loop, so you can change it to: > > while True: > chunk = self.raw.read() > if chunk in empty_values: > break > ... > nodata_val = chunk > > which then leads to: > > while (chunk := self.raw.read()) not in empty_values: > ... > nodata_val = chunk > > [snip] > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com From rosuav at gmail.com Wed Jul 4 20:15:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jul 2018 10:15:02 +1000 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 10:03 AM, Victor Stinner wrote: > On the 3360 for loops of the stdlib (*), I only found 2 loops which > would benefit of assignment expressions. > > It's not easy to find loops which: > - build a list, > - are simple enough to be expressed as list comprehension, > - use a condition (if), > - use an expression different than just a variable name as the list > value (value appended to the list). Are you implying that the above conditions are essential for assignment expressions to be useful, or that this defines one particular way in which they can be of value? Your opening implies the former, but I think the latter is more accurate here. ChrisA From steve.dower at python.org Wed Jul 4 20:20:46 2018 From: steve.dower at python.org (Steve Dower) Date: Wed, 4 Jul 2018 17:20:46 -0700 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> Message-ID: <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> On 04Jul2018 1518, Tim Peters wrote: > The only new thing is specifying the scope of `a`, where "local to f" > means exactly the same thing as for any other name local to a function > today.? So far as the PEP semantics go, it doesn't even matter whether > an implementation _does_ implement some form of closure as such.? It > just has to provide the visible semantics of?_lexically_ nested scopes > with indefinite extent, by whatever means it likes best.? That's what > "local to f" means (and has meant all along - well, since lexically > nested scopes were first introduced). In that case, please provide more examples of how it should work when the assignment expression appears to define a variable in a scope that is not on the call stack. Whether intentional or not, there will be changes to how and when names are resolved. The specification should provide enough information to determine the preferred behaviour, so we can tell the difference between intention changes and implementation bugs. For example, what should be returned from this function? >>> A = 0 >>> def f(x): ... if x: ... [A := i for i in [1]] ... return A As far as I can tell, the closest current equivalent will not compile: >>> A = 0 >>> def f(x): ... if x: ... def g(): ... nonlocal A ... A = 1 ... g() ... return A ... File "", line 4 SyntaxError: no binding for nonlocal 'A' found Is this the equivalent behaviour you want? Or do you want an UnboundLocalError when calling f(0)? Or do you want the global A to be returned? How should we approach decision making about these cases as we implement this? The PEP does not provide enough information for me to choose the right behaviour here, and I argue that it should. Cheers, Steve From rosuav at gmail.com Wed Jul 4 20:22:54 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jul 2018 10:22:54 +1000 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 9:52 AM, Mike Miller wrote: > Compromise: > > Fortunately there is a compromise design that is chosen often these days in > new languages---restricting these assignments to if/while (potentially > comp/gen) statements. We can also reuse the existing "EXPR as NAME" syntax > that already exists and is widely enjoyed. > > This compromise design: > > 1 Handles the most common cases (of a group of infrequent cases) > 0 Doesn't handle more obscure cases. > 1 No new syntax (through reuse) > 1 Looks Pythonic as hell > 1 Difficult to misuse, complexity capped > > Score: 4/5 PLEASE can people stop rehashing this one and go and read previous discussions and the PEP? Your first point is a failure, not a success - the "if expr as name:" syntax is able to handle only the tiniest proportion of cases, because many MANY situations require a condition after that. You can't write this, for instance: if f(x) as spam < 0: print(spam) So it fails the first, fails the second, is identical on the third (it's still new syntax, just reusing a keyword - that's no better than ":="), is dubious on the fourth (Python uses "as NAME" for things that are quite different from this, so it's confusing), and scores a definite win only on the fifth - it's hard to misuse because it's nerfed to oblivion. This is old news. It's not a compromise design - it's a failed previous iteration that is already mentioned in the PEP. ChrisA From rosuav at gmail.com Wed Jul 4 20:25:17 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jul 2018 10:25:17 +1000 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> Message-ID: On Thu, Jul 5, 2018 at 10:20 AM, Steve Dower wrote: > On 04Jul2018 1518, Tim Peters wrote: >> The only new thing is specifying the scope of `a`, where "local to f" >> means exactly the same thing as for any other name local to a function >> today. So far as the PEP semantics go, it doesn't even matter whether >> an implementation _does_ implement some form of closure as such. It >> just has to provide the visible semantics of _lexically_ nested scopes >> with indefinite extent, by whatever means it likes best. That's what >> "local to f" means (and has meant all along - well, since lexically >> nested scopes were first introduced). > > In that case, please provide more examples of how it should work when > the assignment expression appears to define a variable in a scope that > is not on the call stack. Using an assignment expression in a comprehension is the same as using an assignment expression outside a comprehension at the exact same point in the code. I'm not sure what else needs to be explained further. It's assignment, so it creates a name at that scope. > For example, what should be returned from this function? > > >>> A = 0 > >>> def f(x): > ... if x: > ... [A := i for i in [1]] > ... return A A would be local to f(). This will either return 1 or raise UnboundLocalError; it will never return 0. ChrisA From tim.peters at gmail.com Wed Jul 4 20:25:54 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 4 Jul 2018 19:25:54 -0500 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: [Victor Stinner]Let's say that the PEP 572 (assignment expression) is going to be > approved. Let's move on and see how it can be used in the Python stdlib. > Ugh - how adult ;-) > I propose to start the discussion about "coding style" (where are > assignment expressions appropriate or not?) with the "while True" > case. > > I wrote a WIP pull request to use assignment expressions in "while True": > https://github.com/python/cpython/pull/8095/files > > In short, replace: > > while True: > x = expr > if not x: > break > ... > with: > > while (x := expr): Better is to translate it to: while x := expr: That is, ;parentheses aren't needed in this context, and adding them anyway will quickly look as strange here as, e.g., return (result) already looks. (Always requiring parens was rejected - see the PEP's "Always requiring parentheses " section). ... > ... > == Pattern 3, double condition == > > while True: > s = self.__read(1) > if not s or s == NUL: > break > .... > > replaced with: > > while (s := self.__read(1)) and s != NUL: > ... > Honestly, here, I don't know if it's appropriate... > Then leave it be! My rule was "if it's not obviously better - at least a little - don't use it". This one is a wash (tie) to me, so I'd save the bother of changing it. Or just do the obvious part: while s := self.__read(1): if s == NUL: break No matter how the code may change in the future, the loop body surely requires a non-empty `s` to stare at. and now the `while` header makes that abundantly clear at a glance. ... > == Pattern 4, while (...): pass == > > Sometimes, the loop body is replaced by "pass". > > while True: > tarinfo = self.next() > if tarinfo is None: > break > > replaced with: > > while (tarinfo := self.next()) is not None: > pass > > It reminds me the *surprising* "while (func());" or "while (func()) > {}" in C (sorry for theorical C example, I'm talking about C loops > with an empty body). > > Maybe it's acceptable here, I'm not sure. > > Note: such loop is rare (see my PR). > I decided "slight loss - don't bother" for most such in my own code. At least the first spelling above cuts the number of statements in half. Replacing random.py's r = getrandbits(k) while r >= n: r = getrandbits(k) with while (r := getrandbits(k)) >= n: pass is more attractive, for eliminating a textually identical (except for indentation) line. == Pattern 5, two variables == > > while True: > m = match() > if not m: > break > j = m.end() > if i == j: > break > ... > > replaced with: > > while (m := match()) and (j := m.end()) == i: > ... > > Maybe we reached here the maximum acceptable complexity of a single > Python line? :-) > It's at my limit. But, as in an earlier example, I'd be tempted to do "the obvious part": while m:= match(): j = m.end() if i == j:: break Then the start reads like "while there's something _to_ look at::" and the body of the loop is happily guaranteed that there is. ... > I chose to not use assignment expressions for the following while loops. > > (A) > > while True: > name, token = _getname(g) > if not name: > break > ... > > "x, y := ..." is invalid. It can be tricked using "while (x_y :=...)[0]: > x, y = x_y; ...". IMHO it's not worth it. Indeed, it's quite worth _not_ doing it :-) > (B) > > while True: > coeff = _dlog10(c, e, places) > # assert len(str(abs(coeff)))-p >= 1 > if coeff % (5*10**(len(str(abs(coeff)))-p-1)): > break > places += 3 > > NOT replaced with: > > while not (coeff := _dlog10(c, e, places)) % > (5*10**(len(str(abs(coeff)))-p-1)): > places += 3 > > ^-- Tim Peters, I'm looking at you :-) > Not my code ;-) - and it's _already_ too "busy" to be my code. The `5*10**...` part is already crying to be broken into simpler pieces with a comment explaining what the intent is. > coeff is defined and then "immediately" used in "y" expression of > x%y... Yeah, it's valid code, but it looks too magic to me... > And the code was already too dense to follow easily. (C) > > while True: > chunk = self.raw.read() > if chunk in empty_values: > nodata_val = chunk > break > ... > > "nodata_val = chunk" cannot be put into the "chunk := self.raw.read()" > assignment expression combined with a test. At least, I don't see how. > > No need to strain, either! If it's not obvious, don't bother. > (D) > > while 1: > u1 = random() > if not 1e-7 < u1 < .9999999: > continue > ... > > Again, I don't see how to use assignment expression here. > > It could be, in context, but not for the outermost `while 1:`. while 1: while not 1e-7 < (u1 := random()) < 9999999: pass # code that uses u1, and possibly returns, else goes # around the outer loop again That one is fine by me either way. In all, I'd say our tastes here are pretty similar! So there's hope ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Wed Jul 4 20:32:03 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 03:32:03 +0300 Subject: [Python-Dev] "as" postfix notation highlights In-Reply-To: References: Message-ID: <01313d97-31f7-6a20-18f7-c0d5c2ebea66@mail.mipt.ru> On 04.07.2018 10:10, Nathaniel Smith wrote: > > I think the most striking evidence for this is that during the > discussion of PEP 572 we discovered that literally none of us ? > including Guido ? even *know* what the order-of-evaluation is inside > expressions. If has stricken me that this is a highlight for the "as" syntax right here! * Since it's a postfix, it preserves the forward reading order and is thus more fitting for inline syntax: ??? while inv == (make_step(state, inv) as new_inv) and is_valid(new_inv): inv = new_inv Makes it clear that in the first two cases, the old value of `inv' is used, and only after that, it's reassigned. The prefix syntax of an assignment is instead read "from `=' on, then return to the start". This is okay for a standalone construct, but if embedded, the reading order becomes nontrivial: ??? while inv == (new_inv := make_step(state, inv)) and is_valid(new_inv): inv = new_inv * In the light of "Don't assign to a variable used later in the expression" , "as" looks completely different from assignment, which will deter folks from trying to do the problematic augmented assignments and demand expression syntax for them. > In fact PEP 572 now has a whole section talking about the > oddities that have turned up here so far, and how to fix them. Which > just goes to show that even its proponents don't actually think that > anyone uses side-effects inside expressions, because if they did, then > they'd consider these changes to be compatibility-breaking changes. Of > course the whole point of PEP 572 is to encourage people to embed > side-effects inside expressions, so I hope they've caught all the > weird cases, because even if we can still change them now we won't be > able to after PEP 572 is implemented. > > Some people make fun of Python's expression/statement dichotomy, > because hey don't you know that everything can be an expression, > functional languages are awesome hurhur, but I think Python's approach > is actually very elegant. Python is unapologetically an imperative > language, but even we dirty imperative programmers can agree with the > functional fanatics that reasoning about side-effects and sequencing > is hard. One-side-effect-per-line is a very elegant way to keep > sequencing visible on the page and as easy to reason about as > possible. > > Or as Dijkstra put it: "our intellectual powers are rather geared to > master static relations and that our powers to visualize processes > evolving in time are relatively poorly developed. For that reason we > should do (as wise programmers aware of our limitations) our utmost to > shorten the conceptual gap between the static program and the dynamic > process, to make the correspondence between the program (spread out in > text space) and the process (spread out in time) as trivial as > possible." > > It's very disheartening that not only is PEP 572 apparently going to > be accepted, but as far as I can tell neither the text nor its > proponents have even addressed this basic issue. > > -n > -- Regards, Ivan From vstinner at redhat.com Wed Jul 4 20:33:24 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 02:33:24 +0200 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: 2018-07-05 2:15 GMT+02:00 Chris Angelico : > On Thu, Jul 5, 2018 at 10:03 AM, Victor Stinner wrote: >> On the 3360 for loops of the stdlib (*), I only found 2 loops which >> would benefit of assignment expressions. >> >> It's not easy to find loops which: >> - build a list, >> - are simple enough to be expressed as list comprehension, >> - use a condition (if), >> - use an expression different than just a variable name as the list >> value (value appended to the list). > > Are you implying that the above conditions are essential for > assignment expressions to be useful, or that this defines one > particular way in which they can be of value? Hum, maybe I wasn't specific enough. I'm looking for "for loops" and list comprehensions in stdlib which *can be* written using assignment expression, like: [var for ... in ... if (var := expr)]. I'm not discussing if such change is worth it or not. I just counted how many for loops/list comprehensions *can* be modified to use assingment expressions in the stdlib. Maybe I missed some loops/comprehensions, and I would be happy to see more examples ;-) Victor From rosuav at gmail.com Wed Jul 4 20:36:38 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jul 2018 10:36:38 +1000 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 10:33 AM, Victor Stinner wrote: > 2018-07-05 2:15 GMT+02:00 Chris Angelico : >> On Thu, Jul 5, 2018 at 10:03 AM, Victor Stinner wrote: >>> On the 3360 for loops of the stdlib (*), I only found 2 loops which >>> would benefit of assignment expressions. >>> >>> It's not easy to find loops which: >>> - build a list, >>> - are simple enough to be expressed as list comprehension, >>> - use a condition (if), >>> - use an expression different than just a variable name as the list >>> value (value appended to the list). >> >> Are you implying that the above conditions are essential for >> assignment expressions to be useful, or that this defines one >> particular way in which they can be of value? > > Hum, maybe I wasn't specific enough. I'm looking for "for loops" and > list comprehensions in stdlib which *can be* written using assignment > expression, like: > > [var for ... in ... if (var := expr)]. > > I'm not discussing if such change is worth it or not. I just counted > how many for loops/list comprehensions *can* be modified to use > assingment expressions in the stdlib. > > Maybe I missed some loops/comprehensions, and I would be happy to see > more examples ;-) Cool. So you're looking for ones that fit a particular pattern that can benefit, but there are almost certainly other patterns out there. Just making sure that you weren't trying to say "out of 3360 loops, exactly 3358 of them absolutely definitely cannot be improved here". ChrisA From vano at mail.mipt.ru Wed Jul 4 20:40:36 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 03:40:36 +0300 Subject: [Python-Dev] Don't assign to a variable used later in the expression In-Reply-To: References: <77d12d3f-6220-c644-f481-d08c1a0e8c93@mail.mipt.ru> Message-ID: <855a51ea-2d09-7d3c-51c9-5a5200137971@mail.mipt.ru> On 05.07.2018 2:29, Nathaniel Smith wrote: > On Wed, Jul 4, 2018 at 4:10 PM, Ivan Pozdeev via Python-Dev > wrote: >> On 04.07.2018 10:10, Nathaniel Smith wrote: >>> Right, Python has a *very strong* convention that each line should >>> have at most one side-effect, and that if it does have a side-effect >>> it should be at the outermost level. >>> I think the most striking evidence for this is that during the >>> discussion of PEP 572 we discovered that literally none of us ? >>> including Guido ? even *know* what the order-of-evaluation is inside >>> expressions. In fact PEP 572 now has a whole section talking about the >>> oddities that have turned up here so far, and how to fix them. Which >>> just goes to show that even its proponents don't actually think that >>> anyone uses side-effects inside expressions, because if they did, then >>> they'd consider these changes to be compatibility-breaking changes. Of >>> course the whole point of PEP 572 is to encourage people to embed >>> side-effects inside expressions, so I hope they've caught all the >>> weird cases, because even if we can still change them now we won't be >>> able to after PEP 572 is implemented. >> I may have a fix to this: >> >> Do not recommend assigning to the variable that is used later in the >> expression. > This would rule out all the comprehension use cases. Only those outside of the outermost iterable. > I'd be fine with that personally. Reading through the PEP again I see > that there are more examples of them than I previously realized, > inside the semantics discussion and... well, this may be a personal > thing but for me they'd all be better described as "incomprehensions". > But, nonetheless, the comprehension use cases are supposed to be a > core motivation for the whole PEP. Far from it. If/while, too. Any construct that accepts an expression and uses its result but doesn't allow to insert an additional line in the middle qualifies. > Also, some of the main arguments > for why a full-fledged := is better than the more limited alternative > proposals rely on using a variable on the same line where it's > assigned (e.g. Tim's gcd example). So I don't see this recommendation > getting any official traction within PEP 572 or PEP 8. That's actually a valid use case! In the aforementioned example, if (diff := x - x_base) and (g := gcd(diff, n)) > 1: return g ?the variable `diff' doesn't exist before, so there's no confusion which value is used. Okay, I stay corrected: _Do not recommend *changing* a variable that is used later in the expression._ I.e. the variable should not exist before assignment (or effectively not exist -- i.e. the old value should not be used). E.g., good: ??? [ rem for x in range(10) if rem := x%5 ] bad: ??? [ sum_ for x in range(10) if (sum_ := sum_ + x) % 5 ]???? # good luck figuring out what sum_ will hold > Of course you're free to use whatever style rules you prefer locally ? > python-dev has nothing to do with that. > > -n > -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Wed Jul 4 20:59:30 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 03:59:30 +0300 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On 04.07.2018 4:26, Tim Peters wrote: > [INADA Naoki] > > ... > > On the other hand, I understand PEP 572 allows clever code > > simplifies tedious code.? It may increase readability of non-dirty > code. > > The latter is the entire intent ,of course.? We can't force people to > write readable code, but I don't understand the widespread assumption > that other programmers are our enemies who have to be preemptively > disarmed ;-) > > Use code review to enforce readable code.? If you want a coding > standard here, use mine:? "if using an assignment expression isn't > obviously better (at least a little so), DON'T USE IT".? That's the > same standard I use for lots of things (e.g., is such-&-such better as > a listcomp or as nested loops?).? It only requires that you have > excellent taste in what "better" means ;-) > > As I noted in the PEP's Appendix A, I refuse to even write code like > > i = j = count = nerrors = 0 > because it squashes conceptually distinct things into a single > statement .? I'll always write that as > > i = j = 0 count = 0 nerrors = 0 > instead - or even in 4 lines if `i` and `j` aren't conceptually related. > > That's how annoyingly pedantic I can be ;-)? ?Yet after staring at > lots of code, starting from a neutral position (why have an opinion > about anything before examination?), I became a True Believer. > > I really don't know what Guido likes best about this, but for me it's > the large number of objectively small wins in `if` and `while` > contexts.? ?They add up.? That conclusion surprised me.? That there > are occasionally bigger wins to be had is pure gravy. > > But in no case did I count "allows greater cleverness" as a win.? The > Appendix contains a few examples of "bad" uses too, where cleverness > in pursuit of brevity harms clarity.? In fact, to this day, I believe > those examples derived from abusing assignment expressions in > real-life code are more horrifying than any of the examples anyone > else _contrived_ to "prove" how bad the feature is. > > I apparently have more faith that people will use the feature as > intended.? Not all people, just most.? The ones who don't can be > beaten into compliance, same as with any other abused feature ;-) > It's not about if a syntax can be used right or wrong. It's about how easy it is to use it right vs wrong. A syntax, any syntax, naturally nudges the user to use it in specific ways, by making these ways easy to write and read. One of Python's hightlights is that it strives to make the easiest solutions the right ones -- "make right things easy, make wrong things hard". How many of the users are "professional" vs "amateur" programmers is irrelevant. (E.g. while newbies are ignorant, pros are instead constantly pressed for time.) Python Zen rather focuses on making it easy to write correct code for everyone, beginners and pros alike. (As St?fane Fermigier righly showed in message from 4 Jul 2018 11:59:47 +0200, there are always orders of magnitude more "amateurs" than "professionals", and even fewer competent ones.) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Wed Jul 4 21:05:52 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 4 Jul 2018 20:05:52 -0500 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> Message-ID: [Steve Dower] > In that case, please provide more examples of how it should work when > the assignment expression appears to define a variable in a scope that > is not on the call stack. > Sorry, I'm not clear what you're asking about. Python's scopes are determined statically, at compile-time - they have nothing directly to do with what's on the call stack at runtime. This PEP doesn't change anything about that, either. Whether intentional or not, there will be changes to how and when names > are resolved. Assignment expressions don't currently exist, so it's not possible to change how their targets' names get resolved ;-) If the resolution of any _other_ names got changed, that would be a bug, not an unintended consequence. > The specification should provide enough information to > determine the preferred behaviour, so we can tell the difference between > intention changes and implementation bugs. > > For example, what should be returned from this function? > Thanks! This clearly requires examples to flesh it out. > > >>> A = 0 > >>> def f(x): > ... if x: > ... [A := i for i in [1]] > ... return A > > It depends on how it's called. `A` is local to `f` in any case (your global `A` is wholly irrelevant here).. f(x) would raise UnboundLocalError if `x` is not truthy, and return 1 if `x` is truthy. > As far as I can tell, the closest current equivalent will not compile: > > >>> A = 0 > >>> def f(x): > ... if x: > ... def g(): > ... nonlocal A > ... A = 1 > ... g() > ... return A > ... > File "", line 4 > SyntaxError: no binding for nonlocal 'A' found > That's because it's an incorrect translation: the "if the name is otherwise unknown in the containing block, establish it as local in the containing block" requires code in a "by hand" translation to force that to be the case. "Otherwise unknown" means "is not declared global or nonlocal in the block, and is not already known to be local to the block". Because `A` isn't assigned to in the body of `f` outside the listcomp, CPython _today_ has no idea `A` is intended to be local to `f`, so a by-hand translation requires adding silly cruft to force `A` to be recognized as local to `f`. Perhaps the easiest "covers all cases" way: def f(x): if 0: # NEW CODE HERE A = None # AND HERE if x: # etc That doesn't generate any code at all in CPython (the optimizer throws away the entire "if 0:" block).. But compile-time analysis _does_ see the "A = None" binding before the code is thrown away, and that's enough to establish that `A` is local to `f`. Then it compiles fine today, and: >>> f(0) Traceback (most recent call last): .... return A UnboundLocalError: local variable 'A' referenced before assignment >>> f(1) 1 > Is this the equivalent behaviour you want? No, `A` is local to `f` - no compile-time error is appropriate here. > Or do you want an > UnboundLocalError when calling f(0)? Yup, because `A` is local to `f` but not bound at the time it's referenced. > Or do you want the global A to be returned? If and only if there was a `global A` declaration in `f`. > How should we approach decision making about these cases as we > implement this? The PEP does not provide enough information for me to > choose the right behaviour here, and I argue that it should. > > Well, as before, the PEP pretty clearly (to me) already says that `A` is local to `f`. Everything else about the semantics follows from that. But also as in an earlier reply, I think the PEP could help by providing some worked-out workalike-function examples. The "if 0:" trick above isn't deep, but it is too "clever" to be obvious. Then again, it's not required to _specify_ the semantics, only to illustrate one possible way of _implementing_ the semantics. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Jul 4 21:19:19 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 11:19:19 +1000 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: References: Message-ID: <20180705011919.GB14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 04:52:54PM -0700, Mike Miller wrote: > Additionally, I have noticed a dichotomy between prolific "C programmers" > who've supported this PEP and many Python programmers who don't want it. Prolific C programmers like me, hey? *shakes head in a combination of amusement and bemusement* -- Steve From vano at mail.mipt.ru Wed Jul 4 21:22:36 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 04:22:36 +0300 Subject: [Python-Dev] Don't assign to a variable used later in the expression In-Reply-To: <855a51ea-2d09-7d3c-51c9-5a5200137971@mail.mipt.ru> References: <77d12d3f-6220-c644-f481-d08c1a0e8c93@mail.mipt.ru> <855a51ea-2d09-7d3c-51c9-5a5200137971@mail.mipt.ru> Message-ID: <1162016e-0b85-c443-a94d-7add74a38031@mail.mipt.ru> On 05.07.2018 3:40, Ivan Pozdeev via Python-Dev wrote: > On 05.07.2018 2:29, Nathaniel Smith wrote: >> On Wed, Jul 4, 2018 at 4:10 PM, Ivan Pozdeev via Python-Dev >> wrote: >>> On 04.07.2018 10:10, Nathaniel Smith wrote: >>>> Right, Python has a *very strong* convention that each line should >>>> have at most one side-effect, and that if it does have a side-effect >>>> it should be at the outermost level. >>>> I think the most striking evidence for this is that during the >>>> discussion of PEP 572 we discovered that literally none of us ? >>>> including Guido ? even *know* what the order-of-evaluation is inside >>>> expressions. In fact PEP 572 now has a whole section talking about the >>>> oddities that have turned up here so far, and how to fix them. Which >>>> just goes to show that even its proponents don't actually think that >>>> anyone uses side-effects inside expressions, because if they did, then >>>> they'd consider these changes to be compatibility-breaking changes. Of >>>> course the whole point of PEP 572 is to encourage people to embed >>>> side-effects inside expressions, so I hope they've caught all the >>>> weird cases, because even if we can still change them now we won't be >>>> able to after PEP 572 is implemented. >>> I may have a fix to this: >>> >>> Do not recommend assigning to the variable that is used later in the >>> expression. >> This would rule out all the comprehension use cases. > Only those outside of the outermost iterable. Scratch this line, it was from an earlier edit of the letter. I invalidate this myself further on. >> I'd be fine with that personally. Reading through the PEP again I see >> that there are more examples of them than I previously realized, >> inside the semantics discussion and... well, this may be a personal >> thing but for me they'd all be better described as "incomprehensions". >> But, nonetheless, the comprehension use cases are supposed to be a >> core motivation for the whole PEP. > > Far from it. If/while, too. Any construct that accepts an expression > and uses its result but doesn't allow to insert an additional line in > the middle qualifies. >> Also, some of the main arguments >> for why a full-fledged := is better than the more limited alternative >> proposals rely on using a variable on the same line where it's >> assigned (e.g. Tim's gcd example). So I don't see this recommendation >> getting any official traction within PEP 572 or PEP 8. > That's actually a valid use case! > In the aforementioned example, > if (diff := x - x_base) and (g := gcd(diff, n)) > 1: > return g > ?the variable `diff' doesn't exist before, so there's no confusion > which value is used. > > > Okay, I stay corrected: > > _Do not recommend *changing* a variable that is used later in the > expression._ > > I.e. the variable should not exist before assignment (or effectively > not exist -- i.e. the old value should not be used). > > > E.g., good: > > ??? [ rem for x in range(10) if rem := x%5 ] > > bad: > > ??? [ sum_ for x in range(10) if (sum_ := sum_ + x) % 5 ]???? # good > luck figuring out what sum_ will hold > > >> Of course you're free to use whatever style rules you prefer locally ? >> python-dev has nothing to do with that. >> >> -n >> > > -- > Regards, > Ivan > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Wed Jul 4 21:33:44 2018 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 5 Jul 2018 02:33:44 +0100 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: On 2018-07-05 01:11, Victor Stinner wrote: > The code comes from Lib/_pyio.py. Simplified code: > --- > nodata_val = b"" > ... > if n is None or n == -1: > ... > current_size = 0 > while True: > chunk = self.raw.read() > if chunk in empty_values: > nodata_val = chunk > break > current_size += len(chunk) > chunks.append(chunk) > return b"".join(chunks) or nodata_val > > ... > while avail < n: > chunk = self.raw.read(wanted) > if chunk in empty_values: > nodata_val = chunk > break > avail += len(chunk) > chunks.append(chunk) > > ... > return out[:n] if out else nodata_val > --- > > It seems like "nodata_val = " assignment can be moved out of the first > loop, but cannot be moved for the second loop (since the second loop > has no iteration if "avail >= n"). > > Yeah, maybe for this specific file, assignment expressions could be > used for the (C) case and would be worth it. In this case, the second loop might be better left as-is because there are 2 conditions for leaving the loop. Stylistically, it might be starting to hurt readability with something like: while avail < n or (chunk := self.raw.read(wanted)) not in empty_values: [snip] From vano at mail.mipt.ru Wed Jul 4 21:38:59 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 04:38:59 +0300 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: <5549fe9f-6215-eba6-3b0b-dea5eff83c8b@mail.mipt.ru> On 05.07.2018 3:36, Chris Angelico wrote: > On Thu, Jul 5, 2018 at 10:33 AM, Victor Stinner wrote: >> 2018-07-05 2:15 GMT+02:00 Chris Angelico : >>> On Thu, Jul 5, 2018 at 10:03 AM, Victor Stinner wrote: >>>> On the 3360 for loops of the stdlib (*), I only found 2 loops which >>>> would benefit of assignment expressions. >>>> >>>> It's not easy to find loops which: >>>> - build a list, >>>> - are simple enough to be expressed as list comprehension, >>>> - use a condition (if), >>>> - use an expression different than just a variable name as the list >>>> value (value appended to the list). >>> Are you implying that the above conditions are essential for >>> assignment expressions to be useful, or that this defines one >>> particular way in which they can be of value? >> Hum, maybe I wasn't specific enough. I'm looking for "for loops" and >> list comprehensions in stdlib which *can be* written using assignment >> expression, like: >> >> [var for ... in ... if (var := expr)]. >> >> I'm not discussing if such change is worth it or not. I just counted >> how many for loops/list comprehensions *can* be modified to use >> assingment expressions in the stdlib. >> >> Maybe I missed some loops/comprehensions, and I would be happy to see >> more examples ;-) > Cool. So you're looking for ones that fit a particular pattern that > can benefit, but there are almost certainly other patterns out there. > Just making sure that you weren't trying to say "out of 3360 loops, > exactly 3358 of them absolutely definitely cannot be improved here". Well, if no-one knows how to find something that can be improved, it can't be improved :) > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From rob.cliffe at btinternet.com Wed Jul 4 21:25:07 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 5 Jul 2018 02:25:07 +0100 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: On 05/07/2018 01:25, Tim Peters wrote: > > == Pattern 5, two variables == > > while True: > ? ? m = match() > ? ? if not m: > ? ? ? ? break > ? ? j = m.end() > ? ? if i == j: > ? ? ? ? break > ? ? ... > > replaced with: > > while (m := match()) and (j := m.end()) == i: > I assume (sorry to be pedantic :-)) this is a typo for ??? ??? while (m := match()) and (j := m.end()) != i: > > ? ? ... > > Maybe we reached here the maximum acceptable complexity of a single > Python line? :-) > > > It's at my limit.? But, as in an earlier example, I'd be tempted to do > "the obvious part": > > ? ? while m:= match(): > ? ? ? ? j = m.end() > ? ? ? ? if i == j:: > ? ? ? ? ? ? break > > Then the start reads like "while there's something _to_ look at::" and > the body of the loop is happily guaranteed that there is. > > . Or you could compromise with this "intermediate density" version that does two "obvious parts": ??? while m:=match(): ??? ??? if? (j:=m.end()) == i: ??? ??? ??? break (or as I might write it ??? while m:=match(): ??? ??? if? (j:=m.end()) == i:? break ). Some might prefer this as shorter than non-AE version but less dense than the one-liner.? Others might not. /De gustibus non est disputandum./ My conclusion:? Assignment expressions are - like any other Python feature - a tool, to be used with discretion and judgement.? Not the start of a competition to see who can write the most slick/unreadable code. Regards Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Wed Jul 4 21:35:42 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 5 Jul 2018 02:35:42 +0100 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: <646b57ab-1210-6702-ad69-556c58d233a4@btinternet.com> On 05/07/2018 00:15, Nathaniel Smith wrote: > On Wed, Jul 4, 2018 at 3:51 PM, Victor Stinner wrote: >> My question is now: for which "while True" patterns are the assignment >> expression appropriate? There identified different patterns. >> >> >> == Pattern 1, straighforward == >> >> while True: >> line = input.readline() >> if not line: >> break >> ... >> >> IMHO here assingment expression is appropriate here. The code remains >> straighfoward to read. >> >> while (line := input.readline()): >> ... > There are some obvious caveats here ? no-one has real experience with > := yet, so any opinions right now are informed guesswork that will > probably change some if/when we get more experience with it. Of course, this is absolutely true.? But ... > Also, the > stdlib is a big place, and it seems inevitable that maintainers of > different modules will have different preferences. So I don't think it > makes sense to try to define some Official Universal Rule about when > := is appropriate and when it isn't. > > That said, FWIW, my current feeling is that this simplest case is the > only one where I would use :=; for your other examples I'd stick with > the loop-and-a-half style. ... even you, Nathaniel (correct me if I'm wrong, but you seem to be generally against PEP 572) would use := in (at least) one case. I predict:? We'll all end up loving Assignment Expressions (like f-strings) once we've got used to them.? Or just taking them for granted.? No doubt some of us will use them more than others, some perhaps not at all. Regards Rob Cliffe From vano at mail.mipt.ru Wed Jul 4 22:29:51 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 05:29:51 +0300 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: <06cc38e1-5571-3686-07a7-5e1ece4cb7c5@mail.mipt.ru> On 05.07.2018 1:51, Victor Stinner wrote: > Hi, > > Let's say that the PEP 572 (assignment expression) is going to be > approved. Let's move on and see how it can be used in the Python > stdlib. > > I propose to start the discussion about "coding style" (where are > assignment expressions appropriate or not?) with the "while True" > case. > > I wrote a WIP pull request to use assignment expressions in "while True": > https://github.com/python/cpython/pull/8095/files > > In short, replace: > > while True: > x = expr > if not x: > break > ... > with: > > while (x := expr): > ... > > My question is now: for which "while True" patterns are the assignment > expression appropriate? There identified different patterns. > > > == Pattern 1, straighforward == > > while True: > line = input.readline() > if not line: > break > ... > > IMHO here assingment expression is appropriate here. The code remains > straighfoward to read. > > while (line := input.readline()): > ... > > > == Pattern 2, condition == > > Condition more complex than just "not line": > > while True: > q = c//n > if n <= q: > break > ... > > replaced with: > > while (q := c//n) < n: > ... > > IMHO it's still acceptable to use assignement expression... Maybe only > for basic conditions? (see above) > > > == Pattern 3, double condition == > > while True: > s = self.__read(1) > if not s or s == NUL: > break > .... > > replaced with: > > while (s := self.__read(1)) and s != NUL: > ... > > Honestly, here, I don't know if it's appropriate... > > At the first look, "s != NUL" is surprising, since "s" is not defined > before the while, it's only defined in the first *test* (defining a > variable inside a test is *currently* uncommon in Python). > > > == Pattern 4, while (...): pass == > > Sometimes, the loop body is replaced by "pass". > > while True: > tarinfo = self.next() > if tarinfo is None: > break > > replaced with: > > while (tarinfo := self.next()) is not None: > pass > > It reminds me the *surprising* "while (func());" or "while (func()) > {}" in C (sorry for theorical C example, I'm talking about C loops > with an empty body). > > Maybe it's acceptable here, I'm not sure. Would be more readable with a more descriptive variable name: while (chunk := self.next()) is not None: pass tarinfo = chunk > Note: such loop is rare (see my PR). > > > == Pattern 5, two variables == > > while True: > m = match() > if not m: > break > j = m.end() > if i == j: > break > ... > > replaced with: > > while (m := match()) and (j := m.end()) == i: > ... > > Maybe we reached here the maximum acceptable complexity of a single > Python line? :-) Would be more readable with additional parentheses: while (m := match()) and ((j := m.end()) == i): > > > == Other cases == > > I chose to not use assignment expressions for the following while loops. > > (A) > > while True: > name, token = _getname(g) > if not name: > break > ... > > "x, y := ..." is invalid. It can be tricked using "while (x_y := > ...)[0]: x, y = x_y; ...". IMHO it's not worth it. > > (B) > > while True: > coeff = _dlog10(c, e, places) > # assert len(str(abs(coeff)))-p >= 1 > if coeff % (5*10**(len(str(abs(coeff)))-p-1)): > break > places += 3 > > NOT replaced with: > > while not (coeff := _dlog10(c, e, places)) % (5*10**(len(str(abs(coeff)))-p-1)): > places += 3 > > ^-- Tim Peters, I'm looking at you :-) > > coeff is defined and then "immediately" used in "y" expression of > x%y... Yeah, it's valid code, but it looks too magic to me... > > (C) > > while True: > chunk = self.raw.read() > if chunk in empty_values: > nodata_val = chunk > break > ... > > "nodata_val = chunk" cannot be put into the "chunk := self.raw.read()" > assignment expression combined with a test. At least, I don't see how. > (D) > > while 1: > u1 = random() > if not 1e-7 < u1 < .9999999: > continue > ... > > Again, I don't see how to use assignment expression here. Here, unlike the previous example, the assignment is a subexpression of the conditions, so it _can_ be inlined: while? (u1:=random()) < 1e-7 or u1 > .9999999: > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From vano at mail.mipt.ru Wed Jul 4 22:33:50 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 05:33:50 +0300 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: References: Message-ID: <51a528fa-e602-4441-501e-d700661f2795@mail.mipt.ru> On 05.07.2018 2:52, Mike Miller wrote: > Recently on Python-Dev: > > On 2018-07-03 15:24, Chris Barker wrote: > > On Tue, Jul 3, 2018 at 2:51 PM, Chris Angelico >???? On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka > > > > >???? > I believe most Python users are not > >???? > professional programmers -- they are sysadmins, scientists, > hobbyists > >???? > and kids -- > > > >???? [citation needed] > > > > fair enough, but I think we all agree that *many*, if not most, > Python users > > are "not professional programmers". While on the other hand everyone > involved > > in discussion on python-dev and python-ideas is a serious (If not > > "professional") programmer. > > > Python Audience - wants clarity: > > Not sure I'd say that most users are not professionals, but one major > strength of Python is its suitability as a teaching language, which > enlarges the community every year. > > Additionally, I have noticed a dichotomy between prolific "C > programmers" who've supported this PEP and many Python programmers who > don't want it.? While C-devs use this construct all the time, their > stereotypical Python counterpart is often looking for simplicity and > clarity instead.? That's why we're here, folks. > > > Value - good: > > Several use cases are handled well by PEP 572.? However it has been > noted that complexity must be capped voluntarily relatively early?or > the cure soon becomes worse than the disease. > > > Frequency - not much: > > The use cases for assignment-expressions are not exceedingly common, > coming up here and there.? Their omission has been a very mild burden > and we've done without for a quarter century. > > Believe the authors agreed that it won't be used too often and won't > typically be mis- or overused. > > > New Syntax - a high burden: > > For years I've read on these lists that syntax changes must clear a > high threshold of the (Value*Frequency)/Burden (or VF/B) ratio. > > Likewise, a few folks have compared PEP 572 to 498 (f-strings) which > some former detractors have come to appreciate.? Don't believe this > comparison applies well, since string interpolation is useful a > hundred times a day, more concise, clear, and runs faster than > previous functionality.? Threshold was easily cleared there. > > > Conclusion: > > An incongruous/partially redundant new syntax to perform existing > functionality more concisely feels too low on the VF/B ratio IMHO.? > Value is good though mixed, frequency is low, and burden is higher > than we'd like, resulting in "meh" and binary reactions. > > Indeed many modern languages omit this feature specifically in an > effort to reduce complexity, ironically citing the success of Python > in support.? Less is more. > > > Compromise: > > Fortunately there is a compromise design that is chosen often these > days in new languages---restricting these assignments to if/while > (potentially comp/gen) statements. https://mail.python.org/pipermail/python-dev/2018-July/154343.html : "Any construct that accepts an expression and uses its result but doesn't allow to insert an additional line in the middle qualifies." If/when is not enough. And https://mail.python.org/pipermail/python-dev/2018-June/154160.html disproves the "chosen often these days in new languages". > We can also reuse the existing "EXPR as NAME" syntax that already > exists and is widely enjoyed. > For the record, with "as", Victor Stinner's examples from the 5 Jul 2018 00:51:37 +0200 letter would look like: while expr as x: while input.readline() as line: while (c//n as q) < n: while (self.__read(1) as s) and s != NUL: while (self.next() as tarinfo) is not None: pass while (match() as m) and (m.end() as j) == i: > This compromise design: > > ??? 1? Handles the most common cases (of a group of infrequent cases) > ??? 0? Doesn't handle more obscure cases. > ??? 1? No new syntax (through reuse) > ??? 1? Looks Pythonic as hell > ??? 1? Difficult to misuse, complexity capped > > ??? Score: 4/5 > > PEP 572: > > ??? 1? Handles the most common cases (of a group of infrequent cases) > ??? 1? Handles even more obscure cases. > ??? 0? New syntax > ??? 0? Denser look: more colons, parens, expression last > ??? 0? Some potential for misuse, complexity uncapped > > ??? Score: 2/5 > > > Thanks for reading, happy independence, > -Mike Very fitting, given the recent mentions of "dictatorship" and all :-) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From steve at pearwood.info Wed Jul 4 22:41:05 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 12:41:05 +1000 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: <20180704133125.GR14437@ando.pearwood.info> <20180704180215.GW14437@ando.pearwood.info> Message-ID: <20180705024105.GC14437@ando.pearwood.info> On Wed, Jul 04, 2018 at 01:00:41PM -0700, Devin Jeanpierre wrote: > On Wed, Jul 4, 2018 at 11:04 AM Steven D'Aprano wrote: > > Did you actually mean arbitrary simple statements? > > > > if import math; mylist.sort(); print("WTF am I reading?"); True: > > pass > > Yes. Okay, you just broke my brain. I was *sure* that you were going to complain that I was misrepresenting your position, or attacking a strawman, or something. I did not imagine for a second that you *actually* would prefer to allow the above code over assignment expressions. -- Steve From vano at mail.mipt.ru Wed Jul 4 23:28:20 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 06:28:20 +0300 Subject: [Python-Dev] PEP 572 semantics: all capabilities of the assignment statement Message-ID: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> Victor Stinner in "Assignment expression and coding style: the while True case" and others have brought to attention that the AE as currently written doesn't support all the capabilities of the assignment statement, namely: * tuple unpacking * augmented assignment (I titled the letter "all capabilities" 'cuz I may've missed something.) Should it? Personally, I'm for the unpacking but against augmentation 'cuz it has proven incomprehensible as per the 5 Jul 2018 04:22:36 +0300 letter. -- Regards, Ivan From songofacandy at gmail.com Wed Jul 4 23:41:15 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 5 Jul 2018 12:41:15 +0900 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3CEEEE.30204@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> Message-ID: On Thu, Jul 5, 2018 at 1:13 AM Jeroen Demeyer wrote: > > On 2018-07-04 03:31, INADA Naoki wrote: > > I think both PEPs are relying on FASTCALL calling convention, > > and can't be accepted until FASTCALL is stable & public. > > First of all, the fact that FASTCALL has not been made public should > not prevent from discussing those PEPs and even making a > (provisional?) decision on them. I don't think that the precise > API of FASTCALL really matters that much. > > More importantly, I don't think that you can separate making FASTCALL > public from PEP 576/580. As you noted in [1], making FASTCALL public > means more than just documenting METH_FASTCALL. > > In particular, a new API should be added for calling objects using the > FASTCALL convention. I meant _PyArg_ParseStack should be public when METH_FASTCALL is public. Without argument parsing API, it's not practical to implement methods with METH_FASTCALL. I didn't mean other APIs for calling (e.g. _PyObject_FastCall, etc). Without these APIs, 3rd parties can use METH_FASTCALL for tp_methods and m_methods, like stdlibs. Existing public APIs like PyObject_CallMethod() use FASTCALL internally too. So we **can** make public METH_FASTCALL, before make calling APIs public. And stabilizing calling convention is prerequirements of designing new calling APIs. That's why I suggest discussing METH_FASTCALL first. > > Here I mean both an abstract API for arbitrary > callables as well as a specific API for certain classes. Since PEP 580 > (and possibly also PEP 576) proposes changes to the implementation of > FASTCALL, it makes sense to design the public API for FASTCALL after > it is clear which of those PEPs (if any) is accepted. If we fix the > FASTCALL API now, it might not be optimal when either PEP 576 or PEP 580 > is accepted. > I agree that calling APIs should be discusses with PEP 580. But I didn't mean FASTCALL calling API, but low level FASTCALL calling convention used for tp_methods and m_methods and parsing APIs for it. Does both PEPs suggests changing it? I didn't think so. -- INADA Naoki From tim.peters at gmail.com Thu Jul 5 00:06:37 2018 From: tim.peters at gmail.com (Tim Peters) Date: Wed, 4 Jul 2018 23:06:37 -0500 Subject: [Python-Dev] PEP 572 semantics: all capabilities of the assignment statement In-Reply-To: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> References: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> Message-ID: [Ivan Pozdeev] > Victor Stinner in "Assignment expression and coding style: the while > True case" and others have brought to attention > > that the AE as currently written doesn't support all the capabilities of > the assignment statement, namely: > > * tuple unpacking > * augmented assignment > > (I titled the letter "all capabilities" 'cuz I may've missed something.) > > Assignment statements (and `for` headers) also allow arbitrarily complex applications of subscripting and attribute references. Like for a[b].cde[f, g(h, i, j)].k in range(10): > Should it? > Already been considered and rejected, so no. Victor already showed why, but didn't quite realize it ;-) while True: name, token = _getname(g) if not name: break Allowing to unpack the two names in an AE wouldn't really help, because there's still no satisfying way then to _reference_ just the `name` part in the `while` test: while ((name, token) := _getname(g)) ... and now what?? You could use "a trick", relying on that a 2-tuple is always truthy: while ((name, token) := _getname(g)) and name: Using a trick sucks. Not relying on a trick is ugly and inefficient: while [((name, token) := _getname(g)), name][-1]: or while [((name, token) := _getname(g)), name].pop(): where the thing to be tested is the second list element (e.g., `name == "not done yet"`). So no plausible use cases were ever presented, and the idea was dropped. If we _also_ added something akin to C's comma expressions (evaluate a sequence of expressions and throw away all the results except for the last), then a reasonable spelling akin to the last one above could be used, but without the strained cruft to muck with a result list (or tuple). At which point my rule for AE would likely never trigger ("if it's not obviously better, don't use it"). -------------- next part -------------- An HTML attachment was scrubbed... URL: From shawnchen1996 at outlook.com Thu Jul 5 00:01:24 2018 From: shawnchen1996 at outlook.com (Shawn Chen) Date: Thu, 5 Jul 2018 04:01:24 +0000 Subject: [Python-Dev] PEP 484 Message-ID: Hello, Here, I am proposing a change on python type annotation. Python was born to be a simple and elegant language. However recent change has again introduce new incompatibility to python. The PEP 484 is proposing a type hint which can annotate the type of each parameters. How ever code written in this format can not be run for python3.5 and below. It is an exciting new feature to be able to know the data type from the code, But I am afraid this is not worth such a incompatibility. Here I want to propose a new way of annotation in python as follows def reportAge(name, age): ''' this a a greeting function and some other comment... !str, int -> str ''' return name+' is ' + age we can put the annotation in the comment block and use a symbol '!' or other symbol suitable to lead a annotation line. the annotation should be positionally corresponding to the parameters. Shawn Sent from Mail for Windows 10 -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Jul 5 00:00:08 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jul 2018 14:00:08 +1000 Subject: [Python-Dev] PEP 572 semantics: all capabilities of the assignment statement In-Reply-To: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> References: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> Message-ID: On Thu, Jul 5, 2018 at 1:28 PM, Ivan Pozdeev via Python-Dev wrote: > Victor Stinner in "Assignment expression and coding style: the while True > case" and others have brought to attention > > that the AE as currently written doesn't support all the capabilities of the > assignment statement, namely: > > * tuple unpacking > * augmented assignment > > (I titled the letter "all capabilities" 'cuz I may've missed something.) > > Should it? > > Personally, I'm for the unpacking but against augmentation 'cuz it has > proven incomprehensible as per the 5 Jul 2018 04:22:36 +0300 letter. > Definitely against augmentation, for several reasons: 1) Spelling - should it be :+= or +:= ? 2) Is the result of the expression the modified value or the original? 3) The use-cases simply aren't as strong. Supporting arbitrary assignment targets (rather than just a simple name) could be useful, but can be deferred to a future enhancement without impacting the simpler version. I would divide this up into two subgroups: * Multiple assignment (sequence unpacking) * Assignment to non-simple names eg "x[1] := expr" Assigning directly to an item or attribute could in theory be immensely valuable. So could multiple assignment, though I suspect to a lesser extent. But tell me: Without looking it up, do you know which of these constructs support non-simple-name assignment and which don't? [x[1] for x[1] in seq] with ctx() as x[1]: except Exception as x[1]: from spam import ham as x[1] In the enormous majority of cases, every one of these constructs is going to be used with a simple name, even though some (I won't say how many) do permit you to do what I did here. If Python 3.8 ships with assignment expressions restricted to simple names, we can discuss how valuable the other forms of assignment target would be, and then figure out what to do about the ambiguities - for instance, is "x, y := expr" going to be equivalent to "x, (y := expr)" or "(x, y) := expr" ? As it is, we neatly dodge that. ChrisA From rymg19 at gmail.com Thu Jul 5 00:21:09 2018 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Wed, 04 Jul 2018 23:21:09 -0500 Subject: [Python-Dev] PEP 484 In-Reply-To: References: Message-ID: <16468ac2308.27a3.db5b03704c129196a4e9415e55413ce6@gmail.com> Type hints like in PEP 484 work on all Python 3 versions, and something similar to your proposal is already supported on Python 2 [1]. [1]: https://mypy.readthedocs.io/en/latest/python2.html On July 4, 2018 11:08:27 PM Shawn Chen wrote: > Hello, > > Here, I am proposing a change on python type annotation. > > Python was born to be a simple and elegant language. However recent change > has again introduce new incompatibility to python. > > The PEP 484 is proposing a type hint which can annotate the type of each > parameters. How ever code written in this format can not be run for > python3.5 and below. > > It is an exciting new feature to be able to know the data type from the > code, But I am afraid this is not worth such a incompatibility. > > Here I want to propose a new way of annotation in python as follows > > def reportAge(name, age): > ''' this a a greeting function and some other comment... > !str, int -> str > ''' > return name+' is ' + age > > > we can put the annotation in the comment block and use a symbol '!' or > other symbol suitable to lead a annotation line. > the annotation should be positionally corresponding to the parameters. > > Shawn > > Sent from Mail for Windows 10 > > > > > ---------- > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Jul 5 01:02:37 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 5 Jul 2018 01:02:37 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: <5c3acc1a-55d8-72f1-11a8-9bdd64c406af@mail.mipt.ru> Message-ID: On 7/4/2018 1:50 PM, Yury Selivanov wrote: > On Wed, Jul 4, 2018 at 1:35 PM Ivan Pozdeev via Python-Dev > wrote: >> >> On 04.07.2018 11:54, Serhiy Storchaka wrote: > >>>> while total != (total := total + term): >>>> term *= mx2 / (i*(i+1)) >>>> i += 2 >>>> return total >>> >>> This code looks clever that the original while loop with a break in a >>> middle. I like clever code. But it needs more mental efforts for >>> understanding it. >>> >>> I admit that this is a good example. >>> >>> There is a tiny problem with it (and with rewriting a while loop as a >>> for loop, as I like). Often the body contains not a single break. In >>> this case the large part of cleverness is disappeared. :-( >> >> It took me a few minutes to figure out that this construct actually >> checks term == 0. No. Floats are not reals. The test is that term is small enough *relative to the current total* that we should stop adding more terms. >>> 1e50 + 1e30 == 1e50 True 1e30 in not 0 ;-) > Wow, I gave up on this example before figuring this out (and I also > stared at it for a good couple of minutes). Now it makes sense. It's > funny that this super convoluted snippet is shown as a good example > for PEP 572. Although almost all PEP 572 examples are questionable. -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Thu Jul 5 01:04:51 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 05 Jul 2018 17:04:51 +1200 Subject: [Python-Dev] PEP 484 In-Reply-To: References: Message-ID: <5B3DA6F3.5030808@canterbury.ac.nz> Shawn Chen wrote: > The PEP 484 is proposing a type hint which can annotate the type of each > parameters. How ever code written in this format can not be run for > python3.5 and below. You're a bit late. Parameter annotations have been a part of the language since at least 3.1. PEP 484 just codifies a way of using them to represent types. Also, PEP 484 does specify a way of using comments to indicate types. -- Greg From tjreedy at udel.edu Thu Jul 5 01:13:13 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 5 Jul 2018 01:13:13 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On 7/4/2018 2:32 PM, Sven R. Kunze wrote: > Sorry for adding yet another mail. :-( > > On 04.07.2018 10:54, Serhiy Storchaka wrote: >> Sorry, this PEP was rewritten so many times that I missed your Appendix. >> >>> while total != (total := total + term): >>> ??? term *= mx2 / (i*(i+1)) >>> ??? i += 2 >>> return total >> > > This very example here caught my eye. > > Isn't total not always equal to total Repeat response given elsewhere. No. Python has limited precision floats, not reals. Floats do not obey laws of real numbers. >>> 1e50 + 1e34 == 1e50 True >>> 1e50 + 1e35 == 1e50 False Basic numerical analysis operation: approximate infinite sum of decreasing terms by summing terms until they become small enough that they can be ignored. -- Terry Jan Reedy From guido at python.org Thu Jul 5 01:15:35 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 4 Jul 2018 22:15:35 -0700 Subject: [Python-Dev] PEP 572 semantics: all capabilities of the assignment statement In-Reply-To: References: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> Message-ID: Let me be slightly contrarian. :-) On Wed, Jul 4, 2018 at 9:12 PM Chris Angelico wrote: > Definitely against augmentation, for several reasons: > > 1) Spelling - should it be :+= or +:= ? > That one's easy. As Nick's (withdrawn) PEP 577 shows it should be simply `+=`. > 2) Is the result of the expression the modified value or the original? > Someone (sadly I forget who) showed, convincingly (to me anyways :-) that it should return whatever the `__iadd__` method returns, or (if there isn't one) the result of `a = a + b`. > 3) The use-cases simply aren't as strong. > Here I agree. > Supporting arbitrary assignment targets (rather than just a simple > name) could be useful, but can be deferred to a future enhancement > without impacting the simpler version. I would divide this up into two > subgroups: > > * Multiple assignment (sequence unpacking) > Tim Peters showed in his response this isn't all that helpful. I also think we shouldn't open the can of worms about priorities this presents, e.g. is (a, b := foo()) equivalent to ((a, b) := foo()) or is it like (a, (b := foo()))? > * Assignment to non-simple names eg "x[1] := expr" > > Assigning directly to an item or attribute could in theory be > immensely valuable. So could multiple assignment, though I suspect to > a lesser extent. But tell me: Without looking it up, do you know which > of these constructs support non-simple-name assignment and which > don't? > > [x[1] for x[1] in seq] > with ctx() as x[1]: > except Exception as x[1]: > from spam import ham as x[1] > > In the enormous majority of cases, every one of these constructs is > going to be used with a simple name, even though some (I won't say how > many) do permit you to do what I did here. If Python 3.8 ships with > assignment expressions restricted to simple names, we can discuss how > valuable the other forms of assignment target would be, and then > figure out what to do about the ambiguities - for instance, is "x, y > := expr" going to be equivalent to "x, (y := expr)" or "(x, y) := > expr" ? As it is, we neatly dodge that. > Again, the biggest argument against this is that there just aren't enough use cases. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Jul 5 01:19:56 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jul 2018 15:19:56 +1000 Subject: [Python-Dev] PEP 572 semantics: all capabilities of the assignment statement In-Reply-To: References: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> Message-ID: On Thu, Jul 5, 2018 at 3:15 PM, Guido van Rossum wrote: > Let me be slightly contrarian. :-) > > On Wed, Jul 4, 2018 at 9:12 PM Chris Angelico wrote: >> 2) Is the result of the expression the modified value or the original? > > Someone (sadly I forget who) showed, convincingly (to me anyways :-) that it > should return whatever the `__iadd__` method returns, or (if there isn't > one) the result of `a = a + b`. I happen to feel the same way, but it does destroy the elegance of "x := expr" having the exact same value as "expr", and ALSO having the exact same value as "x". But yes, the biggest argument is the lack of use cases, for both iadd and unpacking. ChrisA From tjreedy at udel.edu Thu Jul 5 01:40:20 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 5 Jul 2018 01:40:20 -0400 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704193411.GA14437@ando.pearwood.info> References: <20180704133539.GS14437@ando.pearwood.info> <20180704193411.GA14437@ando.pearwood.info> Message-ID: On 7/4/2018 3:34 PM, Steven D'Aprano wrote: > On Wed, Jul 04, 2018 at 03:24:08PM -0400, Terry Reedy wrote: >> On 7/4/2018 9:35 AM, Steven D'Aprano wrote: >>> On Wed, Jul 04, 2018 at 05:02:07PM +1000, Chris Angelico wrote: >>>> On Wed, Jul 4, 2018 at 4:07 PM, Serhiy Storchaka >>>> wrote: >>> >>>> "Assignment is a statement" -- that's exactly the point under discussion. >> >> I believe that this is Chris quoting and commenting on Serhiy having >> said 'assigment is a statement' >> >>> Not any more it isn't. We've now gone from discussion to bitter >>> recriminations *wink* >> >> I don't see any recrimination in what either said. > > Haven't you been reading the rest of the thread? About half of it. But I was responding here only to a few. > The Twitter storm? No, not until I followed a link in something posted after I wrote the above. I have seen maybe 100 tweets in my life. > The ranting on Reddit that this is the end of the world and Python is > doomed? No. > I wasn't referring specifically to Chris or Serhiy's comments, but about > the general over-reaction, here and elsewhere. The few tweets I read gave me an idea of what you meant. -- Terry Jan Reedy From steve at pearwood.info Thu Jul 5 02:20:06 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 16:20:06 +1000 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: <51a528fa-e602-4441-501e-d700661f2795@mail.mipt.ru> References: <51a528fa-e602-4441-501e-d700661f2795@mail.mipt.ru> Message-ID: <20180705062005.GA7318@ando.pearwood.info> On Thu, Jul 05, 2018 at 05:33:50AM +0300, Ivan Pozdeev via Python-Dev wrote: > And https://mail.python.org/pipermail/python-dev/2018-June/154160.html > disproves the "chosen often these days in new languages". Ivan, I think you may have linked to the wrong page. That page was Chris kindly referring you to my post here: https://mail.python.org/pipermail/python-ideas/2018-May/050938.html which refutes Mike's original, biased selection of a handful of languages. Which he then misrepresented as not including assignment expressions when half of them actually do, at least in a limited form. (3 out of the 5 of Mike's examples include *at least* some limited assignment expression. My survey found 13 out of 18 modern languages have at least some form of assignment expression. See link above for details.) It simply isn't true that modern languages are moving away from assignment expressions. Some are. Some aren't. Even those that don't support assignment expressions in general usually support special syntax to allow it in a few contexts. But even if we pretended that, let's say, Go for example has no assignment expressions (it actually does, but limited only to the special case of if statements), what conclusion should we draw? That Rob Pike is ever so much a better language designer than Guido? Maybe he is, maybe he isn't, but Go is just eight years old. Python is 27. When Python was 8, it lacked a lot of features we find indispensible now: https://www.python.org/download/releases/1.5/whatsnew/ Who is to say that when Go is 27, or even 10, it won't have added assignment expressions? Some of Go's choices seem a bit... idiosyncratic. Strings are still ASCII byte-strings. Unicode text is relegated to a seperate type, "runes", the naming of which is a tad patronising and contemptuous of non-ASCII users. There are no exceptions or try...finally. The designers bowed to public pressure and added a sort of poor-man's exception system, panic/recover, but for most purposes, they still requiring the "check a flag to test success" anti-pattern. The designers are actively opposed to assertions. I dare say a lot of Python's choices seem strange to Go programmers too. Rather than saying "Go got it right", maybe we should be saying "Go got it wrong". > >We can also reuse the existing "EXPR as NAME" syntax that already > >exists and is widely enjoyed. > > > > For the record, with "as", Victor Stinner's examples from the 5 Jul 2018 > 00:51:37 +0200 letter would look like: Enough with the "as" syntax. This discussion has been going on since FEBRUARY, and "as" was eliminated as ambiguous months ago. Stop beating that dead horse. -- Steve From storchaka at gmail.com Thu Jul 5 02:23:34 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 5 Jul 2018 09:23:34 +0300 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: 05.07.18 01:51, Victor Stinner ????: > == Pattern 1, straighforward == > > while True: > line = input.readline() > if not line: > break > ... > > IMHO here assingment expression is appropriate here. The code remains > straighfoward to read. > > while (line := input.readline()): > ... We already have an idiom for this: for line in input: ... From storchaka at gmail.com Thu Jul 5 02:30:25 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 5 Jul 2018 09:30:25 +0300 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: 05.07.18 03:03, Victor Stinner ????: > + labels = [slabel for label > + in self._file.readline()[1:].split(b',') > + if (slabel := label.strip())] labels = [slabel for label in self._file.readline()[1:].split(b',') for slabel in [label.strip()] if slabel] > + lines = [match.group(1, 2) > + for raw_line in raw_lines > + if (match := line_pat.search(raw_line.strip()))] lines = [match.group(1, 2) for raw_line in raw_lines for match in [line_pat.search(raw_line.strip())] if match] But in all these cases I prefer the original loop. From steve at pearwood.info Thu Jul 5 02:47:47 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 5 Jul 2018 16:47:47 +1000 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: <20180705064746.GB7318@ando.pearwood.info> On Thu, Jul 05, 2018 at 12:51:37AM +0200, Victor Stinner wrote: > I propose to start the discussion about "coding style" (where are > assignment expressions appropriate or not?) with the "while True" > case. We don't even have an official implementation yet, and you already want to start prescribing coding style? We probably have months before 3.8 alpha comes out. I appreciate your enthusiasm, but what's the rush? Give people a chance to play with the syntax in the REPL before making Thou Shalt and Thou Shalt Not rules for coding style and making wholesale changes to the std lib. This topic has been argued and argued and argued on two mailing lists for over four months. Let's take a couple of weeks to catch our breath, wait for the implementation to actually hit the 3.8 repo, before trying to prescribe coding style or thinking about which parts of the std lib should be refactored to use it and which shouldn't. There is no need to rush into making changes. Let them happen naturally. -- Steve From tim.peters at gmail.com Thu Jul 5 03:10:44 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 5 Jul 2018 02:10:44 -0500 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: <20180705064746.GB7318@ando.pearwood.info> References: <20180705064746.GB7318@ando.pearwood.info> Message-ID: [Victor Stinner] > > > I propose to start the discussion about "coding style" (where are > > assignment expressions appropriate or not?) with the "while True" > > case. > [Steven D'Aprano] > We don't even have an official implementation yet, and you already want > to start prescribing coding style? We probably have months before 3.8 > alpha comes out. > > I appreciate your enthusiasm, but what's the rush? Give people a chance > to play with the syntax in the REPL before making Thou Shalt and Thou > Shalt Not rules for coding style and making wholesale changes to the std > lib. > I'm all in favor of what Victor is doing: looking at how this stuff will work in actual code. That's a great antidote to the spread of theoretical fears. Wholesale changes to the std lib are unlikely to happen regardless. Broad patches just to spell things differently without _need_ are discouraged. > This topic has been argued and argued and argued on two mailing lists > for over four months. Which is why I strongly welcome looking at code instead. A few people have already noticed that some of Victor's changes aren't actually disasters ;-) > Let's take a couple of weeks to catch our breath, wait for the implementation to actually hit the 3.8 repo, before trying > to prescribe coding style or thinking about which parts of the std lib > should be refactored to use it and which shouldn't. > The more code people look at, the better. It won't be merged, but having the diffs makes it all concrete. And while I don't particularly care to argue about coding style myself, at least that would be a _new_ thing to argue about ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Thu Jul 5 05:31:02 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 5 Jul 2018 11:31:02 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> Message-ID: <5B3DE556.5040903@UGent.be> On 2018-07-05 05:41, INADA Naoki wrote: > And stabilizing calling convention is prerequirements of designing new > calling APIs. I don't see why. I made my PEP with the assumption that the METH_FASTCALL calling convention won't change. As far as I know, nobody advocated for changing it. But even if we decide to change METH_FASTCALL, I could trivially adapt my PEP. > That's why I suggest discussing METH_FASTCALL first. I certainly agree that it's a good idea to discuss METH_FASTCALL, but I still don't see why that should block the discussion of PEP 576/580. I can understand that you want to wait to *implement* PEP 576/580 as long as METH_FASTCALL isn't public. But we should not wait to *discuss* those PEPs. Jeroen. From vano at mail.mipt.ru Thu Jul 5 06:58:42 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 13:58:42 +0300 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: On 05.07.2018 9:23, Serhiy Storchaka wrote: > 05.07.18 01:51, Victor Stinner ????: >> == Pattern 1, straighforward == >> >> while True: >> ???? line = input.readline() >> ???? if not line: >> ???????? break >> ???? ... >> >> IMHO here assingment expression is appropriate here. The code remains >> straighfoward to read. >> >> while (line := input.readline()): >> ???? ... > > We already have an idiom for this: > > for line in input: > ??? ... > This is not strictly equivalent: it has internal caching unaffected by -u and you can't iterate and .read() at the same time. Though in this specific case (the example is from Lib\base64.py AFAICS), the change to `for' is fine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From solipsis at pitrou.net Thu Jul 5 07:06:12 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 5 Jul 2018 13:06:12 +0200 Subject: [Python-Dev] Assignment expression and coding style: the while True case References: Message-ID: <20180705130612.603a3754@fsol> On Thu, 5 Jul 2018 13:58:42 +0300 Ivan Pozdeev via Python-Dev wrote: > On 05.07.2018 9:23, Serhiy Storchaka wrote: > > 05.07.18 01:51, Victor Stinner ????: > >> == Pattern 1, straighforward == > >> > >> while True: > >> ???? line = input.readline() > >> ???? if not line: > >> ???????? break > >> ???? ... > >> > >> IMHO here assingment expression is appropriate here. The code remains > >> straighfoward to read. > >> > >> while (line := input.readline()): > >> ???? ... > > > > We already have an idiom for this: > > > > for line in input: > > ??? ... > > > > This is not strictly equivalent: it has internal caching unaffected by > -u and you can't iterate and .read() at the same time. You are only talking about Python 2 here. Regards Antoine. From vano at mail.mipt.ru Thu Jul 5 07:22:46 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 14:22:46 +0300 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: <20180705064746.GB7318@ando.pearwood.info> References: <20180705064746.GB7318@ando.pearwood.info> Message-ID: On 05.07.2018 9:47, Steven D'Aprano wrote: > On Thu, Jul 05, 2018 at 12:51:37AM +0200, Victor Stinner wrote: > >> I propose to start the discussion about "coding style" (where are >> assignment expressions appropriate or not?) with the "while True" >> case. > We don't even have an official implementation yet, and you already want > to start prescribing coding style? We probably have months before 3.8 > alpha comes out. This is an excellent way to look at the picture with a user's eyes. You immediately see what parts of the design lend themselves well to practice and what don't, which details are unclear, which additional features are missing (note how he revealed the lack of tuple unpacking) -- by all means, something to do at the design stage, before doing the implementation. It's also a testimony of how much an improvement a feature will _really_ be -- something that contrived examples can't tell. I salute Victor for such an enlightemed idea and going for the trouble to carry it out! > I appreciate your enthusiasm, but what's the rush? Give people a chance > to play with the syntax in the REPL before making Thou Shalt and Thou > Shalt Not rules for coding style and making wholesale changes to the std > lib. > > This topic has been argued and argued and argued on two mailing lists > for over four months. Let's take a couple of weeks to catch our breath, > wait for the implementation to actually hit the 3.8 repo, before trying > to prescribe coding style or thinking about which parts of the std lib > should be refactored to use it and which shouldn't. > > There is no need to rush into making changes. Let them happen naturally. > > > -- Regards, Ivan From vano at mail.mipt.ru Thu Jul 5 07:28:18 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 14:28:18 +0300 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: <20180705062005.GA7318@ando.pearwood.info> References: <51a528fa-e602-4441-501e-d700661f2795@mail.mipt.ru> <20180705062005.GA7318@ando.pearwood.info> Message-ID: <329901ab-62d1-9ec4-29c9-cfacef321e5f@mail.mipt.ru> On 05.07.2018 9:20, Steven D'Aprano wrote: > On Thu, Jul 05, 2018 at 05:33:50AM +0300, Ivan Pozdeev via Python-Dev wrote: > >> And https://mail.python.org/pipermail/python-dev/2018-June/154160.html >> disproves the "chosen often these days in new languages". > Ivan, I think you may have linked to the wrong page. That page was Chris > kindly referring you to my post here: > > https://mail.python.org/pipermail/python-ideas/2018-May/050938.html This is as intended. I wanted to show my summary and Chris' refuttal, with links to both original posts. Because my letter is much shorter than the originals while carrying the same message. Also to show that I've made the same mistake, which puts things in perspective: how an outsider could get the wrong idea. > which refutes Mike's original, biased selection of a handful of > languages. Which he then misrepresented as not including assignment > expressions when half of them actually do, at least in a limited form. > > (3 out of the 5 of Mike's examples include *at least* some limited > assignment expression. My survey found 13 out of 18 modern languages > have at least some form of assignment expression. See link above for > details.) > > It simply isn't true that modern languages are moving away from > assignment expressions. Some are. Some aren't. Even those that don't > support assignment expressions in general usually support special syntax > to allow it in a few contexts. > > But even if we pretended that, let's say, Go for example has no > assignment expressions (it actually does, but limited only to the > special case of if statements), what conclusion should we draw? > > That Rob Pike is ever so much a better language designer than Guido? > Maybe he is, maybe he isn't, but Go is just eight years old. Python is > 27. When Python was 8, it lacked a lot of features we find indispensible > now: > > https://www.python.org/download/releases/1.5/whatsnew/ > > Who is to say that when Go is 27, or even 10, it won't have added > assignment expressions? > > Some of Go's choices seem a bit... idiosyncratic. Strings are still > ASCII byte-strings. Unicode text is relegated to a seperate type, > "runes", the naming of which is a tad patronising and contemptuous of > non-ASCII users. There are no exceptions or try...finally. The designers > bowed to public pressure and added a sort of poor-man's exception system, > panic/recover, but for most purposes, they still requiring the "check a > flag to test success" anti-pattern. The designers are actively opposed > to assertions. > > I dare say a lot of Python's choices seem strange to Go programmers too. > > Rather than saying "Go got it right", maybe we should be saying "Go got > it wrong". > > > >>> We can also reuse the existing "EXPR as NAME" syntax that already >>> exists and is widely enjoyed. >>> >> For the record, with "as", Victor Stinner's examples from the 5 Jul 2018 >> 00:51:37 +0200 letter would look like: > > Enough with the "as" syntax. This discussion has been going on since > FEBRUARY, and "as" was eliminated as ambiguous months ago. Stop beating > that dead horse. > > > > -- Regards, Ivan From songofacandy at gmail.com Thu Jul 5 07:32:20 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 5 Jul 2018 20:32:20 +0900 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3DE556.5040903@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> Message-ID: On Thu, Jul 5, 2018 at 6:31 PM Jeroen Demeyer wrote: > > On 2018-07-05 05:41, INADA Naoki wrote: > > And stabilizing calling convention is prerequirements of designing new > > calling APIs. > > I don't see why. I made my PEP with the assumption that the > METH_FASTCALL calling convention won't change. As far as I know, nobody > advocated for changing it. But even if we decide to change > METH_FASTCALL, I could trivially adapt my PEP. Serhiy said "the protocol for keyword parameters is more complex and still can be changed." https://mail.python.org/pipermail/python-dev/2018-June/153949.html > > > That's why I suggest discussing METH_FASTCALL first. > > I certainly agree that it's a good idea to discuss METH_FASTCALL, but I > still don't see why that should block the discussion of PEP 576/580. Core devs interested in this area is limited resource. As far as I understand, there are some important topics to discuss. a. Low level calling convention, including argument parsing API. b. New API for calling objects without argument tuple and dict. c. How more types can support FASTCALL, LOAD_METHOD and CALL_METHOD. d. How to reorganize existing builtin types, without breaking stable ABI. It's difficult to understand all topics in both PEPs at once. I suggested to focus on prerequirements first because it helps us to join discussion without understand whole two PEPs. > > I can understand that you want to wait to *implement* PEP 576/580 as > long as METH_FASTCALL isn't public. But we should not wait to *discuss* > those PEPs. > I didn't want wait to "implement". Discussion is the most critical path. Reference implementation helps discussion. Regards, > > Jeroen. > -- INADA Naoki From J.Demeyer at UGent.be Thu Jul 5 07:59:10 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 5 Jul 2018 13:59:10 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> Message-ID: <5B3E080E.4030107@UGent.be> On 2018-07-05 13:32, INADA Naoki wrote: > Core devs interested in this area is limited resource. I know and unfortunately there is nothing that I can do about that. It would be a pity that PEP 580 (or a variant like PEP 576) is not accepted simply because no core developer cares enough. > As far as I understand, there are some important topics to discuss. > > a. Low level calling convention, including argument parsing API. > b. New API for calling objects without argument tuple and dict. > c. How more types can support FASTCALL, LOAD_METHOD and CALL_METHOD. > d. How to reorganize existing builtin types, without breaking stable ABI. Right, that's why I wanted PEP 580 to be only about (c) and nothing else. I made the mistake in PEP 575 of also involving (d). I still don't understand why we must finish (a) before we can even start discussing (c). > Reference implementation helps discussion. METH_FASTCALL and argument parsing for METH_FASTCALL is already implemented in CPython. Not in documented public functions, but the implementation exists. And PEP 580 also has a reference implementation: https://github.com/jdemeyer/cpython/tree/pep580 Jeroen. From vstinner at redhat.com Thu Jul 5 08:13:48 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 14:13:48 +0200 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: <20180705064746.GB7318@ando.pearwood.info> Message-ID: 2018-07-05 9:10 GMT+02:00 Tim Peters : > I'm all in favor of what Victor is doing: looking at how this stuff will > work in actual code. That's a great antidote to the spread of theoretical > fears. FYI I'm trying to use assignment expressions on the stdlib because *all* examples of the PEP 572 look artificial to me. Like "group = re.match(data).group(1) if re.match(data) else None" which is followed by "(TODO: Include Guido's evidence, and do a more systematic search.)" I cannot find such inefficient code in the stdlib. I'm not saying that nobody writes code like that, just that developers with a good Python expertise would avoid to write such code. "filtered_data = [y for x in data if (y := f(x)) is not None]" also seems artificial. In the 711,617 lines of Python code of the stdlib, I only found *one* example: labels = [label.strip() for label in self._file.readline()[1:].split(b',') if label.strip()] => https://github.com/python/cpython/pull/8098/files And I also only found a single for loop which can be converted to a list comprehension thanks to assignement expression. lines = [] for raw_line in raw_lines: match = line_pat.search(raw_line.strip()) if match: lines.append(match.group(1, 2)) > Wholesale changes to the std lib are unlikely to happen regardless. Broad > patches just to spell things differently without _need_ are discouraged. So PEP 572 is purely syntax sugar? It doesn't bring anything to the stdlib for example? My current 3 pull requests showing how assignment expressions can be used in the stdlib: while True: https://github.com/python/cpython/pull/8095/files match/group: https://github.com/python/cpython/pull/8097/files list comp: https://github.com/python/cpython/pull/8098/files Right now, I'm still not really excited by the new code. If you spotted other parts of the stdlib where assignment expressions would be appropriate, please tell me and I will try to write more pull requests :-) Victor From vstinner at redhat.com Thu Jul 5 08:20:01 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 14:20:01 +0200 Subject: [Python-Dev] PEP 572: intended scope of assignment expression Message-ID: Hi, My work (*) in the "Assignment expression and coding style: the while True case" thread helped me to understand something about the *intended* scope. While technically, assignment expressions keep the same scoping rules than assignment statements, writing "if (x := func()): ..." or "while (x := func()): ..." shows the "intented" scope of the variable. Even if, as explained properly in the PEP, the scope is wider (for good reasons) as "for line in file: ..." keeps line alive after the loop (nothing new under the sun). It's something subtle that I missed at the first read (of the code and the PEP), the difference is not obvious. x = func() if x: ... # obviously use x # do we still plan to use x here? # it's non obvious just by reading the if versus if (x := func()): ... # obviously use x # ":=" in the if "announces" that usually x is no longer used # here, even if technically x is still defined See my match/group PR for more concrete examples: https://github.com/python/cpython/pull/8097/files I understand the current PEP 572 rationale as: assignment expressions reduces the number of lines and the indentation level... pure syntax sugar. IMHO this "intended" scope is a much better way to sell assignment expressions than the current rationale. In fact, it's explained later very quickly in the PEP: https://www.python.org/dev/peps/pep-0572/#capturing-condition-values But it could be better explained than just "good effect in the header of an if or while statement". The PEP contains a good example of the intended scope: if pid := os.fork(): # Parent code # pid is valid and is only intended to be used in this scope ... # use pid else: # Child code # pid is "invalid" (equal to zero) ... # don't use pid # since this code path is common to parent and child, # the pid is considered invalid again here # (since the child does also into this path) ... # don't use pid (*) My work: my current 3 pull requests showing how assignment expressions can be used in the stdlib: while True: https://github.com/python/cpython/pull/8095/files match/group: https://github.com/python/cpython/pull/8097/files list comp: https://github.com/python/cpython/pull/8098/files Victor From songofacandy at gmail.com Thu Jul 5 08:20:57 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 5 Jul 2018 21:20:57 +0900 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3E080E.4030107@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> Message-ID: On Thu, Jul 5, 2018 at 9:02 PM Jeroen Demeyer wrote: > > On 2018-07-05 13:32, INADA Naoki wrote: > > Core devs interested in this area is limited resource. > > I know and unfortunately there is nothing that I can do about that. It > would be a pity that PEP 580 (or a variant like PEP 576) is not accepted > simply because no core developer cares enough. What you can do is "divide and conquer". Split PEP in small topics we can focus. > > > As far as I understand, there are some important topics to discuss. > > > > a. Low level calling convention, including argument parsing API. > > b. New API for calling objects without argument tuple and dict. > > c. How more types can support FASTCALL, LOAD_METHOD and CALL_METHOD. > > d. How to reorganize existing builtin types, without breaking stable ABI. > > Right, that's why I wanted PEP 580 to be only about (c) and nothing > else. I made the mistake in PEP 575 of also involving (d). > > I still don't understand why we must finish (a) before we can even start > discussing (c). Again, "discussing" takes much critical resources. And we got nothing in Python 3.8 in worst case. (c) won't be public unless (a) is public, although main motivation of (c) is 3rd party tools. That's why I prefer discussing (a) first. Without (a), discussion about (c) will not born anything in Python 3.8. This is only advice from me and you can start discussion about (c), like you ignored my advice about creating realistic benchmark for calling 3rd party callable before talking about performance... > > > Reference implementation helps discussion. > > METH_FASTCALL and argument parsing for METH_FASTCALL is already > implemented in CPython. Not in documented public functions, but the > implementation exists. > > And PEP 580 also has a reference implementation: > https://github.com/jdemeyer/cpython/tree/pep580 > Yes I know. I described just "I didn't say wait for implement". -- INADA Naoki From gjcarneiro at gmail.com Thu Jul 5 09:14:35 2018 From: gjcarneiro at gmail.com (Gustavo Carneiro) Date: Thu, 5 Jul 2018 14:14:35 +0100 Subject: [Python-Dev] PEP 572: intended scope of assignment expression In-Reply-To: References: Message-ID: On Thu, 5 Jul 2018 at 13:43, Victor Stinner wrote: > Hi, > > My work (*) in the "Assignment expression and coding style: the while > True case" thread helped me to understand something about the > *intended* scope. > > While technically, assignment expressions keep the same scoping rules > than assignment statements, writing "if (x := func()): ..." or "while > (x := func()): ..." shows the "intented" scope of the variable. Even > if, as explained properly in the PEP, the scope is wider (for good > reasons) as "for line in file: ..." keeps line alive after the loop > (nothing new under the sun). It's something subtle that I missed at > the first read (of the code and the PEP), the difference is not > obvious. > > x = func() > if x: > ... # obviously use x > # do we still plan to use x here? > # it's non obvious just by reading the if > > versus > > if (x := func()): > ... # obviously use x > # ":=" in the if "announces" that usually x is no longer used > # here, even if technically x is still defined > I don't know if you're trying to propose something clever here, like "if (x := func()):" would assign to 'x' only inside the "then" body of the if, but IMHO that would be a terrible idea: 1. it makes AE harder to explain. Right now you can simply say AE works just like any regular assignment, except that you can use in an expression. If you start adding exceptions to the rule, things get harder and harder to explain; 2. it would probably have a cost in terms of performance: you'd have to create another scope for every "then" body of any if statement that contains an AE. And, again, more scopes = things harder to understand. -- Gustavo J. A. M. Carneiro Gambit Research "The universe is always one step beyond logic." -- Frank Herbert -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Jul 5 09:18:12 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 5 Jul 2018 23:18:12 +1000 Subject: [Python-Dev] PEP 572: intended scope of assignment expression In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 11:14 PM, Gustavo Carneiro wrote: > On Thu, 5 Jul 2018 at 13:43, Victor Stinner wrote: >> >> Hi, >> >> My work (*) in the "Assignment expression and coding style: the while >> True case" thread helped me to understand something about the >> *intended* scope. >> >> While technically, assignment expressions keep the same scoping rules >> than assignment statements, writing "if (x := func()): ..." or "while >> (x := func()): ..." shows the "intented" scope of the variable. Even >> if, as explained properly in the PEP, the scope is wider (for good >> reasons) as "for line in file: ..." keeps line alive after the loop >> (nothing new under the sun). It's something subtle that I missed at >> the first read (of the code and the PEP), the difference is not >> obvious. >> >> x = func() >> if x: >> ... # obviously use x >> # do we still plan to use x here? >> # it's non obvious just by reading the if >> >> versus >> >> if (x := func()): >> ... # obviously use x >> # ":=" in the if "announces" that usually x is no longer used >> # here, even if technically x is still defined > > > I don't know if you're trying to propose something clever here, like "if (x > := func()):" would assign to 'x' only inside the "then" body of the if, but > IMHO that would be a terrible idea: > > 1. it makes AE harder to explain. Right now you can simply say AE works > just like any regular assignment, except that you can use in an expression. > If you start adding exceptions to the rule, things get harder and harder to > explain; > 2. it would probably have a cost in terms of performance: you'd have to > create another scope for every "then" body of any if statement that contains > an AE. And, again, more scopes = things harder to understand. > The time machine strikes again. There were actually semantics defined in PEP 572 that would have done exactly this. They were sane, coherent, and internally consistent, but ultimately, not as useful as just maintaining the equivalence of "x = 2" and "x := 2". Subscope semantics aren't as impossible as you're implying here. ChrisA From J.Demeyer at UGent.be Thu Jul 5 10:53:05 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 5 Jul 2018 16:53:05 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention Message-ID: <5B3E30D1.3080706@UGent.be> Hello all, As discussed in some other threads ([1], [2]), we should discuss the METH_FASTCALL calling convention. For passing only positional arguments, a C array of Python objects is used, which is as fast as it can get. When the Python interpreter calls a function, it builds that C array on the interpreter stack: >>> from dis import dis >>> def f(x, y): return g(x, y, 12) >>> dis(f) 1 0 LOAD_GLOBAL 0 (g) 2 LOAD_FAST 0 (x) 4 LOAD_FAST 1 (y) 6 LOAD_CONST 1 (12) 8 CALL_FUNCTION 3 10 RETURN_VALUE A C array can also easily and efficiently be handled by the C function receiving it. So I consider this uncontroversial. The convention for METH_FASTCALL|METH_KEYWORDS is that keyword *names* are passed as a tuple and keyword *values* in the same C array with positional arguments. An example: >>> from dis import dis >>> def f(x, y, z): return f(x, foo=y, bar=z) >>> dis(f) 1 0 LOAD_GLOBAL 0 (f) 2 LOAD_FAST 0 (x) 4 LOAD_FAST 1 (y) 6 LOAD_FAST 2 (z) 8 LOAD_CONST 1 (('foo', 'bar')) 10 CALL_FUNCTION_KW 3 12 RETURN_VALUE This is pretty clever: it exploits the fact that ('foo', 'bar') is a constant tuple stored in f.__code__.co_consts. Also, a tuple can be efficiently handled by the called code: it is essentially a thin wrapper around a C array of Python objects. So this works well. The only case when this handling of keywords is suboptimal is when using **kwargs. In that case, a dict must be converted to a tuple. It looks hard to me to support efficiently both the case of fixed keyword arguments (f(foo=x)) and a keyword dict (f(**kwargs)). Since the former is more common than the latter, the current choice is optimal. In other words: I see nothing to improve in the calling convention of METH_FASTCALL. I suggest to keep it and make it public as-is. Jeroen. [1] https://mail.python.org/pipermail/python-dev/2018-June/153945.html [2] https://mail.python.org/pipermail/python-dev/2018-July/154251.html From J.Demeyer at UGent.be Thu Jul 5 11:07:10 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 5 Jul 2018 17:07:10 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> Message-ID: <5B3E341E.70402@UGent.be> On 2018-07-05 14:20, INADA Naoki wrote: > What you can do is "divide and conquer". Split PEP in small topics we > can focus. The PEP is already small and focused, I really did my best to make it as minimal as possible. I don't see a meaningful way to split it up even further. From vstinner at redhat.com Thu Jul 5 11:07:58 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 17:07:58 +0200 Subject: [Python-Dev] PEP 572: intended scope of assignment expression In-Reply-To: References: Message-ID: 2018-07-05 15:14 GMT+02:00 Gustavo Carneiro : > I don't know if you're trying to propose something clever here, like "if (x > := func()):" would assign to 'x' only inside the "then" body of the if, but > IMHO that would be a terrible idea: I don't propose to change the PEP 572. I'm trying to explain to myself where "if (var := expr): ..." would add anything to readers compared to "var = expr; if var: ...". I know that var is still valid after the if. I'm not asking to raise a syntax error or even emit a linter warning if var is used after the if. I'm just thinking loudly where the PEP 572 is appropriate. IMHO the following code should not use assignement expression. Good: help = self._root_section.format_help() if help: help = self._long_break_matcher.sub('\n\n', help) help = help.strip('\n') + '\n' return help Bad? if (help := self._root_section.format_help()): help = self._long_break_matcher.sub('\n\n', help) help = help.strip('\n') + '\n' return help IHMO using "help :=" here would send the wrong signal to the reader: as if help is not going to be used after the if, whereas it's the case ("return help"). Victor From python at mrabarnett.plus.com Thu Jul 5 11:39:15 2018 From: python at mrabarnett.plus.com (MRAB) Date: Thu, 5 Jul 2018 16:39:15 +0100 Subject: [Python-Dev] PEP 572: intended scope of assignment expression In-Reply-To: References: Message-ID: <40f6b2f3-b8a6-4a6c-7383-2d8423f0e79b@mrabarnett.plus.com> On 2018-07-05 16:07, Victor Stinner wrote: > 2018-07-05 15:14 GMT+02:00 Gustavo Carneiro : >> I don't know if you're trying to propose something clever here, like "if (x >> := func()):" would assign to 'x' only inside the "then" body of the if, but >> IMHO that would be a terrible idea: > > I don't propose to change the PEP 572. I'm trying to explain to myself > where "if (var := expr): ..." would add anything to readers compared > to "var = expr; if var: ...". I know that var is still valid after the > if. > > I'm not asking to raise a syntax error or even emit a linter warning > if var is used after the if. I'm just thinking loudly where the PEP > 572 is appropriate. > > IMHO the following code should not use assignement expression. > > Good: > > help = self._root_section.format_help() > if help: > help = self._long_break_matcher.sub('\n\n', help) > help = help.strip('\n') + '\n' > return help > > Bad? > > if (help := self._root_section.format_help()): > help = self._long_break_matcher.sub('\n\n', help) > help = help.strip('\n') + '\n' > return help > > IHMO using "help :=" here would send the wrong signal to the reader: > as if help is not going to be used after the if, whereas it's the case > ("return help"). > I think it's OK if it's: Do something, and if that succeeded, act on it. but not if it's: Do something, but if that failed, do something else. From vstinner at redhat.com Thu Jul 5 11:42:13 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 17:42:13 +0200 Subject: [Python-Dev] PEP 572: intended scope of assignment expression In-Reply-To: References: Message-ID: After "Assignment expression and coding style: the while True case", here is the part 2: analysis of the "if (var := expr): ..." case. 2018-07-05 14:20 GMT+02:00 Victor Stinner : > *intended* scope. I generated the giant pull request #8116 to show where I consider that "if (var := expr): ..." would be appropriate in the stdlib: https://github.com/python/cpython/pull/8116/files In short, replace: var = expr if var: ... with: if (var := expr): ... I used a script to replace "var = expr; if var: ..." with "if (var := expr): ...". I restricted my change to the simplest test "if var:", other conditions like "if var > 0:" are left unchaned to keep this change reviewable (short enough). The change is already big enough (62 files modified) to have enough examples! Then I validated each change manually: (*) I reverted all changes when 'var' is still used after the if. (*) I also reverted some changes like "var = regex.match(); if var: return var.group(1)", since it's already handled by my PR 8097: https://github.com/python/cpython/pull/8097/files (*) Sometimes, 'var' is only used in the condition and so has been removed in this change. Example: ans = self._compare_check_nans(other, context) if ans: return False return self._cmp(other) < 0 replaced with: if self._compare_check_nans(other, context): return False return self._cmp(other) < 0 (Maybe such changes should be addressed in a different pull request.) Below, some examples where I consider that assignment expressions give a value to the reader. == Good: site (straighforward) == env_base = os.environ.get("PYTHONUSERBASE", None) if env_base: return env_base replaced with: if (env_base := os.environ.get("PYTHONUSERBASE", None)): return env_base Note: env_base is only used inside the if block. == Good: datetime (more concise code) == New code: def isoformat(self, timespec='auto'): s = _format_time(self._hour, self._minute, self._second, self._microsecond, timespec) if (tz := self._tzstr()): s += tz return s This example shows the benefit of the PEP 572: remove one line without making the code worse to read. == Good: logging.handlers == def close(self): self.acquire() try: sock = self.sock if sock: self.sock = None sock.close() logging.Handler.close(self) finally: self.release() replaced with: def close(self): self.acquire() try: if (sock := self.sock): self.sock = None sock.close() logging.Handler.close(self) finally: self.release() == Good: doctest == New code: # Deal with exact matches possibly needed at one or both ends. startpos, endpos = 0, len(got) if (w := ws[0]): # starts with exact match if got.startswith(w): startpos = len(w) del ws[0] else: return False if (w := ws[-1]): # ends with exact match if got.endswith(w): endpos -= len(w) del ws[-1] else: return False ... This example is interesting: the 'w' variable is reused, but ":=" announces to the reader that the w is only intended to be used in one if block. == Good: csv (reuse var) == New code: n = groupindex['quote'] - 1 if (key := m[n]): quotes[key] = quotes.get(key, 0) + 1 try: n = groupindex['delim'] - 1 key = m[n] except KeyError: continue if key and (delimiters is None or key in delimiters): delims[key] = delims.get(key, 0) + 1 As for doctest: "key := ..." shows that this value is only used in one if block, but later key is reassigned to a new value. == Good: difflib == New code using (isjunk := self.isjunk): # Purge junk elements self.bjunk = junk = set() if (isjunk := self.isjunk): for elt in b2j.keys(): if isjunk(elt): junk.add(elt) for elt in junk: # separate loop avoids separate list of keys del b2j[elt] -*-*-*- == Borderline? sre_parse (two conditions) == code = CATEGORIES.get(escape) if code and code[0] is IN: return code replaced with: if (code := CATEGORIES.get(escape)) and code[0] is IN: return code The test "code[0] is IN" uses 'code' just after it's defined on the same line. Maybe it is surprising me, since I'm not sure to assignment expressions yet. -*-*-*- == BAD! argparse (use after if) == Ok, now let's see cases where I consider that assignment expressions are inappropriate! Here is a first example. help = self._root_section.format_help() if help: help = self._long_break_matcher.sub('\n\n', help) help = help.strip('\n') + '\n' return help 'help' is used after the if block. == BAD! fileinput (use after if) == line = self._readline() if line: self._filelineno += 1 return line if not self._file: return line 'line' is used after the first if block. == BAD! _osx_support (reuse var, use after if) == def _supports_universal_builds(): osx_version = _get_system_version() if osx_version: try: osx_version = tuple(int(i) for i in osx_version.split('.')) except ValueError: osx_version = '' return bool(osx_version >= (10, 4)) if osx_version else False Surprising reusage of the 'osx_version' variable, using ":=" would more the code even more confusing (and osx_version is used after the if). Victor From vano at mail.mipt.ru Thu Jul 5 11:48:56 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 18:48:56 +0300 Subject: [Python-Dev] PEP 572: intended scope of assignment expression In-Reply-To: References: Message-ID: <5bcac6ee-e33a-dc95-43c9-c8a9285987bb@mail.mipt.ru> On 05.07.2018 15:20, Victor Stinner wrote: > Hi, > > My work (*) in the "Assignment expression and coding style: the while > True case" thread helped me to understand something about the > *intended* scope. > > While technically, assignment expressions keep the same scoping rules > than assignment statements, writing "if (x := func()): ..." or "while > (x := func()): ..." shows the "intented" scope of the variable. Even > if, as explained properly in the PEP, the scope is wider (for good > reasons) as "for line in file: ..." keeps line alive after the loop > (nothing new under the sun). It's something subtle that I missed at > the first read (of the code and the PEP), the difference is not > obvious. > > x = func() > if x: > ... # obviously use x > # do we still plan to use x here? > # it's non obvious just by reading the if > > versus > > if (x := func()): > ... # obviously use x > # ":=" in the if "announces" that usually x is no longer used > # here, even if technically x is still defined The construct for temporary variables is `with'. `if' carries no such implications. > See my match/group PR for more concrete examples: > https://github.com/python/cpython/pull/8097/files > > I understand the current PEP 572 rationale as: assignment expressions > reduces the number of lines and the indentation level... pure syntax > sugar. > > IMHO this "intended" scope is a much better way to sell assignment > expressions than the current rationale. In fact, it's explained later > very quickly in the PEP: > https://www.python.org/dev/peps/pep-0572/#capturing-condition-values > > But it could be better explained than just "good effect in the header > of an if or while statement". > > The PEP contains a good example of the intended scope: > > if pid := os.fork(): > # Parent code > # pid is valid and is only intended to be used in this scope > ... # use pid > else: > # Child code > # pid is "invalid" (equal to zero) > ... # don't use pid > # since this code path is common to parent and child, > # the pid is considered invalid again here > # (since the child does also into this path) > ... # don't use pid > > > (*) My work: my current 3 pull requests showing how assignment > expressions can be > used in the stdlib: > > while True: https://github.com/python/cpython/pull/8095/files > match/group: https://github.com/python/cpython/pull/8097/files > list comp: https://github.com/python/cpython/pull/8098/files > > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru From python-dev at mgmiller.net Thu Jul 5 12:50:32 2018 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 5 Jul 2018 09:50:32 -0700 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: <20180705062005.GA7318@ando.pearwood.info> References: <51a528fa-e602-4441-501e-d700661f2795@mail.mipt.ru> <20180705062005.GA7318@ando.pearwood.info> Message-ID: On 2018-07-04 23:20, Steven D'Aprano wrote: > It simply isn't true that modern languages are moving away from > assignment expressions. Some are. Some aren't. Even those that don't > support assignment expressions in general usually support special syntax > to allow it in a few contexts. The older post you are referring and this thread describe the exact situation in your last sentence. The limited assignment "compromise" is a common solution nowadays, just as this thread discusses. Repeating "it won't work" when it has been shown to work well in several languages is nonsensical. Yes, the available solutions are not perfect, but I still maintain "as" is less disruptive and doesn't reverse 25 year-old design choices, but rather works with them. -Mike From python-dev at mgmiller.net Thu Jul 5 13:07:59 2018 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 5 Jul 2018 10:07:59 -0700 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: References: Message-ID: <4d00ba18-e495-2093-ad49-3cdad9d92a72@mgmiller.net> On 2018-07-04 17:22, Chris Angelico wrote: > - the "if expr as name:" syntax is able to handle only the tiniest > proportion of cases, because many MANY situations require a condition > after that. You can't write this, for instance: > > if f(x) as spam < 0: > print(spam) The original use cases didn't ask for these compound conditions. In fact many of the other threads this week are advising folks to break up an expression with compound conditions due to lack of readability. The common cases described: - compute value once in a comprehension - loop and a half (reading file, socket) - common regex match More complex cases can be handled the old way. > Python uses "as NAME" for things that > are quite different from this, so it's confusing), It's less confusing, and limited. No one bats an eyelash after using "as" day after day in Python and SQL. Good day, -Mike From python-dev at mgmiller.net Thu Jul 5 13:21:52 2018 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 5 Jul 2018 10:21:52 -0700 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: <329901ab-62d1-9ec4-29c9-cfacef321e5f@mail.mipt.ru> References: <51a528fa-e602-4441-501e-d700661f2795@mail.mipt.ru> <20180705062005.GA7318@ando.pearwood.info> <329901ab-62d1-9ec4-29c9-cfacef321e5f@mail.mipt.ru> Message-ID: <4d795668-50d0-1371-e7ac-adfb3354a065@mgmiller.net> On 2018-07-05 04:28, Ivan Pozdeev via Python-Dev wrote: > This is as intended. > I wanted to show my summary and Chris' refuttal, with links to both original > posts. Because my letter is much shorter than the originals while carrying the > same message. Also to show that I've made the same mistake, which puts things in > perspective: how an outsider could get the wrong idea. There will always be a long tail of new languages doing any and everything. Which new languages are actually being used? The more limited ones. Static typing, fewer foot-guns. Arguably Elixir should have been in the original list, but it is practically unknown compared to Kotlin, for example. -Mike From tim.peters at gmail.com Thu Jul 5 13:41:07 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 5 Jul 2018 12:41:07 -0500 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: <20180705064746.GB7318@ando.pearwood.info> Message-ID: [Victor Stinner] > FYI I'm trying to use assignment expressions on the stdlib because > *all* examples of the PEP 572 look artificial to me. > All the examples in my Appendix A were derived from real. pre-existing code (although mostly my own). It's turned out that several others with your complaint above never read the PEP after that Appendix was added, so it's possible that applies to you too. > Like "group = re.match(data).group(1) if re.match(data) else None" > which is followed by "(TODO: Include Guido's evidence, and do a more > systematic search.)" I cannot find such inefficient code in the > stdlib. I'm not saying that nobody writes code like that, just that > developers with a good Python expertise would avoid to write such > code. > Sure. It happens, but it's rare. As I've said many times already, I was slightly surprised to conclude that the PEP was a win, but very surprised to conclude it was the frequent dead- obvious little wins that added up to its strongest case. But that's mostly from staring at my code, and there's really nobody on Earth better qualified to make my tradeoffs than I am ;-) I don't discount Guido's impressions either. While it's unlikely you're going to find needlessly inefficient code in my Python, or the std lib, to save a line, I'm not surprised at all that he found people writing worse (on several counts - readability, maintainability, and speed) code _ just_ to save a line in "industrial Python". Rather than dismiss the value for them because they're not as expert as I am, or as our std lib authors have been, I take it as evidence that the PEP provides value to others in ways that don't really apply to bona-fide-Python-expert code. I suspect, but don't know, Guido is more moved by _that_ factor than by the dead obvious little wins in experts' code. "filtered_data = [y for x in data if (y := f(x)) is not None]" also seems artificial. In the 711,617 lines of Python code of the stdlib, I > only found *one* example: > Well, you can't have it both ways. That is, you can't fret that the feature will be massively used while simultaneously claiming there are almost no plausible uses at all ;-) > ... And I also only found a single for loop which can be converted to a > list comprehension thanks to assignement expression. > lines = [] > for raw_line in raw_lines: > match = line_pat.search(raw_line.strip()) > if match: > lines.append(match.group(1, 2)) > And I don't see it as "an obvious win" _to_ change it, so wouldn't rewrite it anyway. Then again, my bar for "obvious win" may be higher than others'. [Tim] > > Wholesale changes to the std lib are unlikely to happen regardless. > Broad > > patches just to spell things differently without _need_ are discouraged. > > So PEP 572 is purely syntax sugar? It doesn't bring anything to the > stdlib for example? > Insert the standard tedious observation about "Turing complete" here. I'm too old to bear it again ;-) Patches to make widespread more-or-less mindless changes are _always_ frowned on, unless it's truly necessary (because, e.g., a language change means the existing idiom will no longer work in the next release, or the existing idiom is known to be 10x slower than a new alternative, or ...). It clutters the history, greatly obscures who is "really responsible" for the line that just showed up in a critical-failure traceback, and any code churn risks introducing bugs. Especially when people with little knowledge of a module are changing it. No matter how careful they are, they suffer brain farts when making widespread changes to code they've basically never seen before, and do introduce errors. That's empirical historic fact. Instead module experts introduce such changes incrementally while going about their regular business, if they so choose. If, for example, Raymond despises assignment expressions. it's fine by me if itertools and random.py (two modules for which he's the primary maintainer) never use them - and it's a Bad Thing if someone else forces them on him without his approval. In any case, I'd be -1 on your current approach regardless, until you drop the outer parens in if (name := f()): etc. It reads better as the intended: if name := f(): and that can matter. There _is_ a side effect going on here, and on general principle it's better to avoid "hiding it" inside parentheses unless there's a need to. There's no possible way to misread the unparenthesized form in this context, so there's no point at all to adding redundant parens. "But some earlier version of the PEP may or may not have required them - I don't recall" is no longer a real point ;-) My current 3 pull requests showing how assignment expressions can be > used in the stdlib: > > while True: https://github.com/python/cpython/pull/8095/files > match/group: https://github.com/python/cpython/pull/8097/files > list comp: https://github.com/python/cpython/pull/8098/files > And again I'm very happy you're doing this! Just drop the uselessly distracting parens cluttering the dead obvious little wins ;-) > Right now, I'm still not really excited by the new code. > Me neiither - but I'm not looking for excitement here, and never was. There's nothing exciting about little wins. They're ... little ;-) I'm more _intrigued_ by the possibilities created by assignment expression targets living in the scope enclosing the outermost comprehension/genexp, but I don't expect to find existing code just waiting to be slightly rewritten to take advantage of that. Although I certainly have "search loop" code that could be rewritten with some real effort to exploit the possibility of "exporting" the value that causes an `any(genexp)` or `all(genexp)` to terminate early. If assignment expressions had been there, I would have written them with any/all from the start (even in Python 2, for-target names are "hidden" in genexps). > If you spotted other parts of the stdlib where assignment expressions > would be appropriate, please tell me and I will try to write more pull > requests :-) I will if it comes up, but don't expect that. I did my code spelunking already, and stopped when I ceased learning anything new from it. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Thu Jul 5 13:52:24 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Thu, 5 Jul 2018 20:52:24 +0300 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: References: Message-ID: <8c66ac44-a56e-7fea-65df-4e4a27fd476b@mail.mipt.ru> On 05.07.2018 3:22, Chris Angelico wrote: > Python uses "as NAME" for things that > are quite different from this, so it's confusing I wrote in https://mail.python.org/pipermail/python-dev/2018-June/154066.html that this is easily refutable. Looks like not for everybody. Okay, here goes: The constructs that currently use `as' are: * import module as m * except Exception as e: * with expr as obj: * In `with', there's no need to assign both `expr' and its __enter__() result -- because the whole idea of `with' is to put the object through `__enter__', and because a sane `__enter__()' implementation will return `self' anyway (or something with the same semantic -- i.e. _effectively_ `self'). But just in case, the double-assignment can be written as: with (expr as obj) as ctx_obj: by giving "as" lower priority than `with'. As I said, the need for this is nigh-nonexistent. * `import' doesn't allow expressions (so no syntactic clash here), but the semantic of "as" here is equivalent to the AE, so no confusion here. * Same goes for `except`: doesn't accept expressions, same semantic. So, with "as" only `with' becomes the exception -- and an easily explainable one since its whole purpose is to implicitly call the context manager interface. From steve at pearwood.info Thu Jul 5 14:21:30 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 6 Jul 2018 04:21:30 +1000 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: <8c66ac44-a56e-7fea-65df-4e4a27fd476b@mail.mipt.ru> References: <8c66ac44-a56e-7fea-65df-4e4a27fd476b@mail.mipt.ru> Message-ID: <20180705182130.GD7318@ando.pearwood.info> On Thu, Jul 05, 2018 at 08:52:24PM +0300, Ivan Pozdeev via Python-Dev wrote: > * Same goes for `except`: doesn't accept expressions, same semantic. py> def make_exception(arg): ... return ValueError if arg else TypeError ... py> expr = [make_exception] py> try: ... 1+"1" ... except expr[0](None) as err: ... print("caught", type(err)) ... print(err) ... caught unsupported operand type(s) for +: 'int' and 'str' -- Steve From alexander.belopolsky at gmail.com Thu Jul 5 14:26:40 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 5 Jul 2018 14:26:40 -0400 Subject: [Python-Dev] Removal of install_misc command from distutils Message-ID: I started porting my project [1] to Python 3.7 and came across bpo-29218: "The unused distutils install_misc command has been removed." [2] Historically, the distutils package was very conservative about changes because many 3rd party packages extended it in ways unforeseen by the Python core developers. As far as I can tell, this removal was done without a deprecation period or any public discussion. The comment above the command class [3] was there for 18 years and promised to "keep it around for the time being." Why did it suddenly become necessary to remove it in 3.7? Is shedding 20 lines of code really worth the risk of breaking user code? [1]: https://github.com/KxSystems/pyq [2]: https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals [3]: https://github.com/python/cpython/commit/aae45f93e7b7708deb1ce9d69b58fa029106613d -------------- next part -------------- An HTML attachment was scrubbed... URL: From koyukukan at gmail.com Thu Jul 5 15:16:05 2018 From: koyukukan at gmail.com (Rin Arakaki) Date: Fri, 6 Jul 2018 04:16:05 +0900 Subject: [Python-Dev] How about integrating "as" semantics and postpone PEP 572 to Python 4.0 Message-ID: Hi, I bring a strong proposal that is prematurely rejected but indeed the most likely to be accepted to our community. That is to select `as` spelling not `:=` and modify `with` and `except` statements. Here is this. * Below is copy-pasted from my gist: https://gist.github.com/rnarkk/cdabceaedbdb498d99802a76cc08b549 # `as` assignment expression ```python # This STATEMENT(EXPR_0 as NAME_0, EXPR_1 as NAME_2, ..., EXPR_n as NAME_n) # can always be replaced by NAME_0 = EXPR_0 NAME_1 = EXPR_1 ... NAME_n = EXPR_n STATEMENT(NAME_0, NAME_1, ..., NAME_n) # except `import` statement since it's special from the beginning which means no `EXPR` cannot be the left hand of `as` in its statement but simply `NAME`: `import NAME_0 as NAME_1` # This interpretation above is valid no matter how `EXPR_i` uses `NAME_j` (i > j). # When you write EXPR as NAME_0, NAME_1 # it's interpreted as (EXPR as NAME_0), NAME_1 # TODO Should unpacking is allowed by this? EXPR as (NAME_0, NAME_1) # `EXPR as NAME` itself can be `EXPR` and it returns just `EXPR` in `EXPR as NAME` which means you can write ((EXPR as NAME_0) as NAME_1) ... as NAME_n # or simply like this even at the top level since it's determininable. EXPR as NAME_0 as NAME_1 ... as NAME_n NAME_0 = EXPR as NAME_1 ... as NAME_n # Also you can write f(x=EXPR as NAME) # since this is valid and `EXPR as NAME` can be `EXPR`. f(x=EXPR) # And also this is passible. if (EXPR as NAME).method(): ... # The `EXPR` in `EXPR as NAME` is matched as long as possible which means if 0 < 1 as x: ... # is interpreted as if (0 < 1) as x: ... # not if 0 < (1 as x): ... # but also `EXPR` is separated by `,` which means EXPR_0, EXPR_1 as NAME # is interpreted as EXPR_0, (EXPR_1 as NAME) # rather than (EXPR_0, EXPR_1) as NAME # even when used `as` assignment expression in list comprehension, # you can apply the same rules above first by putting it to `for` loop form. # There is no equivalence to type annotation and augmented assignment. ``` # `with` statement - `with` statement will no longer responsible for passing returned value from `__enter__` to the right hand of `as` in its statement and merely call `__enter__` when enter the statement and `__exit__` when exit from it an - Context manager can be renamed simply to context since it will no longer be manager but context itself. Context object has status of the context and encapsulates it. # `except` statement - `except` statement will no longer responsible for passing instance of the right hand of `as` in its statement. - Exceptions must be instanciated and also it must be confirmed otherwise `except` statement could rewrite the class globally. Thanks, Rin Arakaki -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 15:57:49 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 12:57:49 -0700 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3E341E.70402@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3E341E.70402@UGent.be> Message-ID: Would it be possible to get outside experts to help? Like Cython or numpy devs? On Thu, Jul 5, 2018 at 8:09 AM Jeroen Demeyer wrote: > On 2018-07-05 14:20, INADA Naoki wrote: > > What you can do is "divide and conquer". Split PEP in small topics we > > can focus. > > The PEP is already small and focused, I really did my best to make it as > minimal as possible. I don't see a meaningful way to split it up even > further. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-dev at mgmiller.net Thu Jul 5 15:57:59 2018 From: python-dev at mgmiller.net (Mike Miller) Date: Thu, 5 Jul 2018 12:57:59 -0700 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: <8c66ac44-a56e-7fea-65df-4e4a27fd476b@mail.mipt.ru> References: <8c66ac44-a56e-7fea-65df-4e4a27fd476b@mail.mipt.ru> Message-ID: <36635cbf-993a-e27f-9926-87fb08aaf600@mgmiller.net> On 2018-07-05 10:52, Ivan Pozdeev via Python-Dev wrote: > Perhaps, however I'm not advocating using "EXPR as NAME" with "with" as it wouldn't be useful enough, only limited to if/while/comp. -Mike From chris.barker at noaa.gov Thu Jul 5 17:03:36 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Thu, 5 Jul 2018 14:03:36 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: On Wed, Jul 4, 2018 at 7:20 AM, David Mertz wrote: > > That said, this is a silly game either way. And even though you CAN > (sometimes) bind in an expression pre-572, that's one of those perverse > corners that one shouldn't actually use. > not only shouldn't by hardly anyone ever does / would. A lot of the argument for this feature is that it doesn't really let you do things you couldn't do before (like manipulate the local namespace in side comprehensions, or the nonocal one, or...) But this IS a silly game -- Python is highly dynamic, you can do all sorts of metaprogamming tricks. I'm pretty sure you can even alter the byte code in a running interpreter (but only pretty sure, 'cause why would I ever try to do that?) But the point is not that you can do tricky namespace manipulations now, it's that you need to do advanced (and obvious) thinks like call locals() or globals() or use nonlocal, or... I don't think these concerns have been ignored, but I also don't think that I've heard anyone on the pro side say something along the line of: "Yes, this does add a significant complication to the language, but we think it will be unlikely to be mis-used in confusing ways, and that complication is worth it." Rather, I've heard a lot of "but you can already do that" or "but we added [ternary expressions, augmented assignment, f-strings, ...]" And none of those add *complication* to the language itself -- they add one more feature that needs to be looked up when you encounter it -- but only effect the line of code where there are used. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Thu Jul 5 17:47:44 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 5 Jul 2018 23:47:44 +0200 Subject: [Python-Dev] Assignment expression and coding style: the while True case In-Reply-To: References: Message-ID: Hi, I wrote more pull requests in the meanwhile to see how assignment expressions could be used in the standard library. I combined my 5 PR into a new single PR to see all changes at once: https://github.com/python/cpython/pull/8122/files Again, all these PR must not be merged. I only wrote them to discuss when it's appropriate or not to use assingment expressions. Victor From ncoghlan at gmail.com Thu Jul 5 18:06:23 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Jul 2018 08:06:23 +1000 Subject: [Python-Dev] Checking that PEP 558 (defined semantics for locals()) handles assignment expressions In-Reply-To: References: Message-ID: On Thu., 5 Jul. 2018, 1:21 am Guido van Rossum, wrote: > Correct, there shouldn't be any additional corner cases for your PEP due > to inline assignments. We're not introducing new scopes nor other runtime > mechanisms; the target of an inline assignment is either a global or a cell > (nonlocal) defined at a specific outer level. > Cool. I'll still review the PEP to see if it makes sense to mention this as a side note anywhere, but it may turn out to make more sense to simply not mention it at all. > What I wish we had (quite independent from PEP 572) is a way in a debugger > to evaluate a comprehension that references a local in the current stack > frame. E.g. here's something I had in a pdb session yesterday: > > (Pdb) p [getattr(context.context, x) for x in dir(context.context)] > *** NameError: name 'context' is not defined > (Pdb) global cc; cc = context > (Pdb) p [getattr(cc.context, x) for x in dir(cc.context)] > [, ............] > (Pdb) > > The first attempt failed because the outer `context` was a local variable > in some function, and pdb uses eval() to evaluate expressions. > Perhaps pdb should be passing something like "ChainMap(frame.f_locals, frame.f_globals)" to the eval call as its global namespace when the current frame uses optimized local variable references? That way even lexically nested scopes inside the eval call will all be able to see the current local variables via dynamic name lookup, even though they still won't see them as lexical closure references. Cheers, Nick. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jul 5 18:32:40 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Jul 2018 08:32:40 +1000 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> Message-ID: On Thu., 5 Jul. 2018, 10:23 am Steve Dower, wrote: > On 04Jul2018 1518, Tim Peters wrote: > > The only new thing is specifying the scope of `a`, where "local to f" > > means exactly the same thing as for any other name local to a function > > today. So far as the PEP semantics go, it doesn't even matter whether > > an implementation _does_ implement some form of closure as such. It > > just has to provide the visible semantics of _lexically_ nested scopes > > with indefinite extent, by whatever means it likes best. That's what > > "local to f" means (and has meant all along - well, since lexically > > nested scopes were first introduced). > > In that case, please provide more examples of how it should work when > the assignment expression appears to define a variable in a scope that > is not on the call stack. > > Whether intentional or not, there will be changes to how and when names > are resolved. The specification should provide enough information to > determine the preferred behaviour, so we can tell the difference between > intention changes and implementation bugs. > > For example, what should be returned from this function? > > >>> A = 0 > >>> def f(x): > ... if x: > ... [A := i for i in [1]] > ... return A > > As far as I can tell, the closest current equivalent will not compile: > > >>> A = 0 > >>> def f(x): > ... if x: > ... def g(): > ... nonlocal A > ... A = 1 > ... g() > ... return A > ... > File "", line 4 > SyntaxError: no binding for nonlocal 'A' found > > Is this the equivalent behaviour you want? Or do you want an > UnboundLocalError when calling f(0)? Or do you want the global A to be > returned? How should we approach decision making about these cases as we > implement this? The PEP does not provide enough information for me to > choose the right behaviour here, and I argue that it should. > Guido did fully specify this in his post on "__parentlocal" scoping, in response to my request that this be clearly spelled out in the PEP (that specification just hasn't been rolled back into the PEP yet). While Tim's correct that the underlying runtime semantics here aren't new (which is why PEP 558 doesn't need to change), there's a new requirement imposed on a Python compiler's symbol table analysis pass to see "A := expr" in a comprehension or generator expression scope and interpret that as equivalent to either: 1. An implied "global A" in the child scope if the parent scope is the module scope, or A is explicitly declared as global in the parent scope 2. An implied "nonlocal A" in the child scope if the parent scope is a function scope and A is not defined as global in that scope. If A is not already declared as local or nonlocal in the parent scope, then it is implicitly declared as local in that scope with no associated annotation (akin to "if 0: for A in (): pass") 3. A compile time error if the parent scope is a class scope (since we don't have any existing scope declaration semantics that can be used to make that case work sensibly) I'm still wondering if it might make sense to define a new "TargetScopeError" subclass of SyntaxError for that last case, since it isn't the assignment expression syntax itself that's the problem: it's where that expression is located. Cheers, Nick. > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/ncoghlan%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 18:40:01 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 15:40:01 -0700 Subject: [Python-Dev] How about integrating "as" semantics and postpone PEP 572 to Python 4.0 In-Reply-To: References: Message-ID: Thanks you for writing up a proposal. There have been many proposals made, including 'EXPR as NAME', similar to yours. It even has a small section in the PEP: https://www.python.org/dev/peps/pep-0572/#alternative-spellings. It's really hard to choose between alternatives, but all things considered I have decided in favor of `NAME := EXPR` instead. Your efforts are appreciated but you would just be wasting your time if you wrote a PEP. If you're interested in helping out, would you be interested in working on the implementation of PEP 572? On Thu, Jul 5, 2018 at 12:21 PM Rin Arakaki wrote: > Hi, > > I bring a strong proposal that is prematurely rejected but indeed the most likely to be accepted to our community. > That is to select `as` spelling not `:=` and modify `with` and `except` statements. Here is this. > > * Below is copy-pasted from my gist: https://gist.github.com/rnarkk/cdabceaedbdb498d99802a76cc08b549 > > > # `as` assignment expression > > ```python > # This > STATEMENT(EXPR_0 as NAME_0, EXPR_1 as NAME_2, ..., EXPR_n as NAME_n) > > # can always be replaced by > NAME_0 = EXPR_0 > NAME_1 = EXPR_1 > ... > NAME_n = EXPR_n > STATEMENT(NAME_0, NAME_1, ..., NAME_n) > > # except `import` statement since it's special from the beginning which means no `EXPR` cannot be the left hand of `as` in its statement but simply `NAME`: `import NAME_0 as NAME_1` > > # This interpretation above is valid no matter how `EXPR_i` uses `NAME_j` (i > j). > > # When you write > EXPR as NAME_0, NAME_1 > > # it's interpreted as > (EXPR as NAME_0), NAME_1 > > # TODO Should unpacking is allowed by this? > EXPR as (NAME_0, NAME_1) > > # `EXPR as NAME` itself can be `EXPR` and it returns just `EXPR` in `EXPR as NAME` which means you can write > ((EXPR as NAME_0) as NAME_1) ... as NAME_n > > # or simply like this even at the top level since it's determininable. > EXPR as NAME_0 as NAME_1 ... as NAME_n > > NAME_0 = EXPR as NAME_1 ... as NAME_n > > # Also you can write > f(x=EXPR as NAME) > > # since this is valid and `EXPR as NAME` can be `EXPR`. > f(x=EXPR) > > # And also this is passible. > if (EXPR as NAME).method(): ... > > # The `EXPR` in `EXPR as NAME` is matched as long as possible which means > if 0 < 1 as x: ... > > # is interpreted as > if (0 < 1) as x: ... > > # not > if 0 < (1 as x): ... > > # but also `EXPR` is separated by `,` which means > EXPR_0, EXPR_1 as NAME > > # is interpreted as > EXPR_0, (EXPR_1 as NAME) > > # rather than > (EXPR_0, EXPR_1) as NAME > > # even when used `as` assignment expression in list comprehension, > # you can apply the same rules above first by putting it to `for` loop form. > > # There is no equivalence to type annotation and augmented assignment. > ``` > > > # `with` statement > > - `with` statement will no longer responsible for passing returned value from `__enter__` to the right hand of `as` in its statement and merely call `__enter__` when enter the statement and `__exit__` when exit from it an > - Context manager can be renamed simply to context since it will no longer be manager but context itself. Context object has status of the context and encapsulates it. > > # `except` statement > > - `except` statement will no longer responsible for passing instance of the right hand of `as` in its statement. > - Exceptions must be instanciated and also it must be confirmed otherwise `except` statement could rewrite the class globally. > > > Thanks, > Rin Arakaki > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Jul 5 18:44:46 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 6 Jul 2018 08:44:46 +1000 Subject: [Python-Dev] PEP 572 semantics: all capabilities of the assignment statement In-Reply-To: References: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> Message-ID: On Thu., 5 Jul. 2018, 3:17 pm Guido van Rossum, wrote: > Let me be slightly contrarian. :-) > > On Wed, Jul 4, 2018 at 9:12 PM Chris Angelico wrote: > >> Definitely against augmentation, for several reasons: >> >> 1) Spelling - should it be :+= or +:= ? >> > > That one's easy. As Nick's (withdrawn) PEP 577 shows it should be simply > `+=`. > > >> 2) Is the result of the expression the modified value or the original? >> > > Someone (sadly I forget who) showed, convincingly (to me anyways :-) that > it should return whatever the `__iadd__` method returns, or (if there isn't > one) the result of `a = a + b`. > I think I had it as an open question in one of the earlier drafts of PEP 577. The subsequent rationale for it returning the modified value was that we have this existing equivalence at the statement level: a += b a = operator.iadd(a, b) So the natural expression level semantics would be: a := operator.iadd(a, b) Cheers, Nick. > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 18:57:36 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 15:57:36 -0700 Subject: [Python-Dev] Checking that PEP 558 (defined semantics for locals()) handles assignment expressions In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 3:06 PM Nick Coghlan wrote: > > > On Thu., 5 Jul. 2018, 1:21 am Guido van Rossum, wrote: > >> Correct, there shouldn't be any additional corner cases for your PEP due >> to inline assignments. We're not introducing new scopes nor other runtime >> mechanisms; the target of an inline assignment is either a global or a cell >> (nonlocal) defined at a specific outer level. >> > > Cool. I'll still review the PEP to see if it makes sense to mention this > as a side note anywhere, but it may turn out to make more sense to simply > not mention it at all. > Would it be helpful if PEP 572 spelled out the translations of some common and corner cases? It's difficult to present a general rule for translating because of the required static analysis (not impossible but it would be low information density compared to the spec that's already there), but it might help clarify the spec for implementers who aren't from Mars. > What I wish we had (quite independent from PEP 572) is a way in a debugger >> to evaluate a comprehension that references a local in the current stack >> frame. E.g. here's something I had in a pdb session yesterday: >> >> (Pdb) p [getattr(context.context, x) for x in dir(context.context)] >> *** NameError: name 'context' is not defined >> (Pdb) global cc; cc = context >> (Pdb) p [getattr(cc.context, x) for x in dir(cc.context)] >> [, ............] >> (Pdb) >> >> The first attempt failed because the outer `context` was a local variable >> in some function, and pdb uses eval() to evaluate expressions. >> > > Perhaps pdb should be passing something like "ChainMap(frame.f_locals, > frame.f_globals)" to the eval call as its global namespace when the current > frame uses optimized local variable references? That way even lexically > nested scopes inside the eval call will all be able to see the current > local variables via dynamic name lookup, even though they still won't see > them as lexical closure references. > That sounds intriguing, but I can't easily get it to work. I get either "TypeError: globals must be a real dict; try eval(expr, {}, mapping)", when I pass that for globals, or the same error as before, "NameError: name 'a' is not defined", when I follow that advice and pass the ChainMap instance as the locals. Plus it(ironically, perhaps :-) it screws up the output of "p locals()". -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 19:11:43 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 16:11:43 -0700 Subject: [Python-Dev] PEP 572, VF/B, and "Shark Jumping" In-Reply-To: References: Message-ID: Please, Mike, can you stop? The race is over and your horse has lost. I really value all the input I've received during the months of discussion (including your research into what other languages do), but in the end my "evaluation function" (to use somewhat hip lingo :-) is different from yours. For a while I seriously considered "EXPR as NAME", but when comparing examples written in one style vs. another I just liked "NAME := EXPR" better every time. In addition to the problems with "with" (which everyone has read about already) and the general rule that "as" is preceded by specific syntax that predicts it (i.e. "import", "except" or "with"), I found that when skimming the code it was easier to miss the definition of NAME with the "as" form than with the ":=" variant. There was also someone who posted that they believe that the problems with evaluation order (alluded to in the PEP) would be solved by the "as" form. However that's not the case -- in fact I believe that Python would have to generate exactly the same bytecode for "EXPR as NAME" as for " NAME := EXPR", since in both cases it comes down to "LOAD EXPR; STORE NAME" (in pseudo bytecode). So neither form supports things like "x == (x := f())" since this comes down to LOAD x LOAD f CALL STORE x COMPARE with either syntax variant (again, in pseudo bytecode). On Wed, Jul 4, 2018 at 5:09 PM Mike Miller wrote: > Recently on Python-Dev: > > On 2018-07-03 15:24, Chris Barker wrote: > > On Tue, Jul 3, 2018 at 2:51 PM, Chris Angelico > On Wed, Jul 4, 2018 at 7:37 AM, Serhiy Storchaka < > storchaka at gmail.com> > > > > > I believe most Python users are not > > > professional programmers -- they are sysadmins, scientists, > hobbyists > > > and kids -- > > > > [citation needed] > > > > fair enough, but I think we all agree that *many*, if not most, Python > users > > are "not professional programmers". While on the other hand everyone > involved > > in discussion on python-dev and python-ideas is a serious (If not > > "professional") programmer. > > > Python Audience - wants clarity: > > Not sure I'd say that most users are not professionals, but one major > strength > of Python is its suitability as a teaching language, which enlarges the > community every year. > > Additionally, I have noticed a dichotomy between prolific "C programmers" > who've > supported this PEP and many Python programmers who don't want it. While > C-devs > use this construct all the time, their stereotypical Python counterpart is > often > looking for simplicity and clarity instead. That's why we're here, folks. > > > Value - good: > > Several use cases are handled well by PEP 572. However it has been noted > that > complexity must be capped voluntarily relatively early?or the cure soon > becomes > worse than the disease. > > > Frequency - not much: > > The use cases for assignment-expressions are not exceedingly common, > coming up > here and there. Their omission has been a very mild burden and we've done > without for a quarter century. > > Believe the authors agreed that it won't be used too often and won't > typically > be mis- or overused. > > > New Syntax - a high burden: > > For years I've read on these lists that syntax changes must clear a high > threshold of the (Value*Frequency)/Burden (or VF/B) ratio. > > Likewise, a few folks have compared PEP 572 to 498 (f-strings) which some > former > detractors have come to appreciate. Don't believe this comparison applies > well, > since string interpolation is useful a hundred times a day, more concise, > clear, > and runs faster than previous functionality. Threshold was easily cleared > there. > > > Conclusion: > > An incongruous/partially redundant new syntax to perform existing > functionality > more concisely feels too low on the VF/B ratio IMHO. Value is good though > mixed, frequency is low, and burden is higher than we'd like, resulting in > "meh" > and binary reactions. > > Indeed many modern languages omit this feature specifically in an effort > to > reduce complexity, ironically citing the success of Python in support. > Less is > more. > > > Compromise: > > Fortunately there is a compromise design that is chosen often these days > in new > languages---restricting these assignments to if/while (potentially > comp/gen) > statements. We can also reuse the existing "EXPR as NAME" syntax that > already > exists and is widely enjoyed. > > This compromise design: > > 1 Handles the most common cases (of a group of infrequent cases) > 0 Doesn't handle more obscure cases. > 1 No new syntax (through reuse) > 1 Looks Pythonic as hell > 1 Difficult to misuse, complexity capped > > Score: 4/5 > > PEP 572: > > 1 Handles the most common cases (of a group of infrequent cases) > 1 Handles even more obscure cases. > 0 New syntax > 0 Denser look: more colons, parens, expression last > 0 Some potential for misuse, complexity uncapped > > Score: 2/5 > > > Thanks for reading, happy independence, > -Mike > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Jul 5 19:42:34 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 5 Jul 2018 19:42:34 -0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? Message-ID: I wish I had more time to make my case, but with the PEP 572 pronouncement imminent, let me make an attempt to save Python from having two assignment operators. I've re-read the PEP, and honestly I am warming up to the idea of allowing a limited form of assignment in expressions. It looks like in the current form, the PEP supports only well-motivated cases where the return value of the assignment expression is non-controversial. It also appears that there are no cases where = can be substituted for := and not cause a syntax error. This means that ":" in ":=" is strictly redundant. Interestingly, Python already has a precedent for using redundant ":" - the line-ending ":" in various statements is redundant, but it is helpful both when reading and writing the code. On the other hand, ':' in ':=' looks like an unnecessary embellishment. When we use ':=', we already know that we are inside an expression and being inside an expression is an obvious context for the reader, the writer and the interpreter. I also believe, allowing a limited form of assignment in expressions is a simpler story to tell to the existing users than an introduction of a new operator that is somewhat like '=', but cannot be used where you currently use '=' and only in places where '=' is currently prohibited. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 19:46:45 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 16:46:45 -0700 Subject: [Python-Dev] PEP 572 semantics: all capabilities of the assignment statement In-Reply-To: References: <57c2b82c-39dd-5cad-77ee-4392e19f32cd@mail.mipt.ru> Message-ID: On Thu, Jul 5, 2018 at 3:45 PM Nick Coghlan wrote: > > > On Thu., 5 Jul. 2018, 3:17 pm Guido van Rossum, wrote: > >> Let me be slightly contrarian. :-) >> >> On Wed, Jul 4, 2018 at 9:12 PM Chris Angelico wrote: >> >>> Definitely against augmentation, for several reasons: >>> >>> 1) Spelling - should it be :+= or +:= ? >>> >> >> That one's easy. As Nick's (withdrawn) PEP 577 shows it should be simply >> `+=`. >> >> >>> 2) Is the result of the expression the modified value or the original? >>> >> >> Someone (sadly I forget who) showed, convincingly (to me anyways :-) that >> it should return whatever the `__iadd__` method returns, or (if there isn't >> one) the result of `a = a + b`. >> > > I think I had it as an open question in one of the earlier drafts of PEP > 577. > > The subsequent rationale for it returning the modified value was that we > have this existing equivalence at the statement level: > > a += b > a = operator.iadd(a, b) > > So the natural expression level semantics would be: > > a := operator.iadd(a, b) > Thinking about it more, I think the real stumper was what should happen for either of these: x = (a.b := 1) x = (a.b += 1) Take the first one and let's try to compile it to pseudo bytecode (which I'm making up on the spot but should be simple enough to understand if you've seen output from the "dis" module): LOAD 1 LOAD a SETATTR b ??? What do we do next? We could do LOAD a GETATTR b STORE x But this gives a's class the opportunity to change the value (because its __setattr__ could normalize the value, e.g. to a string or a float, and then its __getattribute__ would return the normalized value). But this seems a rare case and wastes time in the common case, and also seems somewhat more surprising than always assinging 1 to x regardless of what SETATTR did, so I'd rather forgo the extra GETATTR operation. So I think it should be LOAD 1 DUP LOAD a SETATTR b LOAD r1 STORE x I think the second example could be translated as follows (using a register rather than a sequence of DUPs and ROTs to save the extra copy of the result of the IADD that we want to store into x). LOAD a DUP GETATTR b LOAD 1 IADD COPY .r1 # copy into register r1 SETATTR b # this uses the reference to a that was left on the stack by DUP LOAD .r1 STORE x I used pen and paper to figure out what was on the stack at each point and had to rewrite the pseudo bytecode several times. But now I'm pretty sure this will pose no serious challenge for bytecode generation. But I'd like to point out to anyone who made it this far that this is not part of PEP 572! The PEP currently proposes neither "+=" in expressions nor targets that are attributes -- it only proposes "NAME := EXPR" because for the others the use cases are just too thin. (But in the past we said that about things like "(1, 2, *args, *more_args)" and eventually we found enough use cases for them that they were added. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Thu Jul 5 19:47:22 2018 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Thu, 5 Jul 2018 19:47:22 -0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: I think I tried a variation of your proposal here https://mail.python.org/pipermail/python-dev/2018-April/152939.html and nobody really liked it. Yury On Thu, Jul 5, 2018 at 7:44 PM Alexander Belopolsky wrote: > > I wish I had more time to make my case, but with the PEP 572 pronouncement imminent, let me make an attempt to save Python from having two assignment operators. > > I've re-read the PEP, and honestly I am warming up to the idea of allowing a limited form of assignment in expressions. It looks like in the current form, the PEP supports only well-motivated cases where the return value of the assignment expression is non-controversial. It also appears that there are no cases where = can be substituted for := and not cause a syntax error. This means that ":" in ":=" is strictly redundant. > > Interestingly, Python already has a precedent for using redundant ":" - the line-ending ":" in various statements is redundant, but it is helpful both when reading and writing the code. > > On the other hand, ':' in ':=' looks like an unnecessary embellishment. When we use ':=', we already know that we are inside an expression and being inside an expression is an obvious context for the reader, the writer and the interpreter. > > I also believe, allowing a limited form of assignment in expressions is a simpler story to tell to the existing users than an introduction of a new operator that is somewhat like '=', but cannot be used where you currently use '=' and only in places where '=' is currently prohibited. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/yselivanov.ml%40gmail.com -- Yury From alexander.belopolsky at gmail.com Thu Jul 5 19:58:00 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 5 Jul 2018 19:58:00 -0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 7:47 PM Yury Selivanov wrote: > I think I tried a variation of your proposal here > https://mail.python.org/pipermail/python-dev/2018-April/152939.html > and nobody really liked it. > > Right. I now recall your proposal. I think I did not support it at the time because I was against having expressions with side-effects regardless of syntax. Now, as I mentioned, in the current form the PEP makes a strong case for allowing a limited form of variable assignment in expressions. I also think that the dreadfulness of mistyping = where == is expected is exaggerated. In all motivating cases, := is used to introduce new bindings rather than rebinding existing names. Automated code checkers can easily warn users when they rebind variables in if statements and suggest that they silence the warnings with redundant (..) if they really want what they wrote. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 20:17:07 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 17:17:07 -0700 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> Message-ID: On Thu, Jul 5, 2018 at 3:35 PM Nick Coghlan wrote: > Guido did fully specify this in his post on "__parentlocal" scoping, in > response to my request that this be clearly spelled out in the PEP (that > specification just hasn't been rolled back into the PEP yet). > Having written it up that way, I don't think it actually would add clarity to the PEP. What would probably add clarity is some examples showing the equivalent "classic" code for the various edge cases involving comprehensions, showing where the "nonlocal" or "global" would go. > While Tim's correct that the underlying runtime semantics here aren't new > (which is why PEP 558 doesn't need to change), there's a new requirement > imposed on a Python compiler's symbol table analysis pass to see "A := > expr" in a comprehension or generator expression scope > (To clarify: as always, this scope excludes the "outermost iterable".) > and interpret that as equivalent to either: > > 1. An implied "global A" in the child scope if the parent scope is the > module scope, or A is explicitly declared as global in the parent scope > Yes. Two examples (so I can copy them easily into the PEP): # Module scope x = [a := i for i in range(10)] # Translates roughly into def comprehension(iterator): global a result = [] for i in iterator: result.append(a := i) # The meaning of this := should be clear return result x = comprehension(iter(range(10))) # Explicit global declaration def foo(): global a x = [a := i for i in range(10)] # The translation is identical to the above except nested inside foo() 2. An implied "nonlocal A" in the child scope if the parent scope is a > function scope and A is not defined as global in that scope. If A is not > already declared as local or nonlocal in the parent scope, then it is > implicitly declared as local in that scope with no associated annotation > (akin to "if 0: for A in (): pass") > I'm confused why you use both "if 0:" and "for A in ():" here -- I'd think that you can use either "if 0: A = 0" or "for A in (): pass" (the latter without the "if 0:" prefix). My personal favorite here is currently "A: object" -- since PEP 526 this already means "A is local in this function" without giving it a value. But any of the others will work too. Anyway you're right there are three subcases here: - explicit nonlocal in the parent scope - variable is already assigned to or declared in the parent scope (at least one of "A = ..." or "A: ...") - neither Only the third subcase requires adding a dummy local declaration (i.e., "if 0: A = 0" or one of the others discussed above). (Note that all three subcases add "nonlocal A" to the implied function). There's also another special case when one comprehension occurs inside another comprehension, e.g. [[A := i+j for i in range(3)] for j in range(5)] In this case the function generated for *both* comprehensions needs to contain "nonlocal A" (or, if case (1) above applies, both should "global A"). I believe this case is currently not mentioned in the PEP, though I recall mentioning it before in one of the many threads. I've made a mental note of this. The prohibition on assignment to loop control variables applies to all loop control variables that are visible from the point of the ":=" operator: in this example, the variable at position A is not allowed to reference either i or j. I would also disallow assignment to the loop control variable in the "outer iterable" of a comprehension, since I don't think there's any point in allowing that: [i for i in i := range(3)] should be disallowed. All such prohibitions should be syntax errors. > 3. A compile time error if the parent scope is a class scope (since we > don't have any existing scope declaration semantics that can be used to > make that case work sensibly) > Right. > I'm still wondering if it might make sense to define a new > "TargetScopeError" subclass of SyntaxError for that last case, since it > isn't the assignment expression syntax itself that's the problem: it's > where that expression is located. > Yeah, that would be a good idea. (Though do we currently have any subclasses of SyntaxError? And if so, do any of them have a name that does not include the substring "SyntaxError"?) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Jul 5 20:17:43 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 Jul 2018 10:17:43 +1000 Subject: [Python-Dev] PEP 572 semantics In-Reply-To: References: <6fcd9669-73a3-f2fb-7e4e-2a1e55174690@python.org> <4f80ed50-5aa0-e8a0-f961-a54cae8b592a@python.org> Message-ID: On Fri, Jul 6, 2018 at 10:17 AM, Guido van Rossum wrote: >> I'm still wondering if it might make sense to define a new >> "TargetScopeError" subclass of SyntaxError for that last case, since it >> isn't the assignment expression syntax itself that's the problem: it's where >> that expression is located. > > > Yeah, that would be a good idea. (Though do we currently have any subclasses > of SyntaxError? And if so, do any of them have a name that does not include > the substring "SyntaxError"?) > There's IndentationError. +1 on the dedicated exception type. ChrisA From guido at python.org Thu Jul 5 20:23:37 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 17:23:37 -0700 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: Sorry, I tried this too. If you think the response to the current version of the PEP is strong, the negative reaction to that version was way stronger, and I decided not to pursue it. On Thu, Jul 5, 2018 at 5:00 PM Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > > > On Thu, Jul 5, 2018 at 7:47 PM Yury Selivanov > wrote: > >> I think I tried a variation of your proposal here >> https://mail.python.org/pipermail/python-dev/2018-April/152939.html >> and nobody really liked it. >> >> Right. I now recall your proposal. I think I did not support it at the > time because I was against having expressions with side-effects regardless > of syntax. Now, as I mentioned, in the current form the PEP makes a strong > case for allowing a limited form of variable assignment in expressions. I > also think that the dreadfulness of mistyping = where == is expected is > exaggerated. In all motivating cases, := is used to introduce new bindings > rather than rebinding existing names. Automated code checkers can easily > warn users when they rebind variables in if statements and suggest that > they silence the warnings with redundant (..) if they really want what they > wrote. > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Thu Jul 5 20:28:09 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 5 Jul 2018 19:28:09 -0500 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: [Alexander Belopolsky] > ... > I also think that the dreadfulness of mistyping = where == is expected > is exaggerated. There are a number of core devs who would be rabidly opposed to allowing that confusion in Python, due to still-remembered real-life nightmares in C. For example, me ;-) It only takes one wasted day of debugging that typo in a time-critical project to sour you on it for life, and several of us run out of fingers counting the number of days it actually did cost over our C careers. > In all motivating cases, := is used to introduce new bindings rather than > rebinding existing names. I've heard people say that several times now, but suspect that's due to that they're looking at "motivating cases" in isolation. In a function, for example, doing a lot with regexps, blocks like if m := pat1.search(string): ... if m := pat2.search(substring): ... may be present any number of times. Only the first such block is _not_ a rebinding. Reusing short temp names for stuff like this may be almost as common as using `i` and `j` as for-loop target names. > Automated code checkers can easily warn users when they rebind > variables in if statements and suggest that they silence the warnings > with redundant (..) if they really want what they wrote. Alas, many people new to Python put parens around _all_ `if` and `while` conditions, due to habit carried over from other languages (common as newbies on, e.g., StackOverflow). They're the most vulnerable. Nobody in their right mind even suspects that putting parens around an entire expression could have semantic significance. a = 1 + 2 # OK, adds 1 and 2 a = (1 + 2) # but who could possibly guess what this means? ;-) But I expect the idea was DOA for the first reason above. -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Thu Jul 5 21:00:39 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 5 Jul 2018 21:00:39 -0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 8:28 PM Tim Peters wrote: > [Alexander Belopolsky] > > ... > > I also think that the dreadfulness of mistyping = where == is expected > > is exaggerated. > > There are a number of core devs who would be rabidly opposed to allowing > that confusion in Python, due to still-remembered real-life nightmares in > C. For example, me ;-) It only takes one wasted day of debugging that > typo in a time-critical project to sour you on it for life, and several of > us run out of fingers counting the number of days it actually did cost over > our C careers. > I still do quite a bit of C programming and I have exactly the opposite experience given modern environments: why is gcc/clang/vs complaining about if (x=a) - I know what I am doing! No, I don't want to put two pairs of (..) around condition - one is one too many! > Alas, many people new to Python put parens around _all_ `if` and `while` > conditions, due to habit carried over from other languages (common as > newbies on, e.g., StackOverflow). They're the most vulnerable. Nobody in > their right mind even suspects that putting parens around an entire > expression could have semantic significance. > > a = 1 + 2 # OK, adds 1 and 2 > a = (1 + 2) # but who could possibly guess what this means? ;-) > > But I expect the idea was DOA for the first reason above. > Not that unlike Yuri, I don't think the language should require (..) around assignment expressions. Seriously, ':=' looks like a reluctantly introduced kludge to allow assignment in expressions. We agree that it is sometimes useful to have, but we will make the feature really hard to use or discover. What happened to the "consenting adults" philosophy? Do we want to protect users who cannot tell = from == so much that we are willing to cause Python to be the first language with two non-interchangeable assignment operators? -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeanpierreda at gmail.com Thu Jul 5 21:07:19 2018 From: jeanpierreda at gmail.com (Devin Jeanpierre) Date: Thu, 5 Jul 2018 18:07:19 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180705024105.GC14437@ando.pearwood.info> References: <20180704133125.GR14437@ando.pearwood.info> <20180704180215.GW14437@ando.pearwood.info> <20180705024105.GC14437@ando.pearwood.info> Message-ID: On Wed, Jul 4, 2018 at 7:42 PM Steven D'Aprano wrote: > On Wed, Jul 04, 2018 at 01:00:41PM -0700, Devin Jeanpierre wrote: > > On Wed, Jul 4, 2018 at 11:04 AM Steven D'Aprano wrote: > > > Did you actually mean arbitrary simple statements? > > > > > > if import math; mylist.sort(); print("WTF am I reading?"); True: > > > pass > > > > Yes. > > Okay, you just broke my brain. > > I was *sure* that you were going to complain that I was misrepresenting > your position, or attacking a strawman, or something. This "brain-breaking" experience is something to keep in mind when we read the reactions to the acceptance assignment expressions, so that we can better empathize with the people who are so shocked by it. After all, assignment expressions allow nearly word for word the same abuse you posted: if [(math := __import__('math')), mylist.sort(), print('WTF?'), True][-1]: ... Anything that allows assignment in an if statement is going to allow some messed up stuff. The PEP seems to intentionally take the stance that it is OK to allow super-ugly code, because we can trust people to just not do that. > I did not imagine for a second that you *actually* would prefer > to allow the above code over assignment expressions. I don't really know how to bridge that disconnect. I thought Nathaniel argued very well what is lost with assignment expressions. -- Devin From tim.peters at gmail.com Thu Jul 5 22:10:24 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 5 Jul 2018 21:10:24 -0500 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: [Alexander Belopolsky] >>> ... >>> I also think that the dreadfulness of mistyping = where == is expected >>> is exaggerated. [Tim] >> There are a number of core devs who would be rabidly opposed >> to allowing that confusion in Python, due to still-remembered >> real-life nightmares in C. For example, me ;-) It only takes one >> wasted day of debugging that typo in a time-critical project to sour >> you on it for life, and several of us run out of fingers counting the >> number of days it actually did cost over our C careers. [Alexander] > I still do quite a bit of C programming and I have exactly the opposite > experience Meaning that confusing "=" and "==" in C _saves_ you days of debugging in time-critical projects? ;-) > given modern environments: why is gcc/clang/vs complaining about if (x=a) - > I know what I am doing! Because gcc/clang/vs is acknowledging how widely and deeply this C wart is despised. Isn't that obvious? You're quite the exception here, not "the rule" - as gcc/clang/vs eternally but apparently futilely remind you ;-) ... > Seriously, ':=' looks like a reluctantly introduced kludge to allow > assignment in expressions. There have been piles of other suggestions, but ":=" remains the least disliked (at least by Guido, and his opinion actually counts ;-) ). > We agree that it is sometimes useful to have, but we will make the feature > really hard to use ? It's very easy to use. Try it - I have. But I've used several languages in which ":=" was _the_ way to spell assignment, so it felt familiar at first touch. > or discover. It doesn't even exist yet, but Googling on python operator := already returns a directly relevant hit on the first page for me: https://stackoverflow.com/questions/26000198/what-does-colon-equal-in-python-mean The hits above it are all to overviews of Python operators. Here on Windows, the interface to the Python doc files in IDLE contains an entry for each operator, so just typing := in the index search box will eventually go directly to its docs. If you can't do something similar on Linux, upgrade to Windows ;-) > What happened to the "consenting adults" philosophy? Struggling mightily the last several months to get consensus on some form of embedded assignment _at all_ :-( > Do we want to protect users who cannot tell = from == Yes. > so much that we are willing to cause Python to be the first language > with two non-interchangeable assignment operators? Sure - I am. _Far_ more than I'm willing to reproduce C's mistake. It's one of the first things singled out in Andrew Koenig's "C Traps and Pitfalls", starting with his original paper that was the basis for his 1989 book of the same name: http://literateprogramming.com/ctraps.pdf You can use Google too - it's a top-10 item in every list of "C warts" I've ever seen. It's impossible to overstate how hated it is. But, sure - if you post in its defense a few more times, everyone sensible is sure to change their mind - and I'll ask Andrew to remove that section from his still-selling book ;-) I solved the problem in my own code by using an editor that displays a single "=" in C source as a left-arrow graphic (that's one of its C-specific display options - again a response to how notorious this bug-magnet is). So assignment and equality-testing in C code look entirely different to me, regardless of context. But to this day, I routinely get a SyntaxError when writing new Python code because I _think_ "if x equals y" and _type_ "if x = y:". So I know for sure that it's still a typo I'm way too prone to make. I've picked up that you're not, but that's scant comfort ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Thu Jul 5 22:35:18 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 6 Jul 2018 12:35:18 +1000 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: <20180706023517.GE7318@ando.pearwood.info> On Thu, Jul 05, 2018 at 09:00:39PM -0400, Alexander Belopolsky wrote: > Do we want to protect users who > cannot tell = from == so much that we are willing to cause Python to be > the first language with two non-interchangeable assignment operators? Not even close to the first. Go beat us to it -- it has both = and := assignment operators. Ocaml also has := for regular assignment and <- for assignment to mutable fields. Similarly, Haskall has = for assignment definitions and <- for binding in monads. -- Steve From alexander.belopolsky at gmail.com Thu Jul 5 22:48:11 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Thu, 5 Jul 2018 22:48:11 -0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 10:10 PM Tim Peters wrote: > .. > I solved the problem in my own code by using an editor that displays a > single "=" in C source as a left-arrow graphic (that's one of its > C-specific display options - again a response to how notorious this > bug-magnet is). So assignment and equality-testing in C code look > entirely different to me, regardless of context. > > But to this day, I routinely get a SyntaxError when writing new Python > code because I _think_ "if x equals y" and _type_ "if x = y:". So I know > for sure that it's still a typo I'm way too prone to make. > Python really has a strong C legacy and this is the area where I agree that C designers made a mistake by picking a symmetric symbol (=) for an asymmetric operation. On top of that, they picked an asymmetric digraph (!=) for a symmetric operation as well and Python (unfortunately) followed the crowd and ditched a much better alternative (<>). My only hope is that Python 4.0 will allow ? to be used in place of either = or :=. :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Thu Jul 5 22:57:53 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 5 Jul 2018 21:57:53 -0500 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: <20180706023517.GE7318@ando.pearwood.info> References: <20180706023517.GE7318@ando.pearwood.info> Message-ID: [Alexander Belopolsky] > > Do we want to protect users who > > cannot tell = from == so much that we are willing to cause Python to be > > the first language with two non-interchangeable assignment operators? > [Steven D'Aprano][ Not even close to the first. Go beat us to it -- it has both = and := > assignment operators. > > Ocaml also has := for regular assignment and <- for assignment to > mutable fields. > > Similarly, Haskall has = for assignment definitions and <- for binding > in monads. > I skipped that part, because nobody actually cares ;-) But back when I looked at this, R had them all beat, and I never found a single source that actually managed to list _all_ of R's variations This source was clearest, but is missing (at least) all the ways to spell all the variations as function calls too: https://stat.ethz.ch/R-manual/R-devel/library/base/html/assignOps.html There are three different assignment operators: two of them have leftwards > and rightwards forms. > The operators <- and = assign into the environment in which they are > evaluated. The operator <- can be used anywhere, whereas the operator = is > only allowed at the top level (e.g., in the complete expression typed at > the command prompt) or as one of the subexpressions in a braced list of > expressions. > The operators <<- and ->> are normally only used in functions, and cause a > ... > .[click the link if you're still awake] > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 23:11:56 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 20:11:56 -0700 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: <5B3E30D1.3080706@UGent.be> References: <5B3E30D1.3080706@UGent.be> Message-ID: I'm not the world's leading expert on Python bytecode anymore, but unless there's something I'm missing your conclusion looks eminently reasonable, and so I expect you'll get very little traction on this thread. (If you had wanted to get a megathread you should have written "FASTCALL considered harmful". :-) I think there was one person in another thread (INADA Naoki?) who thought METH_FASTCALL could use improvements. Maybe that person can write back to this thread? Or perhaps Victor Stinner (who seems to have touched it last) has a suggestion for what could be improved about it? --Guido On Thu, Jul 5, 2018 at 7:55 AM Jeroen Demeyer wrote: > Hello all, > > As discussed in some other threads ([1], [2]), we should discuss the > METH_FASTCALL calling convention. > > For passing only positional arguments, a C array of Python objects is > used, which is as fast as it can get. When the Python interpreter calls > a function, it builds that C array on the interpreter stack: > > >>> from dis import dis > >>> def f(x, y): return g(x, y, 12) > >>> dis(f) > 1 0 LOAD_GLOBAL 0 (g) > 2 LOAD_FAST 0 (x) > 4 LOAD_FAST 1 (y) > 6 LOAD_CONST 1 (12) > 8 CALL_FUNCTION 3 > 10 RETURN_VALUE > > A C array can also easily and efficiently be handled by the C function > receiving it. So I consider this uncontroversial. > > The convention for METH_FASTCALL|METH_KEYWORDS is that keyword *names* > are passed as a tuple and keyword *values* in the same C array with > positional arguments. An example: > > >>> from dis import dis > >>> def f(x, y, z): return f(x, foo=y, bar=z) > >>> dis(f) > 1 0 LOAD_GLOBAL 0 (f) > 2 LOAD_FAST 0 (x) > 4 LOAD_FAST 1 (y) > 6 LOAD_FAST 2 (z) > 8 LOAD_CONST 1 (('foo', 'bar')) > 10 CALL_FUNCTION_KW 3 > 12 RETURN_VALUE > > This is pretty clever: it exploits the fact that ('foo', 'bar') is a > constant tuple stored in f.__code__.co_consts. Also, a tuple can be > efficiently handled by the called code: it is essentially a thin wrapper > around a C array of Python objects. So this works well. > > The only case when this handling of keywords is suboptimal is when using > **kwargs. In that case, a dict must be converted to a tuple. It looks > hard to me to support efficiently both the case of fixed keyword > arguments (f(foo=x)) and a keyword dict (f(**kwargs)). Since the former > is more common than the latter, the current choice is optimal. > > In other words: I see nothing to improve in the calling convention of > METH_FASTCALL. I suggest to keep it and make it public as-is. > > > Jeroen. > > > [1] https://mail.python.org/pipermail/python-dev/2018-June/153945.html > [2] https://mail.python.org/pipermail/python-dev/2018-July/154251.html > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 23:20:20 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 20:20:20 -0700 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 7:12 PM Tim Peters wrote: It doesn't even exist yet, but Googling on python operator := already returns a directly relevant hit on the first page for me: https://stackoverflow.com/questions/26000198/what-does-colon-equal-in-python-mean The hits above it are all to overviews of Python operators. Here on Windows, the interface to the Python doc files in IDLE contains an entry for each operator, so just typing := in the index search box will eventually go directly to its docs. If you can't do something similar on Linux, upgrade to Windows ;-) Since this thread is dead and everybody is just posting to get the last word, I can report that for me, Googling for "Python :=" gives PEP 572 as the first non-ad hit. Also among the top hits are section 6 (Expressions) and section 7 (Simple statements) of the Python 3.7.0 docs. Seems there is at least one thing that Google still does right. :-) (Though Bing also has an ironically relevant hit: apart from the above StackOverflow issue, it found a piece of third party documentation titled Conditionals and loops. :-) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 5 23:22:12 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 5 Jul 2018 20:22:12 -0700 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: Sorry for messing up the quoting. I blame GMail. Only the text starting "Since this thread is dead" was mine. On Thu, Jul 5, 2018 at 8:20 PM Guido van Rossum wrote: > On Thu, Jul 5, 2018 at 7:12 PM Tim Peters wrote: > It doesn't even exist yet, but Googling on > > python operator := > > already returns a directly relevant hit on the first page for me: > > > https://stackoverflow.com/questions/26000198/what-does-colon-equal-in-python-mean > > The hits above it are all to overviews of Python operators. Here on > Windows, the interface to the Python doc files in IDLE contains an entry > for each operator, so just typing := in the index search box will > eventually go directly to its docs. If you can't do something similar on > Linux, upgrade to Windows ;-) > > Since this thread is dead and everybody is just posting to get the last > word, I can report that for me, Googling for "Python :=" gives PEP 572 as > the first non-ad hit. Also among the top hits are section 6 (Expressions) > and section 7 (Simple statements) of the Python 3.7.0 docs. Seems there is > at least one thing that Google still does right. :-) > > (Though Bing also has an ironically relevant hit: apart from the above > StackOverflow issue, it found a piece of third party documentation titled > Conditionals and loops. :-) > > -- > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Thu Jul 5 23:33:32 2018 From: tim.peters at gmail.com (Tim Peters) Date: Thu, 5 Jul 2018 22:33:32 -0500 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: [Tim] > It doesn't even exist yet, but Googling on > python operator := > > already returns a directly relevant hit on the first page for me: > > > https://stackoverflow.com/questions/26000198/what-does-colon-equal-in-python-mean > ... > [Guido, who later apologized for unclear quoting, hilariously trying to pin the blame on GMail ;-)] Since they moved to the "new" GMail, quoting selected pieces in replies has become a nightmare for me too. I don't understand the options, and it seems there's no option at all to do "just plain text, please - nothing fancy" anymore :-( Since this thread is dead and everybody is just posting to get the last > word, > You're such a cynic. We're competing to get the _first_ word on something destined to convince you that PEP 572 can be vastly improved via a one-character change ;-) > I can report that for me, Googling for "Python :=" gives PEP 572 as the > first non-ad hit. Also among the top hits are section 6 (Expressions) and > section 7 (Simple statements) of the Python 3.7.0 docs. > And I get essentially the same Google results too when I leave out "operator" from my original To verify the obvious guess, Googling on just plain := does indeed return hits supremely relevant to the colon-equal "symbol" regardless of context. Seems there is at least one thing that Google still does right. :-) I'm sure it will be improved out of existence soon enough ;-) > ... > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Thu Jul 5 23:45:07 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 5 Jul 2018 23:45:07 -0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018, 7:46 PM Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > It also appears that there are no cases where = can be substituted for := > and not cause a syntax error. This means that ":" in ":=" is strictly > redundant. > Under your proposal, this is ambiguous: 'myfun(foo=5, bar=7)' -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Jul 6 00:02:59 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 6 Jul 2018 14:02:59 +1000 Subject: [Python-Dev] Symmetric vs asymmetric symbols (was PEP 572: Do we really need a ":" in ":="?) Message-ID: On Fri, Jul 6, 2018 at 12:48 PM, Alexander Belopolsky wrote: > > Python really has a strong C legacy and this is the area where I agree that > C designers made a mistake by picking a symmetric symbol (=) for an > asymmetric operation. On top of that, they picked an asymmetric digraph (!=) > for a symmetric operation as well and Python (unfortunately) followed the > crowd and ditched a much better alternative (<>). My only hope is that > Python 4.0 will allow ? to be used in place of either = or :=. :-) Interesting. Looking over Python's binary operators, we have: |, ^, &, +, *: symmetric (on ints) -, /, //, **: asymmetric <, >: mirrored operations <=, >=: mirrored operations but not reflected <<, >>: non-mirrored asymmetric and, or: technically asymmetric but often treated as symmetric in, not in: asymmetric is, is not: symmetric Which ones ought to have symmetric symbols, in an ideal world? Should <= and >= be proper mirrors of each other? Are << and >> confusing? Is it a problem that the ** operator is most decidedly asymmetric? Personally, I'm very happy that the operators use the same symbols that they do in other languages - U+002B PLUS SIGN means addition, for instance - and everything else is secondary. But maybe this is one of those "hidden elegances" that you're generally not *consciously* aware of, but which makes things "feel right", like how Disney's "Moana" has freedom to the right of the screen and duty to the left. Are there languages where symmetric operations are always represented with symmetric symbols and vice versa? ChrisA From songofacandy at gmail.com Fri Jul 6 00:07:46 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 6 Jul 2018 13:07:46 +0900 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: <5B3E30D1.3080706@UGent.be> References: <5B3E30D1.3080706@UGent.be> Message-ID: I don't know Serhiy's idea about how METH_FASTCALL | METH_KEYWORDS calling convention can be improved in the future. When I reading PEP 580, I liked your Py PyCCall_FastCall signature. Maybe, one way to improve METH_FASTCALL | METH_KEYWORDS can be this. kwds can be either tuple or dict. --- Anyway, if we don't make METH_FASTCALL | METH_KEYWORDS public for now, can we continue both PEPs without exposing keyword arguments support? For example, PEP 576 defines new signature: typedef PyObject *(*extended_call_ptr)(PyObject *callable, PyObject** args, int positional_argcount, PyTupleObject* kwnames); `PyTupleObject *kwnames` can be `PyObject *reserved` and "should be NULL always" in document? PEP 580 is more simple. Keeping CCALL_KEYWORDS private. I think Cython is the most important user of these PEPs. And Cython creates functions supporting keywords easily, like Python. So this can be worthless... -- INADA Naoki From fred at fdrake.net Fri Jul 6 01:00:09 2018 From: fred at fdrake.net (Fred Drake) Date: Fri, 6 Jul 2018 01:00:09 -0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Thu, Jul 5, 2018 at 9:02 PM Alexander Belopolsky wrote: > What happened to the "consenting adults" philosophy? Just anecdotally, I've run into a number of professionals recently who've come out of Java environments who really dislike the "consenting adults" approach. While they value much that Python offers them, they still really want some of the hand-holding languages like Java give them. What exactly they identify as "compelling" varies based on their past experiences, but it's not completely without reason. The reasoning is often tied to providing APIs "less-experienced" programmers can use without breaking things. Some of these programmers at least claim to understand that bigger risks come from not dealing with API boundaries that check input values carefully, or that don't deal cleanly with exception propagation; that seems to be fairly difficult for many. The idiomatic models I've found most desirable in Python are not as widely valued as I'd hope. It's hard to say what that tells us, and I think we'd want more quantifiable evidence before drawing conclusions. The prevalence of expectation that an API constrains the mechanics of how it is used is something I find surprising. While I can suspect that it's misguided, I keep running into situations where code needs to be maintained by people who just aren't as familiar with the idioms I'm accustomed to, so... considering the issue has value. -Fred -- Fred L. Drake, Jr. "A storm broke loose in my mind." --Albert Einstein From vano at mail.mipt.ru Fri Jul 6 04:34:18 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Fri, 6 Jul 2018 11:34:18 +0300 Subject: [Python-Dev] Symmetric vs asymmetric symbols (was PEP 572: Do we really need a ":" in ":="?) In-Reply-To: References: Message-ID: On 06.07.2018 7:02, Chris Angelico wrote: > On Fri, Jul 6, 2018 at 12:48 PM, Alexander Belopolsky > wrote: >> Python really has a strong C legacy and this is the area where I agree that >> C designers made a mistake by picking a symmetric symbol (=) for an >> asymmetric operation. On top of that, they picked an asymmetric digraph (!=) >> for a symmetric operation as well and Python (unfortunately) followed the >> crowd and ditched a much better alternative (<>). My only hope is that >> Python 4.0 will allow ? to be used in place of either = or :=. :-) > Interesting. Looking over Python's binary operators, we have: > > |, ^, &, +, *: symmetric (on ints) > -, /, //, **: asymmetric > <, >: mirrored operations > <=, >=: mirrored operations but not reflected > <<, >>: non-mirrored asymmetric > and, or: technically asymmetric but often treated as symmetric > in, not in: asymmetric > is, is not: symmetric > > Which ones ought to have symmetric symbols, in an ideal world? Should > <= and >= be proper mirrors of each other? Are << and >> confusing? Is > it a problem that the ** operator is most decidedly asymmetric? > > Personally, I'm very happy that the operators use the same symbols > that they do in other languages - U+002B PLUS SIGN means addition, for > instance - and everything else is secondary. But maybe this is one of > those "hidden elegances" that you're generally not *consciously* aware > of, but which makes things "feel right", like how Disney's "Moana" has > freedom to the right of the screen and duty to the left. Are there > languages where symmetric operations are always represented with > symmetric symbols and vice versa? Nothing like that. The connotations for the symbols rather draw from other fields that we're familiar with. Primarily math (that everyone has studied at school -- this is also the reason why we use infix notation even though the postfix one allows to ditch braces); in particular, % draws from ? (division sign), / from fraction sign; & is from English; !, | and ^ are a mash from "the closest unused symbols on keyboard" to symbols from logic algebra: ?? ?? (https://en.wikipedia.org/wiki/Negation),? {\displaystyle \lor } ? ? (https://en.wikipedia.org/wiki/Logical_disjunction), ? ? (https://en.wikipedia.org/wiki/Logical_conjunction), ?? or | (https://en.wikipedia.org/wiki/Sheffer_stroke), ?? (https://en.wikipedia.org/wiki/Peirce's_arrow); "!=" reads literally "not equal" ( ! is "not", = is "equal" ) ?? (while "<>" reads "less or greater" which is mathematically not equivalent to that: not everything has a defined ordering relation. "<>" draws from BASIC AFAIK which was geared towards regular users who don't deal with advanced mathematics.) > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Fri Jul 6 05:50:06 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Fri, 6 Jul 2018 11:50:06 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <1cb5933b3ce7418d90aa8b9488efc8e7@xmail103.UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3E341E.70402@UGent.be> <1cb5933b3ce7418d90aa8b9488efc8e7@xmail103.UGent.be> Message-ID: <5B3F3B4E.6080508@UGent.be> On 2018-07-05 21:57, Guido van Rossum wrote: > Would it be possible to get outside experts to help? I don't understand what you mean: to help with what? Designing the PEP? Discussing the PEP? Accepting the PEP? Lobbying Python core devs? The Cython developers (in particular Stefan Behnel) certainly support my work. I have talked with them in person at a workshop and they posted a few emails to python-dev and they also gave me some personal comments about PEP 580. As for NumPy, one obvious place where some ideas of PEP 579 could be applied is ufuncs. However, typically ufuncs are not *called* in tight loops, they implement tight loops internally. So I don't think that there is pressing need for anything like PEP 580 in NumPy. Jeroen. From J.Demeyer at UGent.be Fri Jul 6 06:23:20 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Fri, 6 Jul 2018 12:23:20 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: <4722134ee2df4fe1893d405ce2903dce@xmail103.UGent.be> References: <5B3E30D1.3080706@UGent.be> <4722134ee2df4fe1893d405ce2903dce@xmail103.UGent.be> Message-ID: <5B3F4318.3020404@UGent.be> On 2018-07-06 06:07, INADA Naoki wrote: > Maybe, one way to improve METH_FASTCALL | METH_KEYWORDS can be this. > kwds can be either tuple or dict. But that would be just pushing the complexity down to the callee. I'd rather have a simpler protocol at the expense of a slightly more complex supporting API. I also don't see the point: the calls where performance truly matters typically don't use keyword arguments anyway (independently of whether the called function accepts them). Moreover, the large majority of functions take normal keyword arguments, not **kwargs. When parsing those arguments, the dict would need to be unpacked anyway. So you don't gain much by forcing the callee to handle that instead of doing it in PyCCall_FASTCALL(). Functions just passing through **kwargs (say, functools.lru_cache) don't need a dict either: they can implement the C call protocol of PEP 580 with METH_FASTCALL and then call the wrapped function also using FASTCALL. So really the only remaining case is when the callee wants to do something with **kwargs as dict. But I find it hard to come up with a natural use case for that, especially one where performance matters. And even then, that function could just use METH_VARARGS. So I don't see any compelling reason to allow a dict in METH_FASTCALL. Jeroen. From J.Demeyer at UGent.be Fri Jul 6 06:50:14 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Fri, 6 Jul 2018 12:50:14 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> Message-ID: <5B3F4966.4020404@UGent.be> On 2018-07-05 14:20, INADA Naoki wrote: > like you ignored my advice about creating realistic benchmark for > calling 3rd party callable before talking about performance... I didn't really want to ignore that, I just didn't know what to do. As far as I can tell, the official Python benchmark suite is https://github.com/python/performance However, that deals only with pure Python code, not with the C API. So those benchmarks are not relevant to PEP 580. Jeroen. From songofacandy at gmail.com Fri Jul 6 07:33:31 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 6 Jul 2018 20:33:31 +0900 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3F4966.4020404@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3F4966.4020404@UGent.be> Message-ID: On Fri, Jul 6, 2018 at 7:50 PM Jeroen Demeyer wrote: > > On 2018-07-05 14:20, INADA Naoki wrote: > > like you ignored my advice about creating realistic benchmark for > > calling 3rd party callable before talking about performance... > > I didn't really want to ignore that, I just didn't know what to do. > > As far as I can tell, the official Python benchmark suite is > https://github.com/python/performance > However, that deals only with pure Python code, not with the C API. > So those benchmarks are not relevant to PEP 580. > These benchmarks uses 3rd party extension modules. For example, mako benchmark uses Mako, and Mako uses MarkupSafe. I optimized MarkupSafe based on the benchmark. https://github.com/pallets/markupsafe/pull/64 If bm_mako or some other existing benchmarks is OK to demonstrate METH_FASTCALL benefits, you can just customize 3rd party library and compare performance. If it's not enough, you should write new benchmark to demonstrate it. One important point is the benchmark should demonstrate "application" performance. Comparing just overhead of METH_VARARGS vs METH_FASTCALL is useless, because we did it already. What you should demonstrate is: METH_FASTCALL (or METH_FASTCALL | METH_KEYWORDS) really boost real world application performance. From python-dev at mgmiller.net Fri Jul 6 11:03:42 2018 From: python-dev at mgmiller.net (Mike Miller) Date: Fri, 6 Jul 2018 08:03:42 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: References: Message-ID: <37f59c3e-fb71-9ec4-5199-c62df19ec265@mgmiller.net> On 2018-07-04 00:25, Nathaniel Smith wrote: > The only cases that seem potentially valuable to me are the ones that > are literally the form 'if := ` and 'while := > '. (I suspect these are the only cases that I would allow in > code that I maintain.) The PEP does briefly discuss the alternative > proposal of restricting to just these two cases, but rejects it > because it would rule out code like 'if ( := ) > '. But those are exactly the cases that I want to > rule out, so that seems like a plus to me :-). > > The 'if as ' syntax would be a simple way to encode > exactly these simple non-harmful cases. The PEP rejects it on the > grounds that 'as' is already used in a different way by 'except' and > 'with'. But... 'as' is *also* used in the *same* way by 'import', so > the argument feels disingenuous. Yeah, there'd be an inconsistency, > but that inconsistency already exists, and adding 'if ... as' and > 'while ... as' wouldn't create any *new* inconsistencies. Agreed, tried to make this point in several threads. -Mike From chris.barker at noaa.gov Fri Jul 6 11:51:16 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 6 Jul 2018 08:51:16 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: <5B3C6F3F.8050400@canterbury.ac.nz> References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: Are we just having fun here? Or might we actually start using a new naming convention for the-syntax-formerly-known-as-generator-expressions? -CHB Sent from my iPhone > On Jul 3, 2018, at 11:54 PM, Greg Ewing wrote: > > Steven D'Aprano wrote: >> - list builder syntax is syntax which returns a list; >> - dict builder syntax is syntax which returns a dict; >> - set builder syntax is syntax which returns a set; >> - generator builder syntax is syntax which returns a generator. > > You only get a list/dict/set from the first three after > you've run the iterators within it, but with a generator > expression, you already have a generator before you've > run it. That makes it feel different to me. > > -- > Greg > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov From status at bugs.python.org Fri Jul 6 12:09:54 2018 From: status at bugs.python.org (Python tracker) Date: Fri, 6 Jul 2018 18:09:54 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20180706160954.915645640E@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2018-06-29 - 2018-07-06) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6721 (+16) closed 39090 (+44) total 45811 (+60) Open issues with patches: 2675 Issues opened (43) ================== #33735: test_multiprocessing_spawn leaked [1, 2, 1] memory blocks on A https://bugs.python.org/issue33735 reopened by vstinner #34001: LibreSSL does not tolerate setting minimum_version greater tha https://bugs.python.org/issue34001 opened by Alan.Huang #34002: minor efficiency and clarity improvements in email package https://bugs.python.org/issue34002 opened by selik #34003: csv.DictReader can return basic dict instead of OrderedDict https://bugs.python.org/issue34003 opened by selik #34004: Acquiring locks not interrupted by signals on musl libc https://bugs.python.org/issue34004 opened by Joseph Sible #34007: test_gdb fails in s390x SLES buildbots https://bugs.python.org/issue34007 opened by pablogsal #34008: Do we support calling Py_Main() after Py_Initialize()? https://bugs.python.org/issue34008 opened by ncoghlan #34009: Document Travis CI / Ubuntu 14.04 OpenSSL compatibility issues https://bugs.python.org/issue34009 opened by ncoghlan #34011: Default preference not given to venv DLL's https://bugs.python.org/issue34011 opened by jonathan-lp #34012: No option to include system headers in distutils.core.Extensio https://bugs.python.org/issue34012 opened by danh #34013: Inconsistent SyntaxError for print https://bugs.python.org/issue34013 opened by corona10 #34014: loop.run_in_executor should propagate current contextvars https://bugs.python.org/issue34014 opened by hellysmile #34017: Tkinter CheckButton not working in EXE https://bugs.python.org/issue34017 opened by elfantroussi #34019: webbrowser: wrong arguments for Opera browser. https://bugs.python.org/issue34019 opened by kbumsik #34021: [2.7] test_regrtest: test_env_changed() fails on x86 Windows X https://bugs.python.org/issue34021 opened by vstinner #34022: 6 tests fail using SOURCE_DATE_EPOCH env var https://bugs.python.org/issue34022 opened by vstinner #34023: timedelta(seconds=x) strange results when type(x) == np.int32 https://bugs.python.org/issue34023 opened by mark.dickinson #34025: SMTP EmailPolicy not using the correct line length for RCF 204 https://bugs.python.org/issue34025 opened by Douglas Thor #34026: Request for 2 Windows installation changes. https://bugs.python.org/issue34026 opened by terry.reedy #34027: python 3.7 openpty/forkpty build failure using nix package man https://bugs.python.org/issue34027 opened by LnL7 #34028: Python 3.7.0 wont compile with SSL Support 1.1.0 > alledged mi https://bugs.python.org/issue34028 opened by simon at simonfoley.net #34029: tkinter.filedialog.askdirectory() crashing before dialog opens https://bugs.python.org/issue34029 opened by rsteel1 #34031: [EASY] Incorrect usage of unittest.TestCase in test_urllib2_lo https://bugs.python.org/issue34031 opened by pablogsal #34032: Add platlibdir to allow distinction between /usr/lib and /usr/ https://bugs.python.org/issue34032 opened by mcepl #34033: distutils is not reproducible https://bugs.python.org/issue34033 opened by vstinner #34034: Python 3.7.0 multiprocessing forkserver ForkingPickler behavio https://bugs.python.org/issue34034 opened by Santiago Hernandez #34035: zipfile: AttributeError in "seek" method of "_SharedFile" clas https://bugs.python.org/issue34035 opened by espdev #34036: ModuleNotFoundError: No module named '_ctypes' when install Py https://bugs.python.org/issue34036 opened by semkin #34037: test_asyncio: test_run_in_executor_cancel() leaked a dangling https://bugs.python.org/issue34037 opened by vstinner #34038: urllib2.urlopen fails if http_proxy(s) is set to a sock5 proxy https://bugs.python.org/issue34038 opened by T L2 #34041: add *deterministic* parameter to sqlite3.Connection.create_fun https://bugs.python.org/issue34041 opened by sir-sigurd #34042: Reference loss for local classes https://bugs.python.org/issue34042 opened by kayhayen #34046: subparsers -> add_parser doesn't support hyphen char '-' https://bugs.python.org/issue34046 opened by nir_barel at bmc.com #34047: IDLE: on macOS, scroll slider 'sticks' at bottom of file https://bugs.python.org/issue34047 opened by belgort #34050: Broken links to "OpenSSL cipher list format" in documentation https://bugs.python.org/issue34050 opened by rolweber #34051: Update multiprocessing example https://bugs.python.org/issue34051 opened by Windson Yang #34052: sqlite's create_function() raises exception on unhashable call https://bugs.python.org/issue34052 opened by sir-sigurd #34055: IDLE: erroneous 'smart' indents in shell https://bugs.python.org/issue34055 opened by grantjenks #34056: checked hash-based pyc files not working with imp module https://bugs.python.org/issue34056 opened by phmccarty #34057: Py_Initialize aborts when using static Python version. Windows https://bugs.python.org/issue34057 opened by illera88 #34058: Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTH https://bugs.python.org/issue34058 opened by tkandell #34059: multiprocessing deadlock https://bugs.python.org/issue34059 opened by gobbedy #34060: regrtest: log "CPU usage" on Windows https://bugs.python.org/issue34060 opened by vstinner Most recent 15 issues with no replies (15) ========================================== #34060: regrtest: log "CPU usage" on Windows https://bugs.python.org/issue34060 #34058: Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTH https://bugs.python.org/issue34058 #34052: sqlite's create_function() raises exception on unhashable call https://bugs.python.org/issue34052 #34051: Update multiprocessing example https://bugs.python.org/issue34051 #34046: subparsers -> add_parser doesn't support hyphen char '-' https://bugs.python.org/issue34046 #34041: add *deterministic* parameter to sqlite3.Connection.create_fun https://bugs.python.org/issue34041 #34038: urllib2.urlopen fails if http_proxy(s) is set to a sock5 proxy https://bugs.python.org/issue34038 #34037: test_asyncio: test_run_in_executor_cancel() leaked a dangling https://bugs.python.org/issue34037 #34036: ModuleNotFoundError: No module named '_ctypes' when install Py https://bugs.python.org/issue34036 #34023: timedelta(seconds=x) strange results when type(x) == np.int32 https://bugs.python.org/issue34023 #34021: [2.7] test_regrtest: test_env_changed() fails on x86 Windows X https://bugs.python.org/issue34021 #34017: Tkinter CheckButton not working in EXE https://bugs.python.org/issue34017 #34012: No option to include system headers in distutils.core.Extensio https://bugs.python.org/issue34012 #34004: Acquiring locks not interrupted by signals on musl libc https://bugs.python.org/issue34004 #34002: minor efficiency and clarity improvements in email package https://bugs.python.org/issue34002 Most recent 15 issues waiting for review (15) ============================================= #34056: checked hash-based pyc files not working with imp module https://bugs.python.org/issue34056 #34052: sqlite's create_function() raises exception on unhashable call https://bugs.python.org/issue34052 #34042: Reference loss for local classes https://bugs.python.org/issue34042 #34041: add *deterministic* parameter to sqlite3.Connection.create_fun https://bugs.python.org/issue34041 #34033: distutils is not reproducible https://bugs.python.org/issue34033 #34032: Add platlibdir to allow distinction between /usr/lib and /usr/ https://bugs.python.org/issue34032 #34031: [EASY] Incorrect usage of unittest.TestCase in test_urllib2_lo https://bugs.python.org/issue34031 #34027: python 3.7 openpty/forkpty build failure using nix package man https://bugs.python.org/issue34027 #34019: webbrowser: wrong arguments for Opera browser. https://bugs.python.org/issue34019 #34014: loop.run_in_executor should propagate current contextvars https://bugs.python.org/issue34014 #34009: Document Travis CI / Ubuntu 14.04 OpenSSL compatibility issues https://bugs.python.org/issue34009 #34008: Do we support calling Py_Main() after Py_Initialize()? https://bugs.python.org/issue34008 #34003: csv.DictReader can return basic dict instead of OrderedDict https://bugs.python.org/issue34003 #34002: minor efficiency and clarity improvements in email package https://bugs.python.org/issue34002 #34001: LibreSSL does not tolerate setting minimum_version greater tha https://bugs.python.org/issue34001 Top 10 most discussed issues (10) ================================= #34022: 6 tests fail using SOURCE_DATE_EPOCH env var https://bugs.python.org/issue34022 17 msgs #33735: test_multiprocessing_spawn leaked [1, 2, 1] memory blocks on A https://bugs.python.org/issue33735 12 msgs #34027: python 3.7 openpty/forkpty build failure using nix package man https://bugs.python.org/issue34027 12 msgs #34042: Reference loss for local classes https://bugs.python.org/issue34042 12 msgs #33955: Implement PyOS_CheckStack on macOS using pthread_get_stack*_np https://bugs.python.org/issue33955 9 msgs #34019: webbrowser: wrong arguments for Opera browser. https://bugs.python.org/issue34019 9 msgs #33720: test_marshal: crash in Python 3.7b5 on Windows 10 https://bugs.python.org/issue33720 8 msgs #33944: Deprecate and remove pth files https://bugs.python.org/issue33944 8 msgs #34013: Inconsistent SyntaxError for print https://bugs.python.org/issue34013 8 msgs #34047: IDLE: on macOS, scroll slider 'sticks' at bottom of file https://bugs.python.org/issue34047 8 msgs Issues closed (41) ================== #20760: test_compileall test getting failed on 3.4 RC https://bugs.python.org/issue20760 closed by vstinner #24596: Script globals in a GC cycle not finalized when exiting with S https://bugs.python.org/issue24596 closed by pitrou #25345: Unable to install Python 3.5 on Windows 10 https://bugs.python.org/issue25345 closed by inada.naoki #28874: test_logging fails and freezes https://bugs.python.org/issue28874 closed by vstinner #30183: [HPUX] compilation error in pytime.c with cc compiler https://bugs.python.org/issue30183 closed by vstinner #30516: Documentation for datetime substract operation incorrect? https://bugs.python.org/issue30516 closed by belopolsky #30660: Lossless Optimization of PNG files https://bugs.python.org/issue30660 closed by inada.naoki #31938: Convert selectmodule.c to Argument Clinic https://bugs.python.org/issue31938 closed by taleinat #32568: Fix handling of sizehint=-1 in select.epoll() https://bugs.python.org/issue32568 closed by taleinat #32942: test_script_helper fails on Windows when run from the source c https://bugs.python.org/issue32942 closed by vstinner #33418: Memory leaks in functions https://bugs.python.org/issue33418 closed by inada.naoki #33627: test-complex of test_numeric_tower.test_complex() crashes inte https://bugs.python.org/issue33627 closed by vstinner #33700: [doc] Old version picker don't understand language tags in URL https://bugs.python.org/issue33700 closed by larry #33701: test_datetime crashed (SIGSEGV) on Travis CI https://bugs.python.org/issue33701 closed by vstinner #33899: Tokenize module does not mirror "end-of-input" is newline beha https://bugs.python.org/issue33899 closed by taleinat #33959: doc Remove time complexity mention from list Glossary entry https://bugs.python.org/issue33959 closed by rhettinger #33968: os.makedirs and empty string https://bugs.python.org/issue33968 closed by emilyemorehouse #33974: _stringify handles quoted strings incorrectly https://bugs.python.org/issue33974 closed by serhiy.storchaka #33978: logging.config.dictConfig with file handler leaks resources https://bugs.python.org/issue33978 closed by vinay.sajip #33984: test_multiprocessing_forkserver leaked [1, 2, 1] memory blocks https://bugs.python.org/issue33984 closed by vstinner #33988: [EASY] [3.7] test_platform fails when run with -Werror https://bugs.python.org/issue33988 closed by vstinner #33996: Crash in gen_send_ex(): _PyErr_GetTopmostException() returns f https://bugs.python.org/issue33996 closed by vstinner #33998: random.randrange completely ignores the step argument when sto https://bugs.python.org/issue33998 closed by rhettinger #34005: Replace inspect.formatargspec by inspect.signature in xmlrpc m https://bugs.python.org/issue34005 closed by remi.lapeyre #34006: Windows HTML Help (chm) has fixed line length https://bugs.python.org/issue34006 closed by terry.reedy #34010: tarfile stream read performance regression https://bugs.python.org/issue34010 closed by inada.naoki #34015: doc Add link to Descriptor HowTo Guide in Data Model https://bugs.python.org/issue34015 closed by rhettinger #34016: Bug in sort() https://bugs.python.org/issue34016 closed by mark.dickinson #34018: SQLite converters are documented to be sensitive to the case o https://bugs.python.org/issue34018 closed by taleinat #34020: Add '%(asctime)s' into default BASIC_FORMAT in logging module https://bugs.python.org/issue34020 closed by vinay.sajip #34024: Builtin types take no keyword arguments anymore https://bugs.python.org/issue34024 closed by ned.deily #34030: easy_install from Python 3.7 (Win64) cannot install modules https://bugs.python.org/issue34030 closed by ned.deily #34039: Loop limited to 1000 https://bugs.python.org/issue34039 closed by steven.daprano #34040: test_multiprocessing_forkserver: test_forkserver_sigkill() fai https://bugs.python.org/issue34040 closed by vstinner #34043: Optimize tarfile uncompression performance https://bugs.python.org/issue34043 closed by inada.naoki #34044: subprocess: reusing STARTUPINFO breaks under 3.7 (Windows) https://bugs.python.org/issue34044 closed by vstinner #34045: DoS due to null pointer dereference in marshal.dumps() https://bugs.python.org/issue34045 closed by serhiy.storchaka #34048: Bad behavior of re.IGNORECASE https://bugs.python.org/issue34048 closed by Jan ??vec #34049: abs() method accept argument that implement __abs__() https://bugs.python.org/issue34049 closed by rhettinger #34053: Support localization of unicode descriptions https://bugs.python.org/issue34053 closed by vstinner #34054: multiprocessing should use time.monotonic() for timeout https://bugs.python.org/issue34054 closed by vstinner From storchaka at gmail.com Fri Jul 6 14:01:01 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 6 Jul 2018 21:01:01 +0300 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: 04.07.18 15:05, Nick Coghlan ????: > So my guess would be that this is a test suite error where we're not > handling the "running in a reproducible build environment with > SOURCE_DATE_EPOCH already set" case. Should SOURCE_DATE_EPOCH be documented together with PYTHONDONTWRITEBYTECODE? From brett at python.org Fri Jul 6 14:17:35 2018 From: brett at python.org (Brett Cannon) Date: Fri, 6 Jul 2018 11:17:35 -0700 Subject: [Python-Dev] Failing tests (on a Linux distro) In-Reply-To: References: <6FF553CD-6580-4939-A5E4-78143633BF1F__13543.8208272264$1530147761$gmane$org@python.org> Message-ID: On Fri, 6 Jul 2018 at 11:02 Serhiy Storchaka wrote: > 04.07.18 15:05, Nick Coghlan ????: > > So my guess would be that this is a test suite error where we're not > > handling the "running in a reproducible build environment with > > SOURCE_DATE_EPOCH already set" case. > > Should SOURCE_DATE_EPOCH be documented together with > PYTHONDONTWRITEBYTECODE? > Since it's not Python-specific I don't know if it's necessary, but it wouldn't hurt either since it is an environment variable that affects Python's behaviour. -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jul 6 14:20:05 2018 From: brett at python.org (Brett Cannon) Date: Fri, 6 Jul 2018 11:20:05 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: On Fri, 6 Jul 2018 at 08:52 Chris Barker - NOAA Federal via Python-Dev < python-dev at python.org> wrote: > Are we just having fun here? > > Or might we actually start using a new naming convention for > the-syntax-formerly-known-as-generator-expressions? > If you can create a groundswell of support then maybe. -Brett > > -CHB > > Sent from my iPhone > > > On Jul 3, 2018, at 11:54 PM, Greg Ewing > wrote: > > > > Steven D'Aprano wrote: > >> - list builder syntax is syntax which returns a list; > >> - dict builder syntax is syntax which returns a dict; > >> - set builder syntax is syntax which returns a set; > >> - generator builder syntax is syntax which returns a generator. > > > > You only get a list/dict/set from the first three after > > you've run the iterators within it, but with a generator > > expression, you already have a generator before you've > > run it. That makes it feel different to me. > > > > -- > > Greg > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.barker%40noaa.gov > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From encukou at gmail.com Fri Jul 6 14:33:26 2018 From: encukou at gmail.com (Petr Viktorin) Date: Fri, 6 Jul 2018 20:33:26 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3E080E.4030107@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> Message-ID: On 07/05/18 13:59, Jeroen Demeyer wrote: > On 2018-07-05 13:32, INADA Naoki wrote: >> Core devs interested in this area is limited resource. > > I know and unfortunately there is nothing that I can do about that. It > would be a pity that PEP 580 (or a variant like PEP 576) is not accepted > simply because no core developer cares enough. Hi, I do care about this, and I'm really sorry I've been mostly silent here. Unfortunately, this is the kind of work that can't be done with a few hours in the evenings, and currently an urgent project is sucking up all the big blocks of time I have :( That project should be done in a month or two, however. > >> As far as I understand, there are some important topics to discuss. >> >> a. Low level calling convention, including argument parsing API. >> b. New API for calling objects without argument tuple and dict. >> c. How more types can support FASTCALL, LOAD_METHOD and CALL_METHOD. >> d. How to reorganize existing builtin types, without breaking stable ABI. > > Right, that's why I wanted PEP 580 to be only about (c) and nothing > else. I made the mistake in PEP 575 of also involving (d). > > I still don't understand why we must finish (a) before we can even start > discussing (c). > >> Reference implementation helps discussion. > > METH_FASTCALL and argument parsing for METH_FASTCALL is already > implemented in CPython. Not in documented public functions, but the > implementation exists. > > And PEP 580 also has a reference implementation: > https://github.com/jdemeyer/cpython/tree/pep580 > > > Jeroen. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/encukou%40gmail.com From brett at python.org Fri Jul 6 14:44:02 2018 From: brett at python.org (Brett Cannon) Date: Fri, 6 Jul 2018 11:44:02 -0700 Subject: [Python-Dev] Examples for PEP 572 In-Reply-To: <20180704164118.0e050c65@fsol> References: <20180704164118.0e050c65@fsol> Message-ID: On Wed, 4 Jul 2018 at 07:42 Antoine Pitrou wrote: > On Wed, 4 Jul 2018 09:43:04 -0300 > Brett Cannon wrote: > > > > I think this is a very key point that the "this is bad" crowd is > > overlooking. Even if this syntax turns out to not be that useful, abusing > > the walrus operator can be fixed with a comment of "hard to follow; > please > > simplify" without teaching any new concepts or idioms > > The same could be said of any language mis-feature. Do we want to > claim that questionable semantics in Javascript and PHP are not a > problem, because the bad constructs can simply be turned away in code > review? > > That sounds like a modern re-phrasing of the old argument, """C is not > dangerous in itself, it's only the fault of incompetent programmers""". > Just replace "incompetent programmers" with "complacent reviewers"... > I would disagree as for some things understanding why it needs a change is much easier than others. For instance saying that some use of := is too convoluted and should be simplified needs much less explanation than other things like why you should typically avoid staticmethod. Or another way to view it, saying := is abused in a review should be universally understood while something else may require a more Python-specific explanation and deeper knowledge. Now none of this isn't to say we should take any idea regardless of potential abuses and maintenance cost. But for me, I don't view this as requiring any innate knowledge to cause people not to abuse it. > > > Another point is we live in a dictatorship by choice, and yet some people > > seem very upset our dictator dictated in the end. > > Not sure what you mean with "by choice". As in people choose to join this team knowing that Guido is the BDFL and what that entails. Since none of us have to be here I view it as a choice to be here. -Brett > When I arrived here, I > certainly wasn't asked whether I wanted the place to be a dictatorship > or not ;-) Granted, I did choose to come here, but not because of a > personal appeal for dictatorship. > > One could claim that the qualities of Python are due to it being a > dictatorship. I think it's impossible to answer that question > rigorously, and all we're left with is our personal feelings and biases > on the subject. > > Regards > > Antoine. > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jul 6 15:11:23 2018 From: brett at python.org (Brett Cannon) Date: Fri, 6 Jul 2018 12:11:23 -0700 Subject: [Python-Dev] Removal of install_misc command from distutils In-Reply-To: References: Message-ID: For referencing, the commit was https://github.com/python/cpython/commit/ef158c3ced3fce39e43f54e8d149dc2714e3456e#diff-ef2e84716aa6196aa0ebf0691e608986 and the issue was https://bugs.python.org/issue29218 . On Thu, 5 Jul 2018 at 11:27 Alexander Belopolsky < alexander.belopolsky at gmail.com> wrote: > I started porting my project [1] to Python 3.7 and came across bpo-29218: > > "The unused distutils install_misc command has been removed." [2] > > Historically, the distutils package was very conservative about changes > because many 3rd party packages extended it in ways unforeseen by the > Python core developers. As far as I can tell, this removal was done > without a deprecation period or any public discussion. > > The comment above the command class [3] was there for 18 years and > promised to "keep it around for the time being." Why did it suddenly > become necessary to remove it in 3.7? Is shedding 20 lines of code really > worth the risk of breaking user code? > > [1]: https://github.com/KxSystems/pyq > [2]: https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals > [3]: > https://github.com/python/cpython/commit/aae45f93e7b7708deb1ce9d69b58fa029106613d > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Fri Jul 6 15:17:43 2018 From: brett at python.org (Brett Cannon) Date: Fri, 6 Jul 2018 12:17:43 -0700 Subject: [Python-Dev] PEP 484 In-Reply-To: <5B3DA6F3.5030808@canterbury.ac.nz> References: <5B3DA6F3.5030808@canterbury.ac.nz> Message-ID: On Wed, 4 Jul 2018 at 22:07 Greg Ewing wrote: > Shawn Chen wrote: > > The PEP 484 is proposing a type hint which can annotate the type of each > > parameters. How ever code written in this format can not be run for > > python3.5 and below. > > You're a bit late. Parameter annotations have been a part > of the language since at least 3.1. PEP 484 just codifies > a way of using them to represent types. > > Also, PEP 484 does specify a way of using comments to > indicate types. > Actually it does in https://www.python.org/dev/peps/pep-0484/#suggested-syntax-for-python-2-7-and-straddling-code and https://www.python.org/dev/peps/pep-0484/#type-comments. So the OP's desire to specify in a comment is already specified. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Fri Jul 6 15:29:47 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Fri, 6 Jul 2018 22:29:47 +0300 Subject: [Python-Dev] A "day of silence" on PEP 572? Message-ID: On 06.07.2018 1:40, Guido van Rossum wrote: > Thanks you for writing up a proposal. There have been many proposals > made, including 'EXPR as NAME', similar to yours. It even has a small > section in the PEP: > https://www.python.org/dev/peps/pep-0572/#alternative-spellings. It's > really hard to choose between alternatives, but all things considered > I have decided in favor of `NAME := EXPR` instead. Your efforts are > appreciated but you would just be wasting your time if you wrote a > PEP. If you're interested in helping out, would you be interested in > working on the implementation of PEP 572? > Maybe we should call for subj? Not a day most probably, rather however much time is needed. AFAICS, all the arguments have already been told and retold. So we should probably give Guido some peace of mind until he officially accepts the PEP or whatever he decides. -- Regards, Ivan From brett at python.org Fri Jul 6 15:30:59 2018 From: brett at python.org (Brett Cannon) Date: Fri, 6 Jul 2018 12:30:59 -0700 Subject: [Python-Dev] Tone it down on Twitter? In-Reply-To: References: <20180704154821.GA7081@bytereef.org> Message-ID: While I agree with Antoine that the wording by Serhiy on Twitter was unnecessary, the tone in responses here are also unnecessary. As has been discussed both here on this mailing list and at the PyCon US language summit, reacting to actual or perceived rudeness with more rudeness is not reasonable behaviour. If something upsets you and you want to discuss it then that's fine, but wait until you can do so with a cool head (I avoided personal email for 2 days because of this email thread in order to not react harshly to it myself). IOW believing you have been disrespected does suddenly give you permission to be disrespectful as well. That simply leads to everyone not caring about your complaint and this whole environment ends up feeling hostile. On Wed, 4 Jul 2018 at 12:15 Ivan Levkivskyi wrote: > Just to clarify, the club of three in _my_ twit are the three authors of > the PEP. > > Also, how about you not telling me what to say? > > -- > Ivan > > > > On 4 July 2018 at 16:48, Stefan Krah wrote: > >> >> Apparently I have made it into "club of three who don't care much about >> opinions of others" for the crime of a single +0.5 for PEP-572 without >> participating in the discussion at all (neither did Jason). >> >> https://twitter.com/SerhiyStorchaka/status/1014274554452209665 >> >> This is a new high for Twitter gossip. Well done. Perhaps in the next >> vote >> the politbureau can indicate the intended outcome beforehand so we know >> how >> to vote. >> >> >> >> Thanks, >> >> Stefan Krah >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/levkivskyi%40gmail.com >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Jul 6 15:39:35 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 6 Jul 2018 21:39:35 +0200 Subject: [Python-Dev] A "day of silence" on PEP 572? References: Message-ID: <20180706213935.62653fa0@fsol> I would gladly welcome a PEP 572-less development mailing-list. Regards Antoine. On Fri, 6 Jul 2018 22:29:47 +0300 Ivan Pozdeev via Python-Dev wrote: > On 06.07.2018 1:40, Guido van Rossum wrote: > > > Thanks you for writing up a proposal. There have been many proposals > > made, including 'EXPR as NAME', similar to yours. It even has a small > > section in the PEP: > > https://www.python.org/dev/peps/pep-0572/#alternative-spellings. It's > > really hard to choose between alternatives, but all things considered > > I have decided in favor of `NAME := EXPR` instead. Your efforts are > > appreciated but you would just be wasting your time if you wrote a > > PEP. If you're interested in helping out, would you be interested in > > working on the implementation of PEP 572? > > > Maybe we should call for subj? Not a day most probably, rather however > much time is needed. > > AFAICS, all the arguments have already been told and retold. So we > should probably give Guido some peace of mind until he officially > accepts the PEP or whatever he decides. > > -- > Regards, > Ivan From brett at python.org Fri Jul 6 15:45:40 2018 From: brett at python.org (Brett Cannon) Date: Fri, 6 Jul 2018 12:45:40 -0700 Subject: [Python-Dev] Tone it down on Twitter? In-Reply-To: References: <20180704154821.GA7081@bytereef.org> Message-ID: On Wed, 4 Jul 2018 at 11:54 Mark Lawrence wrote: > On 04/07/18 16:48, Stefan Krah wrote: > > > > Apparently I have made it into "club of three who don't care much about > > opinions of others" for the crime of a single +0.5 for PEP-572 without > > participating in the discussion at all (neither did Jason). > > > > https://twitter.com/SerhiyStorchaka/status/1014274554452209665 > > > > This is a new high for Twitter gossip. Well done. Perhaps in the next > vote > > the politbureau can indicate the intended outcome beforehand so we know > how > > to vote. > > > > > > > > Thanks, > > > > Stefan Krah > > > > > > > > Quite frankly I don't give a hoot what gets said on twitter, as it > appears to be reserved for morons like the current president of the USA. > I happen to be on Twitter so you just called me a moron, Mark (as well as many other core developers). While I think Stefan's response was uncalled for, he at least felt like the injured party and thus his response could potentially be attributed an impassioned response that should not have occurred. IOW while I understand why the reaction occurred, that doesn't excuse it while warranting a warning of that fact and that it shouldn't be considered acceptable behaviour overall. Your response, though, definitely crossed a line as you were insulting as well as not directly involved in the issue at hand, which means you had no direct motivation to be upset other than to prove a point and be rude entirely on purpose. IOW you were just piling on the rudeness for no good reason. For that reason, consider this a warning from me that this kind of behaviour is simply not acceptable. If it happens again I will request at least a 2 month ban for you from the mailing list. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Jul 6 16:25:25 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 06 Jul 2018 13:25:25 -0700 Subject: [Python-Dev] Tone it down on Twitter? In-Reply-To: References: <20180704154821.GA7081@bytereef.org> Message-ID: <5B3FD035.7080106@stoneleaf.us> On 07/06/2018 12:30 PM, Brett Cannon wrote: > IOW believing you have been disrespected does not > suddenly give you permission to be disrespectful as well. Thanks, Brett, for the reminder. I know we all have rough days, and it's easy to forget to take a break before responding. -- ~Ethan~ From v+python at g.nevcal.com Fri Jul 6 16:40:43 2018 From: v+python at g.nevcal.com (Glenn Linderman) Date: Fri, 6 Jul 2018 13:40:43 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: <0758cec1-1b39-41c3-de12-3f32125f3d2d@g.nevcal.com> On 7/6/2018 11:20 AM, Brett Cannon wrote: > > > On Fri, 6 Jul 2018 at 08:52 Chris Barker - NOAA Federal via Python-Dev > > wrote: > > Are we just having fun here? > > Or might we actually start using a new naming convention for > the-syntax-formerly-known-as-generator-expressions? > > > If you can create a groundswell of support then maybe. "Comprehension" was an incomprehensible term to me, when I first heard it. After reading the documentation, I comprehended it, but the term is meaningless. "generator expression" is actually a little more comprehensible, by analogy to a power (electricity) generator... values are generated. But comprehensions would have been more intuitively understood by me, had they been called "list generators" or "dict generators" or "set generators". The difference between comprehensions and generators seems to be one of timing and sequencing: the comprehensions are comprehended as fast as possible, while generators are lazy (unlike those electricity generators, unless you turn them on and off repeatedly). So neither term is very intuitive to "plain old programmers" unless perhaps they happen to know some of the advanced math concepts where the term "comprehension" is claimed to come from (obviously, in spite of a math degree from some years back, I never encountered that term). Focusing just on lists, at this time, "list builder" would be better than "list generator" because using "generator" to replace "comprehension" would be confusing because of the history of the terminology as used/documented currently. A generator seems to be a "lazy list builder". If the names are changed, for a time both terminologies would have to coexist, so it would be critical to use terms not already in use in Python terminology. I would find it an improvement to use terms like "list builder" rather than "comprehension". > > -Brett > > > -CHB > > Sent from my iPhone > > > On Jul 3, 2018, at 11:54 PM, Greg Ewing > > > wrote: > > > > Steven D'Aprano wrote: > >> - list builder syntax is syntax which returns a list; > >> - dict builder syntax is syntax which returns a dict; > >> - set builder syntax is syntax which returns a set; > >> - generator builder syntax is syntax which returns a generator. > > > > You only get a list/dict/set from the first three after > > you've run the iterators within it, but with a generator > > expression, you already have a generator before you've > > run it. That makes it feel different to me. > > > > -- > > Greg > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jul 6 17:12:08 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 6 Jul 2018 14:12:08 -0700 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3F3B4E.6080508@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3E341E.70402@UGent.be> <1cb5933b3ce7418d90aa8b9488efc8e7@xmail103.UGent.be> <5B3F3B4E.6080508@UGent.be> Message-ID: On Fri, Jul 6, 2018 at 2:52 AM Jeroen Demeyer wrote: > On 2018-07-05 21:57, Guido van Rossum wrote: > > Would it be possible to get outside experts to help? > > I don't understand what you mean: to help with what? Designing the PEP? > Discussing the PEP? Accepting the PEP? Lobbying Python core devs? > It's your PEP. And you seem to be struggling with something. But I can't tell quite what it is you're struggling with. So if my suggestion isn't helpful I won't be offended if you ignore it. At the same time I assume you want your PEP accepted. And PEP 1 clearly states that that acceptance is up to me, or up to a BDFL-delegate appointed by me. In this case I don't feel I understand the details well enough to decide by myself, so I would either need a BDFL-delegate or a lot of help from people I trust. > The Cython developers (in particular Stefan Behnel) certainly support my > work. I have talked with them in person at a workshop and they posted a > few emails to python-dev and they also gave me some personal comments > about PEP 580. > And how do they feel about PEP 576? I'd like to see some actual debate of the pros and cons of the details of PEP 576 vs. PEP 580. So far I mostly see you and INADA Naoki disagreeing about process, which doesn't give me the feeling that there is consensus. As for NumPy, one obvious place where some ideas of PEP 579 could be > applied is ufuncs. However, typically ufuncs are not *called* in tight > loops, they implement tight loops internally. So I don't think that > there is pressing need for anything like PEP 580 in NumPy. > OK, so is it your claim that the NumPy developers don't care about which one of these PEPs is accepted or even whether one is accepted at all? I'd like to hear some NumPy developer agree with that. Finally, I'm unclear on what your opinion on FASTCALL actually is. In the other thread you seemed to praise it as "just perfect" (not a quote, just what I came away with from reading the thread). Yet earlier in *this* thread you seemed to claim that PEP 580 requires changes ro FASTCALL. So which is it? -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jcgoble3 at gmail.com Fri Jul 6 17:16:57 2018 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Fri, 6 Jul 2018 17:16:57 -0400 Subject: [Python-Dev] A "day of silence" on PEP 572? In-Reply-To: <20180706213935.62653fa0@fsol> References: <20180706213935.62653fa0@fsol> Message-ID: On Fri, Jul 6, 2018, 3:41 PM Antoine Pitrou wrote: > > I would gladly welcome a PEP 572-less development mailing-list. > +1. Speaking as a lurker with little interest in the particular topic, PEP 572 has almost driven me to unsubscribe. It's not the massive number of posts that is annoying, but the fact that seemingly (perhaps literally?) scores of separate threads have been created for it. A single thread, or a small number of threads, are easily muted, but when multiple new threads are created seemingly daily or weekly for months on end, trying to gain control of my inbox feels like an endless game of Whac-A-Mole [1] with no prizes, and that's not a good thing. [1] Being a global list, here's a link for anyone not familiar with the game: https://en.m.wikipedia.org/wiki/Whac-A-Mole > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Fri Jul 6 17:50:46 2018 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 6 Jul 2018 23:50:46 +0200 Subject: [Python-Dev] A "day of silence" on PEP 572? In-Reply-To: <20180706213935.62653fa0@fsol> References: <20180706213935.62653fa0@fsol> Message-ID: Last week I proposed to create a mailing list dedicated to discuss only the PEP 572, but nobody reacted to my idea (on python-commiters). Then Guido van Rossum announced his intent to approve it. So I am not sure if a mailing list is still needed if the PEP will be approved soon. Victor Le vendredi 6 juillet 2018, Antoine Pitrou a ?crit : > > I would gladly welcome a PEP 572-less development mailing-list. > > Regards > > Antoine. > > > On Fri, 6 Jul 2018 22:29:47 +0300 > Ivan Pozdeev via Python-Dev wrote: >> On 06.07.2018 1:40, Guido van Rossum wrote: >> >> > Thanks you for writing up a proposal. There have been many proposals >> > made, including 'EXPR as NAME', similar to yours. It even has a small >> > section in the PEP: >> > https://www.python.org/dev/peps/pep-0572/#alternative-spellings. It's >> > really hard to choose between alternatives, but all things considered >> > I have decided in favor of `NAME := EXPR` instead. Your efforts are >> > appreciated but you would just be wasting your time if you wrote a >> > PEP. If you're interested in helping out, would you be interested in >> > working on the implementation of PEP 572? >> > >> Maybe we should call for subj? Not a day most probably, rather however >> much time is needed. >> >> AFAICS, all the arguments have already been told and retold. So we >> should probably give Guido some peace of mind until he officially >> accepts the PEP or whatever he decides. >> >> -- >> Regards, >> Ivan > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Fri Jul 6 17:57:37 2018 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 6 Jul 2018 23:57:37 +0200 Subject: [Python-Dev] Removal of install_misc command from distutils In-Reply-To: References: Message-ID: Hello, I am not sure of what you propose. Do you want to get the feature back in Python 3.7.1? If yes, should it start to emit a deprection warning? Did you manage to workaround the removal? If yes, maybe we can add more doc to the Porting section of What's New in Python 3.7? Victor Le jeudi 5 juillet 2018, Alexander Belopolsky < alexander.belopolsky at gmail.com> a ?crit : > I started porting my project [1] to Python 3.7 and came across bpo-29218: > > "The unused distutils install_misc command has been removed." [2] > > Historically, the distutils package was very conservative about changes because many 3rd party packages extended it in ways unforeseen by the Python core developers. As far as I can tell, this removal was done without a deprecation period or any public discussion. > > The comment above the command class [3] was there for 18 years and promised to "keep it around for the time being." Why did it suddenly become necessary to remove it in 3.7? Is shedding 20 lines of code really worth the risk of breaking user code? > > [1]: https://github.com/KxSystems/pyq > [2]: https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals > [3]: https://github.com/python/cpython/commit/aae45f93e7b7708deb1ce9d69b58fa029106613d > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Fri Jul 6 18:02:37 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 7 Jul 2018 00:02:37 +0200 Subject: [Python-Dev] A "day of silence" on PEP 572? In-Reply-To: References: <20180706213935.62653fa0@fsol> Message-ID: <20180707000237.20cd23ff@fsol> I would simply suggest that people who want to improve or fine-tune the PEP post pull requests for that (or contact the PEP's authors privately). As for the other kinds of threads, as much as I dislike PEP 572, they are useless now. Regards Antoine. On Fri, 6 Jul 2018 23:50:46 +0200 Victor Stinner wrote: > Last week I proposed to create a mailing list dedicated to discuss only the > PEP 572, but nobody reacted to my idea (on python-commiters). Then Guido > van Rossum announced his intent to approve it. So I am not sure if a > mailing list is still needed if the PEP will be approved soon. > > Victor > > Le vendredi 6 juillet 2018, Antoine Pitrou a ?crit : > > > > I would gladly welcome a PEP 572-less development mailing-list. > > > > Regards > > > > Antoine. > > > > > > On Fri, 6 Jul 2018 22:29:47 +0300 > > Ivan Pozdeev via Python-Dev wrote: > >> On 06.07.2018 1:40, Guido van Rossum wrote: > >> > >> > Thanks you for writing up a proposal. There have been many proposals > >> > made, including 'EXPR as NAME', similar to yours. It even has a small > >> > section in the PEP: > >> > https://www.python.org/dev/peps/pep-0572/#alternative-spellings. It's > >> > really hard to choose between alternatives, but all things considered > >> > I have decided in favor of `NAME := EXPR` instead. Your efforts are > >> > appreciated but you would just be wasting your time if you wrote a > >> > PEP. If you're interested in helping out, would you be interested in > >> > working on the implementation of PEP 572? > >> > > >> Maybe we should call for subj? Not a day most probably, rather however > >> much time is needed. > >> > >> AFAICS, all the arguments have already been told and retold. So we > >> should probably give Guido some peace of mind until he officially > >> accepts the PEP or whatever he decides. > >> > >> -- > >> Regards, > >> Ivan > > > > > > > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > > > From vstinner at redhat.com Fri Jul 6 18:26:16 2018 From: vstinner at redhat.com (Victor Stinner) Date: Sat, 7 Jul 2018 00:26:16 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: References: <5B3E30D1.3080706@UGent.be> Message-ID: Hi, I designed FASTCALL with the help of Serhiy for keywords. I prepared a long email reply, but I found an opportunity for optimisation on **kwargs and I need time to see how to optimize it. Maybe there is a need for passing **kwargs as a dict at C level, but use FASTCALL for positional arguments? I only know dict.update() which would benefit of that. All other functions are fine with FASTCALL for keywords. Victor Le vendredi 6 juillet 2018, Guido van Rossum a ?crit : > I'm not the world's leading expert on Python bytecode anymore, but unless there's something I'm missing your conclusion looks eminently reasonable, and so I expect you'll get very little traction on this thread. (If you had wanted to get a megathread you should have written "FASTCALL considered harmful". :-) > > I think there was one person in another thread (INADA Naoki?) who thought METH_FASTCALL could use improvements. Maybe that person can write back to this thread? Or perhaps Victor Stinner (who seems to have touched it last) has a suggestion for what could be improved about it? > --Guido > > On Thu, Jul 5, 2018 at 7:55 AM Jeroen Demeyer wrote: >> >> Hello all, >> >> As discussed in some other threads ([1], [2]), we should discuss the >> METH_FASTCALL calling convention. >> >> For passing only positional arguments, a C array of Python objects is >> used, which is as fast as it can get. When the Python interpreter calls >> a function, it builds that C array on the interpreter stack: >> >> >>> from dis import dis >> >>> def f(x, y): return g(x, y, 12) >> >>> dis(f) >> 1 0 LOAD_GLOBAL 0 (g) >> 2 LOAD_FAST 0 (x) >> 4 LOAD_FAST 1 (y) >> 6 LOAD_CONST 1 (12) >> 8 CALL_FUNCTION 3 >> 10 RETURN_VALUE >> >> A C array can also easily and efficiently be handled by the C function >> receiving it. So I consider this uncontroversial. >> >> The convention for METH_FASTCALL|METH_KEYWORDS is that keyword *names* >> are passed as a tuple and keyword *values* in the same C array with >> positional arguments. An example: >> >> >>> from dis import dis >> >>> def f(x, y, z): return f(x, foo=y, bar=z) >> >>> dis(f) >> 1 0 LOAD_GLOBAL 0 (f) >> 2 LOAD_FAST 0 (x) >> 4 LOAD_FAST 1 (y) >> 6 LOAD_FAST 2 (z) >> 8 LOAD_CONST 1 (('foo', 'bar')) >> 10 CALL_FUNCTION_KW 3 >> 12 RETURN_VALUE >> >> This is pretty clever: it exploits the fact that ('foo', 'bar') is a >> constant tuple stored in f.__code__.co_consts. Also, a tuple can be >> efficiently handled by the called code: it is essentially a thin wrapper >> around a C array of Python objects. So this works well. >> >> The only case when this handling of keywords is suboptimal is when using >> **kwargs. In that case, a dict must be converted to a tuple. It looks >> hard to me to support efficiently both the case of fixed keyword >> arguments (f(foo=x)) and a keyword dict (f(**kwargs)). Since the former >> is more common than the latter, the current choice is optimal. >> >> In other words: I see nothing to improve in the calling convention of >> METH_FASTCALL. I suggest to keep it and make it public as-is. >> >> >> Jeroen. >> >> >> [1] https://mail.python.org/pipermail/python-dev/2018-June/153945.html >> [2] https://mail.python.org/pipermail/python-dev/2018-July/154251.html >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/guido%40python.org > > > -- > --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Fri Jul 6 19:02:41 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 7 Jul 2018 01:02:41 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3E341E.70402@UGent.be> <1cb5933b3ce7418d90aa8b9488efc8e7@xmail103.UGent.be> <5B3F3B4E.6080508@UGent.be> Message-ID: <5B3FF511.7060205@UGent.be> On 2018-07-06 23:12, Guido van Rossum wrote: > It's your PEP. And you seem to be struggling with something. But I can't > tell quite what it is you're struggling with. To be perfectly honest (no hard feelings though!): what I'm struggling with is getting feedback (either positive or negative) from core devs about the actual PEP 580. > At the same time I assume you want your PEP accepted. As I also said during the PEP 575 discussion, my real goal is to solve a concrete problem, not to push my personal PEP. I still think that PEP 580 is the best solution but I welcome other suggestions. > And how do they feel about PEP 576? I'd like to see some actual debate > of the pros and cons of the details of PEP 576 vs. PEP 580. I started this thread to do precisely that. My opinion: PEP 580 has zero performance cost, while PEP 576 does make performance for bound methods worse (there is no reference implementation of the new PEP 576 yet, so that's hard to quantify for now). PEP 580 is also more future-proof: it defines a new protocol which can easily be extended in the future. PEP 576 just builds on PyMethodDef which cannot be extended because of ABI compatibility (putting __text_signature__ and __doc__ in the same C string is a good symptom of that). This extensibility is important because I want PEP 580 to be the first in a series of PEPs working out this new protocol. See PEP 579 for the bigger picture. One thing that might count against PEP 580 is that it defines a whole new protocol, which could be seen as too complicated. However, it must be this complicated because it is meant to generalize the current behavior and optimizations of built-in functions and methods. There are lots of little tricks currently in CPython that must be "ported" to the new protocol. > OK, so is it your claim that the NumPy developers don't care about which > one of these PEPs is accepted or even whether one is accepted at all? I don't know, I haven't contacted any NumPy devs yet, so that was just my personal feeling. These PEPs are about optimizing callables and NumPy isn't really about callables. I think that the audience for PEP 580 is mostly compilers (Cython for sure but possibly also Pythran, numba, cppyy, ...). Also certain C classes like functools.lru_cache could benefit from it. > Yet earlier in > *this* thread you seemed to claim that PEP 580 requires changes ro > FASTCALL. I don't know what you mean with that. But maybe it's also confusing because "FASTCALL" can mean different things: it can refer to a PyMethodDef (used by builtin_function_or_method and method_descriptor) with the METH_FASTCALL flag set. It can also refer to a more general API like _PyCFunction_FastCallKeywords, which supports METH_FASTCALL but also other calling conventions like METH_VARARGS. I don't think that METH_FASTCALL should be changed (and PEP 580 isn't really about that at all). For the latter, I'm suggesting some API changes but nothing fundamental: mainly replacing the 5 existing private functions _PyCFunction_FastCallKeywords, _PyCFunction_FastCallDict, _PyMethodDescr_FastCallKeywords, _PyMethodDef_RawFastCallKeywords, _PyMethodDef_RawFastCallDict by 1 public function PyCCall_FASTCALL. Hopefully this clears some things up, Jeroen. From guido at python.org Fri Jul 6 19:10:36 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 6 Jul 2018 16:10:36 -0700 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3FF511.7060205@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3E341E.70402@UGent.be> <1cb5933b3ce7418d90aa8b9488efc8e7@xmail103.UGent.be> <5B3F3B4E.6080508@UGent.be> <5B3FF511.7060205@UGent.be> Message-ID: Thanks, I will wait until there is a conclusion to the debate. On Fri, Jul 6, 2018 at 4:05 PM Jeroen Demeyer wrote: > On 2018-07-06 23:12, Guido van Rossum wrote: > > It's your PEP. And you seem to be struggling with something. But I can't > > tell quite what it is you're struggling with. > > To be perfectly honest (no hard feelings though!): what I'm struggling > with is getting feedback (either positive or negative) from core devs > about the actual PEP 580. > > > At the same time I assume you want your PEP accepted. > > As I also said during the PEP 575 discussion, my real goal is to solve a > concrete problem, not to push my personal PEP. I still think that PEP > 580 is the best solution but I welcome other suggestions. > > > And how do they feel about PEP 576? I'd like to see some actual debate > > of the pros and cons of the details of PEP 576 vs. PEP 580. > > I started this thread to do precisely that. > > My opinion: PEP 580 has zero performance cost, while PEP 576 does make > performance for bound methods worse (there is no reference > implementation of the new PEP 576 yet, so that's hard to quantify for > now). PEP 580 is also more future-proof: it defines a new protocol which > can easily be extended in the future. PEP 576 just builds on PyMethodDef > which cannot be extended because of ABI compatibility (putting > __text_signature__ and __doc__ in the same C string is a good symptom of > that). This extensibility is important because I want PEP 580 to be the > first in a series of PEPs working out this new protocol. See PEP 579 for > the bigger picture. > > One thing that might count against PEP 580 is that it defines a whole > new protocol, which could be seen as too complicated. However, it must > be this complicated because it is meant to generalize the current > behavior and optimizations of built-in functions and methods. There are > lots of little tricks currently in CPython that must be "ported" to the > new protocol. > > > OK, so is it your claim that the NumPy developers don't care about which > > one of these PEPs is accepted or even whether one is accepted at all? > > I don't know, I haven't contacted any NumPy devs yet, so that was just > my personal feeling. These PEPs are about optimizing callables and NumPy > isn't really about callables. I think that the audience for PEP 580 is > mostly compilers (Cython for sure but possibly also Pythran, numba, > cppyy, ...). Also certain C classes like functools.lru_cache could > benefit from it. > > > Yet earlier in > > *this* thread you seemed to claim that PEP 580 requires changes ro > > FASTCALL. > > I don't know what you mean with that. But maybe it's also confusing > because "FASTCALL" can mean different things: it can refer to a > PyMethodDef (used by builtin_function_or_method and method_descriptor) > with the METH_FASTCALL flag set. It can also refer to a more general API > like _PyCFunction_FastCallKeywords, which supports METH_FASTCALL but > also other calling conventions like METH_VARARGS. > > I don't think that METH_FASTCALL should be changed (and PEP 580 isn't > really about that at all). For the latter, I'm suggesting some API > changes but nothing fundamental: mainly replacing the 5 existing private > functions _PyCFunction_FastCallKeywords, _PyCFunction_FastCallDict, > _PyMethodDescr_FastCallKeywords, _PyMethodDef_RawFastCallKeywords, > _PyMethodDef_RawFastCallDict by 1 public function PyCCall_FASTCALL. > > > Hopefully this clears some things up, > Jeroen. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Fri Jul 6 19:16:58 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Fri, 6 Jul 2018 19:16:58 -0400 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: On 7/6/2018 11:51 AM, Chris Barker - NOAA Federal via Python-Dev wrote: via phone... > Are we just having fun here? I floated the idea as a trial balloon to see what response it got. > Or might we actually start using a new naming convention for > the-syntax-formerly-known-as-generator-expressions? Since Guido, the first respondent, did not immediately shoot the idea down, I intend to flesh it out and make it more concrete. >> On Jul 3, 2018, at 11:54 PM, Greg Ewing wrote: >> >> Steven D'Aprano wrote: >>> - list builder syntax is syntax which returns a list; >>> - dict builder syntax is syntax which returns a dict; >>> - set builder syntax is syntax which returns a set; >>> - generator builder syntax is syntax which returns a generator. To expand on this: an iterable represents a collection of information objects. Some iterables are concrete collections. Others are more abstract, generating objects on demand in an order that may or may not be significant. Set builders in math define a set in terms of 1 set, 0 to 1 filters, and 1 transform: defined-set = map(tranform, filter(predicate, known-set)). (One could say that there is always a filter, which defaults to passing everything.) Python builders generalize the type 'set' to 'iterable' and the first and second numbers 1 to n and specify a particular nested order of iteration and filtration. For n left as 1, the type generalization is new_iterable = output_type(map(transform, filter(predicate, iter(iterable)). I omitted above the potential dependence of iterable, predicate, and transform pre-existing arguments. For generator builders, define output type 'generator' as a identity function when the input is a generator. The generalization to n > 1 is tricky to specify with functions call, as I did above, because the filtered iterations are nested rather than crossed. Consequently, each iterable and filter (as well as the tranform) could depend not only on values existing before the top call but also the current values of surrounding iterations. >> You only get a list/dict/set from the first three after >> you've run the iterators within it, but with a generator >> expression, you already have a generator before you've >> run it. That makes it feel different to me. What all 4 results have in common is that they are (mutable) iterables produced from iterators and other inputs with builder syntax. Aside from that, it is true that there are differences between concrete iterables like set, list, and dict versus generator iterators. But to me, this is secondary in this context. One could note that lists and dicts can be subscripted, sets and generators cannot. Or note that dict builders are 'different' because they use the funny dict item ':' notation instead of the (key,value) notation that would make {dict-builder} = dict(dict-builder) true without needing an asterisk. (But then something else would be needed to mark {(k,v) for k,v in it} as a dict rather than set. The use of ':' is quite clever.) -- Terry Jan Reedy From rymg19 at gmail.com Fri Jul 6 19:18:28 2018 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Fri, 06 Jul 2018 18:18:28 -0500 Subject: [Python-Dev] A "day of silence" on PEP 572? In-Reply-To: <20180707000237.20cd23ff@fsol> References: <20180706213935.62653fa0@fsol> <20180707000237.20cd23ff@fsol> Message-ID: <16471e3bda0.27a3.db5b03704c129196a4e9415e55413ce6@gmail.com> On July 6, 2018 5:04:05 PM Antoine Pitrou wrote: > > (or contact the PEP's authors > privately). > Hoenstly, this feels like a recipe for a disaster... > As for the other kinds of threads, as much as I dislike PEP 572, they > are useless now. > > Regards > > Antoine. > > > On Fri, 6 Jul 2018 23:50:46 +0200 > Victor Stinner wrote: >> Last week I proposed to create a mailing list dedicated to discuss only the >> PEP 572, but nobody reacted to my idea (on python-commiters). Then Guido >> van Rossum announced his intent to approve it. So I am not sure if a >> mailing list is still needed if the PEP will be approved soon. >> >> Victor >> >> Le vendredi 6 juillet 2018, Antoine Pitrou a ?crit : >> > >> > I would gladly welcome a PEP 572-less development mailing-list. >> > >> > Regards >> > >> > Antoine. >> > >> > >> > On Fri, 6 Jul 2018 22:29:47 +0300 >> > Ivan Pozdeev via Python-Dev wrote: >> >> On 06.07.2018 1:40, Guido van Rossum wrote: >> >> >> >> > Thanks you for writing up a proposal. There have been many proposals >> >> > made, including 'EXPR as NAME', similar to yours. It even has a small >> >> > section in the PEP: >> >> > https://www.python.org/dev/peps/pep-0572/#alternative-spellings. It's >> >> > really hard to choose between alternatives, but all things considered >> >> > I have decided in favor of `NAME := EXPR` instead. Your efforts are >> >> > appreciated but you would just be wasting your time if you wrote a >> >> > PEP. If you're interested in helping out, would you be interested in >> >> > working on the implementation of PEP 572? >> >> > >> >> Maybe we should call for subj? Not a day most probably, rather however >> >> much time is needed. >> >> >> >> AFAICS, all the arguments have already been told and retold. So we >> >> should probably give Guido some peace of mind until he officially >> >> accepts the PEP or whatever he decides. >> >> >> >> -- >> >> Regards, >> >> Ivan >> > >> > >> > >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com >> > >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/rymg19%40gmail.com From steve at holdenweb.com Fri Jul 6 19:27:59 2018 From: steve at holdenweb.com (Steve Holden) Date: Sat, 7 Jul 2018 00:27:59 +0100 Subject: [Python-Dev] A "day of silence" on PEP 572? In-Reply-To: <16471e3bda0.27a3.db5b03704c129196a4e9415e55413ce6@gmail.com> References: <20180706213935.62653fa0@fsol> <20180707000237.20cd23ff@fsol> <16471e3bda0.27a3.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: On Sat, Jul 7, 2018 at 12:18 AM, Ryan Gonzalez wrote: > On July 6, 2018 5:04:05 PM Antoine Pitrou wrote: > > >> (or contact the PEP's authors >> privately). >> >> > Hoenstly, this feels like a recipe for a disaster... > > ?Many of the people who have strong opinions in this know the PEP authors from years of working together.? ?They might feel that personal channels are appropriate.? I'd agree it would be a bit presumptuous and spammy of others to use them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jul 6 19:31:36 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 6 Jul 2018 16:31:36 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: On Fri, Jul 6, 2018 at 4:19 PM Terry Reedy wrote: > Since Guido, the first respondent, did not immediately shoot the idea > down, I intend to flesh it out and make it more concrete. > Maybe I should have shot it down. The term is entrenched in multiple languages by now (e.g. https://en.wikipedia.org/wiki/List_comprehension). Regarding "list builder" one could argue that it would just add more confusion, since there's already an unrelated Builder Pattern ( https://en.wikipedia.org/wiki/Builder_pattern) commonly used in Java. (Though I worry about the presence of a Python example in that Wikipedia page. :-) Also, "generator builder" is not much more expressive than "generator expression", and the key observation that led to this idea was that it's such a mouthful to say "comprehensions and generator expressions". Maybe it's not too late to start calling the latter "generator comprehensions" so that maybe by the year 2025 we can say "comprehensions" and everyone will understand we mean all four types? FWIW more people should start using "list display" etc. for things like [a, b, c]. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From alexander.belopolsky at gmail.com Fri Jul 6 19:43:16 2018 From: alexander.belopolsky at gmail.com (Alexander Belopolsky) Date: Fri, 6 Jul 2018 19:43:16 -0400 Subject: [Python-Dev] Removal of install_misc command from distutils In-Reply-To: References: Message-ID: Honestly, I did not realize that 3.7 has been released by the time I wrote my e-mail. I think it will be prudent to get this command back in Python 3.7.1. My work-around was to simply copy the 20-something lines that define install_misc from Python 3.6 to my setup.py file. It was my impression that it is precisely due to situations like this, distutils was considered more or less frozen for development and all new features went to setuptools. On Fri, Jul 6, 2018 at 5:57 PM Victor Stinner wrote: > > Hello, > > I am not sure of what you propose. Do you want to get the feature back in Python 3.7.1? If yes, should it start to emit a deprection warning? > > Did you manage to workaround the removal? If yes, maybe we can add more doc to the Porting section of What's New in Python 3.7? > > Victor > > Le jeudi 5 juillet 2018, Alexander Belopolsky < alexander.belopolsky at gmail.com> a ?crit : > > I started porting my project [1] to Python 3.7 and came across bpo-29218: > > > > "The unused distutils install_misc command has been removed." [2] > > > > Historically, the distutils package was very conservative about changes because many 3rd party packages extended it in ways unforeseen by the Python core developers. As far as I can tell, this removal was done without a deprecation period or any public discussion. > > > > The comment above the command class [3] was there for 18 years and promised to "keep it around for the time being." Why did it suddenly become necessary to remove it in 3.7? Is shedding 20 lines of code really worth the risk of breaking user code? > > > > [1]: https://github.com/KxSystems/pyq > > [2]: https://docs.python.org/3/whatsnew/3.7.html#api-and-feature-removals > > [3]: https://github.com/python/cpython/commit/aae45f93e7b7708deb1ce9d69b58fa029106613d > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Fri Jul 6 19:58:23 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Sat, 7 Jul 2018 02:58:23 +0300 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> On 07.07.2018 2:31, Guido van Rossum wrote: > On Fri, Jul 6, 2018 at 4:19 PM Terry Reedy > wrote: > > Since Guido, the first respondent, did not immediately shoot the idea > down, I intend to flesh it out and make it more concrete. > > > Maybe I should have shot it down. The term is entrenched in multiple > languages by now (e.g. > https://en.wikipedia.org/wiki/List_comprehension). Regarding "list > builder" one could argue that it would just add more confusion, since > there's already an unrelated Builder Pattern > (https://en.wikipedia.org/wiki/Builder_pattern) commonly used in Java. > (Though I worry about the presence of a Python example in that > Wikipedia page. :-) According to https://en.wikipedia.org/wiki/List_comprehension#History, the term's known from at least 1977 and comes from such influential languages as NPL, Miranda and Haskell. So it's not you to blame for it :-) > > Also, "generator builder" is not much more expressive than "generator > expression", "generator builder" is simply incorrect. The GE doesn't "build" generators, it's a generator itself. It's a generator _and_ an expression. What could be a more obvious name? This suggestion looks like coming from someone who hasn't quite grasped generators yet. > and the key observation that led to this idea was that it's such a > mouthful to say "comprehensions and generator expressions". Since "X comprehensions" are advertised as and intended to be functionally equivalent to `X(generator expression)', I use just "generator expressions" to refer to all. That's accurate because the common part with the distinctive syntax -- which is the thing referred to when addressing them all -- effectively _is_ a generator expression (the syntax differences in the leading term are insignificant), what wraps it is of no concern. So, no new terms are necessary, but someone who cares may add a note to the docs to this effect. > Maybe it's not too late to start calling the latter "generator > comprehensions" so that maybe by the year 2025 we can say > "comprehensions" and everyone will understand we mean all four types? > > FWIW more people should start using "list display" etc. for things > like [a, b, c]. > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Fri Jul 6 20:20:40 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 7 Jul 2018 09:20:40 +0900 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: References: <5B3E30D1.3080706@UGent.be> Message-ID: On Sat, Jul 7, 2018 at 7:29 AM Victor Stinner wrote: > > Hi, > > I designed FASTCALL with the help of Serhiy for keywords. I prepared a long email reply, but I found an opportunity for optimisation on **kwargs and I need time to see how to optimize it. > > Maybe there is a need for passing **kwargs as a dict at C level, but use FASTCALL for positional arguments? I only know dict.update() which would benefit of that. All other functions are fine with FASTCALL for keywords. > > Victor > I agree with Jeroen. If only few methods can be improved, it's not necessary. METH_VARARGS | METH_KEYWORDS is fine. -- INADA Naoki From vano at mail.mipt.ru Fri Jul 6 20:33:30 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Sat, 7 Jul 2018 03:33:30 +0300 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> Message-ID: <48547fa0-49a3-de25-8c20-a1e109271f95@mail.mipt.ru> On 07.07.2018 2:58, Ivan Pozdeev via Python-Dev wrote: > On 07.07.2018 2:31, Guido van Rossum wrote: >> On Fri, Jul 6, 2018 at 4:19 PM Terry Reedy > > wrote: >> >> Since Guido, the first respondent, did not immediately shoot the >> idea >> down, I intend to flesh it out and make it more concrete. >> >> >> Maybe I should have shot it down. The term is entrenched in multiple >> languages by now (e.g. >> https://en.wikipedia.org/wiki/List_comprehension). Regarding "list >> builder" one could argue that it would just add more confusion, since >> there's already an unrelated Builder Pattern >> (https://en.wikipedia.org/wiki/Builder_pattern) commonly used in >> Java. (Though I worry about the presence of a Python example in that >> Wikipedia page. :-) > > According to https://en.wikipedia.org/wiki/List_comprehension#History, > the term's known from at least 1977 and comes from such influential > languages as NPL, Miranda and Haskell. So it's not you to blame for it :-) > >> >> Also, "generator builder" is not much more expressive than "generator >> expression", > > "generator builder" is simply incorrect. The GE doesn't "build" > generators, it's a generator itself. It's a generator _and_ an > expression. What could be a more obvious name? > This suggestion looks like coming from someone who hasn't quite > grasped generators yet. > >> and the key observation that led to this idea was that it's such a >> mouthful to say "comprehensions and generator expressions". > > Since "X comprehensions" are advertised as and intended to be > functionally equivalent to `X(generator expression)', I use just > "generator expressions" to refer to all. > That's accurate because the common part with the distinctive syntax -- > which is the thing referred to when addressing them all -- effectively > _is_ a generator expression (the syntax differences in the leading > term are insignificant), what wraps it is of no concern. > > So, no new terms are necessary, but someone who cares may add a note > to the docs to this effect. > >> Maybe it's not too late to start calling the latter "generator >> comprehensions" so that maybe by the year 2025 we can say >> "comprehensions" and everyone will understand we mean all four types? >> https://docs.python.org/3/reference/expressions.html?highlight=comprehension#displays-for-lists-sets-and-dictionaries Oh, I see. So, "comprehension" is actually the official term for this "distinctive syntax", and the fact that "generator expressions" came to use it is but a coincidence. In that case, we can do a Solomon's decision: mention _both_ that "comprehension" is the official term for the syntax in GE's reference entry, _and_ the fact that "X comprehensions" are effectively wrapped GEs in their reference entries. Then everyone will learn both terminologies and could choose which is more convenient to use. >> FWIW more people should start using "list display" etc. for things >> like [a, b, c]. >> -- >> --Guido van Rossum (python.org/~guido ) >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe:https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -------------- next part -------------- An HTML attachment was scrubbed... URL: From nad at python.org Fri Jul 6 20:25:14 2018 From: nad at python.org (Ned Deily) Date: Fri, 6 Jul 2018 20:25:14 -0400 Subject: [Python-Dev] Removal of install_misc command from distutils In-Reply-To: References: Message-ID: <604D49F7-EDC0-4653-80A0-B76C3FCB0D7F@python.org> On Jul 6, 2018, at 19:43, Alexander Belopolsky wrote: > Honestly, I did not realize that 3.7 has been released by the time I wrote my e-mail. So I guess I didn't send out enough notices since January about each of the 5 betas and the release candidate begging everyone to test with the 3.7 pre-releases to find and discuss potentially avoidable incompatibilities? :) > I think it will be prudent to get this command back in Python 3.7.1. My work-around was to simply copy the 20-something lines that define install_misc from Python 3.6 to my setup.py file. > > It was my impression that it is precisely due to situations like this, distutils was considered more or less frozen for development and all new features went to setuptools. If you want to pursue it, I think the best thing to do would be to bring up the issue on the distutils-sig mailing list where the change probably should have been discussed first if it wasn't and also reopen https://bugs.python.org/issue29218. AFAIK, the removal hasn't come up as a problem before in the nearly 18 months since the change was first merged into the feature branch. -- Ned Deily nad at python.org -- [] From greg.ewing at canterbury.ac.nz Fri Jul 6 21:03:06 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 07 Jul 2018 13:03:06 +1200 Subject: [Python-Dev] Symmetric vs asymmetric symbols (was PEP 572: Do we really need a ":" in ":="?) In-Reply-To: References: Message-ID: <5B40114A.5090802@canterbury.ac.nz> Ivan Pozdeev via Python-Dev wrote: > (while "<>" reads "less or greater" which is mathematically not > equivalent to that: not everything has a defined ordering relation. I think this is a silly argument against "<>". If we're going to try to assign meaning to individual characters in an operator, we could equally well say that "!" on its own should mean "not", which it doesn't in Python. Whereas "~" does, kind of, so "not equal" should be spelled "~=". > "<>" > draws from BASIC AFAIK which was geared towards regular users who don't > deal with advanced mathematics.) Criticising something because it comes from BASIC smacks of snobbery. Anyway, it's also used by a number of entirely respectable languages such as Pascal and SQL. And what proportion of Python users deal with advanced mathematics? -- Greg From steve at pearwood.info Fri Jul 6 21:37:59 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 7 Jul 2018 11:37:59 +1000 Subject: [Python-Dev] Symmetric vs asymmetric symbols (was PEP 572: Do we really need a ":" in ":="?) In-Reply-To: <5B40114A.5090802@canterbury.ac.nz> References: <5B40114A.5090802@canterbury.ac.nz> Message-ID: <20180707013758.GJ7318@ando.pearwood.info> On Sat, Jul 07, 2018 at 01:03:06PM +1200, Greg Ewing wrote: > Ivan Pozdeev via Python-Dev wrote: > >(while "<>" reads "less or greater" which is mathematically not > >equivalent to that: not everything has a defined ordering relation. > > I think this is a silly argument against "<>". While I agree with your conclusions, I'd just like to point out that given the existence of float NANs, there's a case to be made for having separate <> and != operators with != keeping the "not equal" meaning and the <> operator meaning literally "less than, or greater than". py> NAN != 23 True py> NAN < 23 or NAN > 23 False (I'm not making the case for this, just pointing out that it exists...) There would be precedent too: at least one of Apple's SANE maths libraries back in the 1990s had a full set of NAN-aware comparison operators including IIRC separate "not equal" and "less than or greater than" comparisons. But I think this is a corner of IEEE-754 esoterica that probably doesn't need to be a builtin operator :-) Also: py> from __future__ import barry_as_FLUFL py> 23 <> 42 True -- Steve From steve at pearwood.info Fri Jul 6 22:13:45 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 7 Jul 2018 12:13:45 +1000 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> References: <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> Message-ID: <20180707021342.GK7318@ando.pearwood.info> On Sat, Jul 07, 2018 at 02:58:23AM +0300, Ivan Pozdeev via Python-Dev wrote: > >Also, "generator builder" is not much more expressive than "generator > >expression", I agree with Guido on that comment. The only advantage (such little as it is) is that we can refer to them all using the same terminology: [list | set | dict | generator] builder syntax but given how prevalent the comprehension terminology has become, maybe the best we can hope for is to start using "generator comprehension". > "generator builder" is simply incorrect. The GE doesn't "build" > generators, it's a generator itself. Nobody suggested that it was an thing that you call to build a generator. The name refers to the syntax, not the object. I did refer to it as *generator builder syntax* in my earlier post, and explicitly noted that "list/set/dict/generator builder" was the abbreviated form. But as Guido says, the possible confusion with the Builder design pattern makes this terminology confusing. If we were back in Python 2.2 days when neither the feature nor the terminology "comprehension" were so well established, perhaps we could have gone with "builder" instead, but I think that ship has sailed. > It's a generator _and_ an > expression. What could be a more obvious name? It's not about the obviousness, it is about it being a mouthful to say "comprehension or generator expression" to represent something which is conceptually a single kind of thing. We can use "comprehension" to group "list comprehension or dict comprehension or dict comprehension", but generator expressions are the odd one out. > This suggestion looks like coming from someone who hasn't quite grasped > generators yet. I assure you that both Greg and I understand generators quite well. -- Steve From tim.peters at gmail.com Fri Jul 6 22:42:24 2018 From: tim.peters at gmail.com (Tim Peters) Date: Fri, 6 Jul 2018 21:42:24 -0500 Subject: [Python-Dev] Symmetric vs asymmetric symbols (was PEP 572: Do we really need a ":" in ":="?) In-Reply-To: <20180707013758.GJ7318@ando.pearwood.info> References: <5B40114A.5090802@canterbury.ac.nz> <20180707013758.GJ7318@ando.pearwood.info> Message-ID: [Steven D'Aprano] > .... I'd just like to point out that > given the existence of float NANs, there's a case to be made for having > separate <> and != operators with != keeping the "not equal" meaning and > the <> operator meaning literally "less than, or greater than". > > py> NAN != 23 > True > py> NAN < 23 or NAN > 23 > False > > (I'm not making the case for this, just pointing out that it exists...) > > There would be precedent too: at least one of Apple's SANE maths > libraries back in the 1990s had a full set of NAN-aware comparison > operators including IIRC separate "not equal" and "less than or greater > than" comparisons. > > But I think this is a corner of IEEE-754 esoterica that probably doesn't > need to be a builtin operator :-) > > The 754 standard's section 5.7 (Comparisons) defines 26(!) distinct comparison predicates. I bet SANE supplied all of them - and quite possibly nothing else in the world ever bothered (the standard _required_ only a relative few of them). I never had the slightest interest in garbaging-up Python with syntax for all those, so never even mentioned it in the early days. My secret plan ;-) was that if someone agitated for it enough to sway Guido, I'd add a math.ieee_compare(x, y, raise=False) function that returned one of the four bit constants IEEE_(LESS, EQUAL, GREATER, UNORDERED} (with values 1, 2, 4, 8), and raised some spelling of "invalid operation" iff `raise` was True and at least one of the comparands was a NaN. That's enough to build any of the standard's predicates (and a few more essentially useless ones, like "always true"). Then, e.g., for your <> above def "<>"(x, y): return ieee_compare(x, y) & (IEEE_LESS | IEEE_GREATER) != 0 and != above would add IEEE_UNORDERED to the bits checked in that. Then it's equal easy to build oddballs like "unordered or greater" and "only equal but raise an exception if a NaN is compared" too. I've been quite relieved that, after all these years, nobody else seemed to care about this either :-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Fri Jul 6 22:52:53 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Sat, 7 Jul 2018 05:52:53 +0300 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: <48547fa0-49a3-de25-8c20-a1e109271f95@mail.mipt.ru> References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> <48547fa0-49a3-de25-8c20-a1e109271f95@mail.mipt.ru> Message-ID: <2538687a-5c82-b2ad-4ba6-6e25c7042284@mail.mipt.ru> https://github.com/python/cpython/pull/8145 On 07.07.2018 3:33, Ivan Pozdeev via Python-Dev wrote: > On 07.07.2018 2:58, Ivan Pozdeev via Python-Dev wrote: >> On 07.07.2018 2:31, Guido van Rossum wrote: >>> On Fri, Jul 6, 2018 at 4:19 PM Terry Reedy >> > wrote: >>> >>> Since Guido, the first respondent, did not immediately shoot the >>> idea >>> down, I intend to flesh it out and make it more concrete. >>> >>> >>> Maybe I should have shot it down. The term is entrenched in multiple >>> languages by now (e.g. >>> https://en.wikipedia.org/wiki/List_comprehension). Regarding "list >>> builder" one could argue that it would just add more confusion, >>> since there's already an unrelated Builder Pattern >>> (https://en.wikipedia.org/wiki/Builder_pattern) commonly used in >>> Java. (Though I worry about the presence of a Python example in that >>> Wikipedia page. :-) >> >> According to >> https://en.wikipedia.org/wiki/List_comprehension#History, the term's >> known from at least 1977 and comes from such influential languages as >> NPL, Miranda and Haskell. So it's not you to blame for it :-) >> >>> >>> Also, "generator builder" is not much more expressive than >>> "generator expression", >> >> "generator builder" is simply incorrect. The GE doesn't "build" >> generators, it's a generator itself. It's a generator _and_ an >> expression. What could be a more obvious name? >> This suggestion looks like coming from someone who hasn't quite >> grasped generators yet. >> >>> and the key observation that led to this idea was that it's such a >>> mouthful to say "comprehensions and generator expressions". >> >> Since "X comprehensions" are advertised as and intended to be >> functionally equivalent to `X(generator expression)', I use just >> "generator expressions" to refer to all. >> That's accurate because the common part with the distinctive syntax >> -- which is the thing referred to when addressing them all -- >> effectively _is_ a generator expression (the syntax differences in >> the leading term are insignificant), what wraps it is of no concern. >> >> So, no new terms are necessary, but someone who cares may add a note >> to the docs to this effect. >> > >>> Maybe it's not too late to start calling the latter "generator >>> comprehensions" so that maybe by the year 2025 we can say >>> "comprehensions" and everyone will understand we mean all four types? >>> > https://docs.python.org/3/reference/expressions.html?highlight=comprehension#displays-for-lists-sets-and-dictionaries > > Oh, I see. So, "comprehension" is actually the official term for this > "distinctive syntax", and the fact that "generator expressions" came > to use it is but a coincidence. > > In that case, we can do a Solomon's decision: mention _both_ that > "comprehension" is the official term for the syntax in GE's reference > entry, _and_ the fact that "X comprehensions" are effectively wrapped > GEs in their reference entries. > > Then everyone will learn both terminologies and could choose which is > more convenient to use. > >>> FWIW more people should start using "list display" etc. for things >>> like [a, b, c]. >>> -- >>> --Guido van Rossum (python.org/~guido ) >>> >>> >>> _______________________________________________ >>> Python-Dev mailing list >>> Python-Dev at python.org >>> https://mail.python.org/mailman/listinfo/python-dev >>> Unsubscribe:https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe:https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Jul 7 00:01:23 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 7 Jul 2018 00:01:23 -0400 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> Message-ID: In response to Guido's reply to my post fleshing out my idea to use 'generator|list|set|dict builder', On 7/6/2018 7:58 PM, Ivan Pozdeev via Python-Dev wrote: > According to https://en.wikipedia.org/wiki/List_comprehension#History, > the term's known from at least 1977 and comes from such influential > languages as NPL, Miranda and Haskell. So it's not you to blame for it :-) "A list comprehension is a syntactic construct available in some programming languages for creating a list based on existing lists. It follows the form of the mathematical set-builder notation (set comprehension) as distinct from the use of map and filter functions." Mathematicians do not always agree on terminology and notation. I believe that 'set builder notatation' is both older and was and perhaps is more widespread than 'set comprehension'. I have read that it is at least a century old. But https://en.wikipedia.org/wiki/Set-builder_notation does not seem to be the place In any case, Python's comprehensions use an English-syntax version of extended set builder notation. "In Python, the set-builder's braces are replaced with square brackets, parentheses, or curly braces, giving list, generator, and set objects, respectively. Python uses an English-based syntax." >> Also, "generator builder" is not much more expressive than "generator >> expression", I looked for an alternative 'x' to 'comprehension' such that 'generator|list|set|dict x' works and is specific to the notation. 'Builder' is a reasonable choice. 'expression' is way too general. A 'list expression', for instance, is any expression that evaluated to a list. In this context, I consider that the specific term 'says more' than the general term. On the face of it, a generator expression is an expression that evaluates to a generator. In this sense, 'f(args)', where f is a generator function, is a generator expression. In any case, 'generator comprehension' is an awkward 8 syllable mouthful. -- Terry Jan Reedy From songofacandy at gmail.com Sat Jul 7 00:10:59 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 7 Jul 2018 13:10:59 +0900 Subject: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods In-Reply-To: <5B2A15FE.4000608@UGent.be> References: <5B2A15FE.4000608@UGent.be> Message-ID: How often "custom method type" are used? I thought Cython use it by default. But when I read code generated by Cython, I can't find it. It uses normal PyMethodDef and tp_methods. I found CyFunction in Cython repository, but I can't find how to use it. Cython document doesn't explain any information about it. When, and how often custom method type is used? Isn't it very rare? If there are only 0.1% custom method type, why reducing 30% calling overhead is important for them? I want more possible target applications to motivate me for such complex protocols. -- INADA Naoki From tjreedy at udel.edu Sat Jul 7 01:19:33 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 7 Jul 2018 01:19:33 -0400 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: On 7/6/2018 7:31 PM, Guido van Rossum wrote: > On Fri, Jul 6, 2018 at 4:19 PM Terry Reedy > wrote: > > Since Guido, the first respondent, did not immediately shoot the idea > down, I intend to flesh it out and make it more concrete. > > Maybe I should have shot it down. I glad you did not do so immediately since some of what I worked out since applies to the alternative of consistently using 'comprehension'. > The term is entrenched in multiple > languages by now (e.g. > https://en.wikipedia.org/wiki/List_comprehension). Regarding "list > builder" one could argue that it would just add more confusion, since > there's already an unrelated Builder Pattern I was not aware of that. I read enough to see that as a relevant conflict. > (https://en.wikipedia.org/wiki/Builder_pattern) commonly used in Java. > (Though I worry about the presence of a Python example in that Wikipedia > page. :-) > > Also, "generator builder" is not much more expressive than "generator > expression", If one views 'generator expression' as a 2-word phrase, as opposed to a defined compound word, it could mean either 'expression that contains a generator' or 'expression that evaluates to a generator. With the latter meaning, 'generator_func(*args)' is a generator expression. I intended 'generator builder' to be more clearly delimited. So is 'generator comprehension'. > and the key observation that led to this idea was that it's > such a mouthful to say "comprehensions and generator expressions". That was part of my motivation also. > Maybe it's not too late > to start calling the latter "generator comprehensions" Having proposed a more drastic change, I obviously think it is not too late to change the doc at least for 3.8. (If we do it now, I would consider 3.7 also.) Rename the Expressions section to just 'Comprehensions'. Define 'comprehension' perhaps as "an expression that defines an iterable using Python's adaptation and generalization of extended set builder notation". Comprehensions have to be fenced for use (except for gencomps in function calls) to determine the concrete type of iterable. The key:value syntax that separates dict from set displays separates dict from set comprehensions. Otherwise: Change to 'generator comprehension'. Do whatever to the doc grammar. Adjust glossary entries. If allowed in our reference format, perhaps link to Wikipedia articles on 'set builder notation' and 'list comprehension'. The 8 syllables of 'generator comprehension' is bit long for a compound word. Python uses '' as the pseudo-name for generators. Some people use 'genexp' as an abbreviation (do they pronounce the 'p'?), along with listcomp. 'Gencomp' should do as well. > so that maybe by the year 2025 we can say "comprehensions" and everyone > will understand we mean all four types? > > FWIW more people should start using "list display" etc. for things like > [a, b, c]. Definitely. -- Terry Jan Reedy From stefan_ml at behnel.de Sat Jul 7 03:19:11 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 7 Jul 2018 09:19:11 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: <5B3E30D1.3080706@UGent.be> References: <5B3E30D1.3080706@UGent.be> Message-ID: Jeroen Demeyer schrieb am 05.07.2018 um 16:53: > The only case when this handling of keywords is suboptimal is when using > **kwargs. In that case, a dict must be converted to a tuple. It looks hard > to me to support efficiently both the case of fixed keyword arguments > (f(foo=x)) and a keyword dict (f(**kwargs)). Since the former is more > common than the latter, the current choice is optimal. Wrappers that adapt or add some arguments (think partial()) aren't all that uncommon, even when speed is not irrelevant. But I agree that actual keyword arguments should rarely be involved in those calls. Typically, it's calls with 1 to ~3 positional arguments that occur in performance critical situations. Often just one, rarely more, and zero arguments is a fast case anyway. Keyword arguments will always suffer some kind of penalty compared to positional arguments, regardless of how they are implemented (at runtime). But they can easily be avoided in many cases, and anyone designing a performance relevant API that *requires* keyword arguments deserves to have their code forked away from them. :) The current keyword calling conventions seem fine with me and I do not see a reason why we should invest discussion time and distributed brain capacity into "improving" them. Stefan PS: Passing keyword arguments through wrappers unchanged might be a case to consider in the future, but the calling PEPs don't seem the right place to discuss those, as it's not just a call issue but also a compiler issue. From stefan_ml at behnel.de Sat Jul 7 03:32:13 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 7 Jul 2018 09:32:13 +0200 Subject: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods In-Reply-To: References: <5B2A15FE.4000608@UGent.be> Message-ID: INADA Naoki schrieb am 07.07.2018 um 06:10: > How often "custom method type" are used? > > I thought Cython use it by default. > But when I read code generated by Cython, I can't find it. > It uses normal PyMethodDef and tp_methods. > > I found CyFunction in Cython repository, but I can't find > how to use it. Cython document doesn't explain any information > about it. Its usage is disabled by default because of some of the problems that Jeroen addresses in his PEP(s). You can enable Cython's own function type by setting the compiler directive "binding=True", e.g. from your setup.py or in a comment at the very top of your source file: # cython: binding=True The directive name "binding" stems from the fact that CyFunctions bind as methods when put into classes, but it's really misleading these days because the main advantage is that it makes Cython compiled functions look and behave much more like Python functions, including introspection etc. Stefan From songofacandy at gmail.com Sat Jul 7 04:08:21 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 7 Jul 2018 17:08:21 +0900 Subject: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods In-Reply-To: References: <5B2A15FE.4000608@UGent.be> Message-ID: On Sat, Jul 7, 2018 at 4:35 PM Stefan Behnel wrote: > > INADA Naoki schrieb am 07.07.2018 um 06:10: > > How often "custom method type" are used? > > > > I thought Cython use it by default. > > But when I read code generated by Cython, I can't find it. > > It uses normal PyMethodDef and tp_methods. > > > > I found CyFunction in Cython repository, but I can't find > > how to use it. Cython document doesn't explain any information > > about it. > > Its usage is disabled by default because of some of the problems that > Jeroen addresses in his PEP(s). > > You can enable Cython's own function type by setting the compiler directive > "binding=True", e.g. from your setup.py or in a comment at the very top of > your source file: > > # cython: binding=True > > The directive name "binding" stems from the fact that CyFunctions bind as > methods when put into classes, but it's really misleading these days > because the main advantage is that it makes Cython compiled functions look > and behave much more like Python functions, including introspection etc. > > Stefan > Thank you. Do you plan to make it default when PEP 580 is accepted and implemented? Personally speaking, I used Cython for quick & easy alternative way to writing extension types. I don't need compatibility with pure Python functions. I prefer minimum and lightweight. So I will disable it explicitly or stop using Cython. But if you believe PEP 580 makes many Cython users happy, I believe you. -- INADA Naoki From v+python at g.nevcal.com Sat Jul 7 04:10:27 2018 From: v+python at g.nevcal.com (Glenn Linderman) Date: Sat, 7 Jul 2018 01:10:27 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> <808f6d74-2899-a19f-54ac-167e88f3ba62@mail.mipt.ru> Message-ID: <79232460-d306-68d4-99ad-6bcc3c966b94@g.nevcal.com> On 7/6/2018 9:01 PM, Terry Reedy wrote: > In any case, Python's comprehensions use an English-syntax version of > extended set builder notation.? "In Python, the set-builder's braces > are replaced with square brackets, parentheses, or curly braces, > giving list, generator, and set objects, respectively. Python uses an > English-based syntax." > >>> Also, "generator builder" is not much more expressive than >>> "generator expression", > > I looked for an alternative 'x' to 'comprehension' such that > 'generator|list|set|dict x' works and is specific to the notation. > 'Builder' is a reasonable choice. I'm not sure if your quote above was quoting documentation, or was a suggested quote to add to the documentation, I think the latter, as Google didn't find it. The conflict between the "Builder pattern" and "set-builder notation" can be disambiguated by consistently using the hyphenated "set-builder" (as wikipedia does). And happily, by using wikipedia terms, they would be easily found with explanations outside of python docs as well as (if this is done) inside.? We do not need [ typ + ' builder' for typ in ('set', 'list', 'dict', 'generator')] only set-builder.? The fencing and : determine the type of the result.? We could use [ typ + ' form of set-builder'? for typ in ('set', 'list', 'dict', 'generator')] in the few places where the type of the set-builder must be disambiguated, avoiding the need for the compound terms. The result of? ( set-builder ) is a generator. We do not need the term "generator expression" or "generator comprehension".? Use "generator form of set-builder"... yes, it is one or two syllables longer, but is clearer. A generator can be produced in one of two ways: either a function containing a yield, or a set-builder delimited by parentheses or used as an actual parameter to a function, both of which can be referred to as the "generator form of set-builder". Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Jul 7 04:45:34 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 7 Jul 2018 10:45:34 +0200 Subject: [Python-Dev] PEP 579 and PEP 580: refactoring C functions and methods In-Reply-To: References: <5B2A15FE.4000608@UGent.be> Message-ID: INADA Naoki schrieb am 07.07.2018 um 10:08: > Thank you. Do you plan to make it default when PEP 580 is accepted > and implemented? It will become the default at some point, I'm sure. Note that we will still have to support older Python versions, though, currently 2.6+, which would not have the improvements available. Some things might be backportable for us, at least to older Py3.x releases, but we'll see. > Personally speaking, I used Cython for quick & easy alternative way to > writing extension types. > I don't need compatibility with pure Python functions. I prefer > minimum and lightweight. > So I will disable it explicitly or stop using Cython. I'll try to keep the directive available as a compatibility switch for you. ;) > But if you believe PEP 580 makes many Cython users happy, I believe you. It's more of a transitive thing, for the users of your code. If the functions and methods in the code that I write behave like Python functions, then people who use my code will not run into surprises and quirks when trying to do their stuff with them and things will generally "just work", e.g. inspecting the functions when debugging or using them interactively, looking up their signature, default arguments and annotations, generating documentation from them, pickling, assigning them as methods ... basically anything that people implicitly expect to be able to do with Python functions (or even callables in general) and that doesn't (not well or not at all) work with PyCFunction. Stefan From storchaka at gmail.com Sat Jul 7 04:55:22 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sat, 7 Jul 2018 11:55:22 +0300 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: <5B3E30D1.3080706@UGent.be> References: <5B3E30D1.3080706@UGent.be> Message-ID: 05.07.18 17:53, Jeroen Demeyer ????: > In other words: I see nothing to improve in the calling convention of > METH_FASTCALL. I suggest to keep it and make it public as-is. You have considered the bytecode for calling functions, but this actually is not directly related to the calling convention. All opcodes (CALL_FUNCTION, CALL_FUNCTION_KW, CALL_FUNCTION_EX and CALL_METHOD) can be used for calling any callable. CALL_FUNCTION, CALL_FUNCTION_KW was designed for reducing the overhead from the caller side for most common cases, and CALL_FUNCTION_EX is used for the rest. Calling conventions METH_FASTCALL and METH_FASTCALL|METH_KEYWORDS was designed for reducing the overhead of creating a temporary tuple and dict. They allow to avoid allocation and copying in case of using opcodes CALL_FUNCTION, CALL_FUNCTION_KW and in most cases of using C API for calling function. But it is not the only possible way, there some details can be changed without performance loss, and may be even with a gain. After passing positional and keyword arguments to the C function, we need to convert Python objects to C values. For positional-only parameters we can use PyArg_ParseTuple() (for METH_VARARGS) and _PyArg_ParseStack() (for METH_FASTCALL). If there are keyword parameters, we need to use more complicated API: PyArg_ParseTupleAndKeywords() or private _PyArg_ParseTupleAndKeywordsFast() (for METH_VARARGS|METH_KEYWORDS) and private _PyArg_ParseStackAndKeywords() (for METH_FASTCALL|METH_KEYWORDS). _PyArg_ParseTupleAndKeywordsFast() and _PyArg_ParseStackAndKeywords() are private, complex, unstable and can be used only internally in CPython (mostly in the code generated by Argument Clinic). They have complex signatures and their code is mostly duplicate one other (but with some important differences). There is a wish of inlining argument parsing functions in the Argument Clinic generated code. It is easier to do for PyArg_ParseTuple() and _PyArg_UnpackStack(). But functions for parsing keyword arguments are more complex, because they perform two things: match keyword argument names to parameter positions and convert argument values to C. There is my idea. Split every of keyword argument parsing functions on two parts. The first part linearize keyword arguments, it converts positional and keyword arguments (in whatever form they were presented) into a linear array of PyObject* (with NULLs for not specified optional arguments). The second part is common and similar to _PyArg_ParseStack(), but supports NULLs. It converts an array of PyObject* to a sequence of C values. I tried to implement this idea, is is not simple, and results were mixed, but I don't loss a hope. And here we return to METH_FASTCALL|METH_KEYWORDS. The first part of handling arguments can be made outside of the C function, by the calling API. Then the signature of the C function can be simpler, the same as for METH_FASTCALL. But we need to expose the list of keyword parameter names as an attribute of CFunction. I don't know whether this ides is vital or dead, but I' going to try it. And implementing it will change the METH_FASTCALL|METH_KEYWORDS calling convention a lot. From p.f.moore at gmail.com Sat Jul 7 05:51:20 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 7 Jul 2018 10:51:20 +0100 Subject: [Python-Dev] Removal of install_misc command from distutils In-Reply-To: <604D49F7-EDC0-4653-80A0-B76C3FCB0D7F@python.org> References: <604D49F7-EDC0-4653-80A0-B76C3FCB0D7F@python.org> Message-ID: On 7 July 2018 at 01:25, Ned Deily wrote: > On Jul 6, 2018, at 19:43, Alexander Belopolsky wrote: >> I think it will be prudent to get this command back in Python 3.7.1. My work-around was to simply copy the 20-something lines that define install_misc from Python 3.6 to my setup.py file. You'll still need those 20 lines in your code if you want to support Python 3.7.0, and *not* supporting 3.7.0 but supporting 3.7.1 doesn't seem like a good idea to me. TBH, I'd consider "copy the 20 lines of code into your own setup.py to be a perfectly acceptable workaround. >> It was my impression that it is precisely due to situations like this, distutils was considered more or less frozen for development and all new features went to setuptools. For a long while distutils was frozen, and yes it was basically because of a fear that nothing could be changed without potentially disrupting someone. But that resulted in essentially a paralysis of work on packaging for many years, and wasn't a good thing. The embargo on changes to distutils was lifted a long time ago - although we remain cautious about making changes because we don't want to break people's code. As Ned noted, you had the whole Python 3.7 alpha and beta phase to test your code, and if you had raised the issue then, we could have considered reverting this change (but my view would still have been "we'll remove it and you should just copy the code"). A quick search of bpo [1] shows many changes to distutils in recent time, FWIW. > If you want to pursue it, I think the best thing to do would be to bring up the issue on the distutils-sig mailing list where the change probably should have been discussed first if it wasn't and also reopen https://bugs.python.org/issue29218. AFAIK, the removal hasn't come up as a problem before in the nearly 18 months since the change was first merged into the feature branch. IMO, a tracker issue was fine, and the OP could have commented there. Distutils-sig isn't really the right audience for minor detail changes like this - although if someone wanted to raise the broader principle "let's reinstate the total freeze on distutils changes" that would be appropriate for distutils-sig. I'd fully expect a resounding rejection for that proposition, though. At this point, I think the only realistic resolution would be to add a note to the "Porting to Python 3.7" docs explaining that users of install_misc should copy the code from 3.6 into their setup.py. But it's quite likely that the only person who needs that documentation is Alexander (we've not had any other reports of this issue to my knowledge) so that may be overkill... Paul [1] https://bugs.python.org/issue?%40search_text=&ignore=file%3Acontent&title=&%40columns=title&id=&%40columns=id&stage=6&creation=&creator=&activity=&%40columns=activity&%40sort=activity&actor=&nosy=&type=&components=3&versions=&dependencies=&assignee=&keywords=&priority=&status=2&%40columns=status&resolution=3&nosy_count=&message_count=&%40group=&%40pagesize=50&%40startwith=0&%40sortdir=on&%40action=search From njs at pobox.com Sat Jul 7 06:22:24 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 7 Jul 2018 03:22:24 -0700 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: References: <5B3E30D1.3080706@UGent.be> Message-ID: On Sat, Jul 7, 2018 at 12:19 AM, Stefan Behnel wrote: > Typically, it's calls with 1 to ~3 positional arguments that occur in > performance critical situations. Often just one, rarely more, and zero > arguments is a fast case anyway. Keyword arguments will always suffer some > kind of penalty compared to positional arguments, regardless of how they > are implemented (at runtime). But they can easily be avoided in many cases, > and anyone designing a performance relevant API that *requires* keyword > arguments deserves to have their code forked away from them. :) In numpy we do sometimes hesitate to use kwargs, even when we otherwise ought to, because of the slowdown they incur. You're right that there's always going to be some overhead here, and I can't say how important these kinds of optimizations are compared to other things people could be spending their time on. But kwargs do improve readability, and it's nice when we can make readable code fast, so people aren't tempted to obfuscate things in the name of speed. -n -- Nathaniel J. Smith -- https://vorpus.org From stefan_ml at behnel.de Sat Jul 7 07:15:37 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 7 Jul 2018 13:15:37 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: References: <5B3E30D1.3080706@UGent.be> Message-ID: Hi Serhiy! Serhiy Storchaka schrieb am 07.07.2018 um 10:55: > There is my idea. Split every of keyword argument parsing functions on two > parts. The first part linearize keyword arguments, it converts positional > and keyword arguments (in whatever form they were presented) into a linear > array of PyObject* (with NULLs for not specified optional arguments). The > second part is common and similar to _PyArg_ParseStack(), but supports > NULLs. It converts an array of PyObject* to a sequence of C values. I tried > to implement this idea, is is not simple, and results were mixed, but I > don't loss a hope. That proposal sounds good to me. Cython currently does something similar /inside/ of its function entry code, in that it executes an unrolled series of PyDict_GetItem() calls for the expected keyword arguments (instead of iterating over a dict, which turned out to be slower), and maps those to an array of arguments, all before it passes over that array to convert the values to the expected C types. I agree that it makes sense to do the name matching outside of the callee since the caller knows best in what way (sequence, dict, ...) the arguments are available and can decide on the fastest way to map them to a flat array, given the expected argument names. And I think the proposal would fit nicely into PEP-580. > And here we return to METH_FASTCALL|METH_KEYWORDS. The first part of > handling arguments can be made outside of the C function, by the calling > API. Then the signature of the C function can be simpler, the same as for > METH_FASTCALL. But we need to expose the list of keyword parameter names as > an attribute of CFunction. And names should be expected to be interned, so that matching the keywords can be done via pointer comparisons in almost all cases. That should make it pretty fast, and errors can be detected in a slow separate pass if the pointer matching fails. I think we cannot strictly assume a predictable, helpful ordering of the keyword arguments on the calling side (that would allow for a one-pass merge), but it's rather common for users to pass keyword arguments in the order in which the signature expects them, so I'm sure there's a fast algorithm (e.g. something like Insertion Sort) to match both sides in negligible time. Stefan From mark at hotpy.org Sat Jul 7 08:54:05 2018 From: mark at hotpy.org (Mark Shannon) Date: Sat, 7 Jul 2018 13:54:05 +0100 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <5B3FF511.7060205@UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3E341E.70402@UGent.be> <1cb5933b3ce7418d90aa8b9488efc8e7@xmail103.UGent.be> <5B3F3B4E.6080508@UGent.be> <5B3FF511.7060205@UGent.be> Message-ID: On 07/07/18 00:02, Jeroen Demeyer wrote: > On 2018-07-06 23:12, Guido van Rossum wrote: >> It's your PEP. And you seem to be struggling with something. But I can't >> tell quite what it is you're struggling with. > > To be perfectly honest (no hard feelings though!): what I'm struggling > with is getting feedback (either positive or negative) from core devs > about the actual PEP 580. > >> At the same time I assume you want your PEP accepted. > > As I also said during the PEP 575 discussion, my real goal is to solve a > concrete problem, not to push my personal PEP. I still think that PEP > 580 is the best solution but I welcome other suggestions. > >> And how do they feel about PEP 576? I'd like to see some actual debate >> of the pros and cons of the details of PEP 576 vs. PEP 580. > > I started this thread to do precisely that. > > My opinion: PEP 580 has zero performance cost, while PEP 576 does make > performance for bound methods worse (there is no reference > implementation of the new PEP 576 yet, so that's hard to quantify for There is a minimal implementation and has been for a while. There is a link at the bottom of the PEP. Why do you claim it will make the performance of bound methods worse? You provide no evidence of that claim. > now). PEP 580 is also more future-proof: it defines a new protocol which > can easily be extended in the future. PEP 576 just builds on PyMethodDef > which cannot be extended because of ABI compatibility (putting > __text_signature__ and __doc__ in the same C string is a good symptom of > that). This extensibility is important because I want PEP 580 to be the > first in a series of PEPs working out this new protocol. See PEP 579 for > the bigger picture. PEP 576 adds a new calling convention which can be used by *any* object. Seems quite extensible to me. > > One thing that might count against PEP 580 is that it defines a whole > new protocol, which could be seen as too complicated. However, it must > be this complicated because it is meant to generalize the current > behavior and optimizations of built-in functions and methods. There are > lots of little tricks currently in CPython that must be "ported" to the > new protocol. > >> OK, so is it your claim that the NumPy developers don't care about which >> one of these PEPs is accepted or even whether one is accepted at all? > > I don't know, I haven't contacted any NumPy devs yet, so that was just > my personal feeling. These PEPs are about optimizing callables and NumPy > isn't really about callables. I think that the audience for PEP 580 is > mostly compilers (Cython for sure but possibly also Pythran, numba, > cppyy, ...). Also certain C classes like functools.lru_cache could > benefit from it. > >> Yet earlier in >> *this* thread you seemed to claim that PEP 580 requires changes ro >> FASTCALL. > > I don't know what you mean with that. But maybe it's also confusing > because "FASTCALL" can mean different things: it can refer to a > PyMethodDef (used by builtin_function_or_method and method_descriptor) > with the METH_FASTCALL flag set. It can also refer to a more general API > like _PyCFunction_FastCallKeywords, which supports METH_FASTCALL but > also other calling conventions like METH_VARARGS. > > I don't think that METH_FASTCALL should be changed (and PEP 580 isn't > really about that at all). For the latter, I'm suggesting some API > changes but nothing fundamental: mainly replacing the 5 existing private > functions _PyCFunction_FastCallKeywords, _PyCFunction_FastCallDict, > _PyMethodDescr_FastCallKeywords, _PyMethodDef_RawFastCallKeywords, > _PyMethodDef_RawFastCallDict by 1 public function PyCCall_FASTCALL. > > > Hopefully this clears some things up, > Jeroen. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mark%40hotpy.org From g.rodola at gmail.com Sat Jul 7 09:09:54 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sat, 7 Jul 2018 15:09:54 +0200 Subject: [Python-Dev] Call for prudence about PEP-572 Message-ID: Sorry in advance for opening yet another topic about PEP-572. With PEP-572 being officially accepted I know debating its inclusion in the language is a useless exercise at this point, but since it's still in "draft" status I would like to express my opinion as I think this is a feature which can potentially be abused fairly easily. FWIW I initially found myself disliking the idea as a whole but https://github.com/python/cpython/pull/8122 made me (and others) reconsider it quite a bit (see: https://twitter.com/grodola/status/1015251302350245888). PR-8122 clearly shows an improvement in expressiveness and compactness (many folks argue this is too much) but PEP-572 as it currently stands is too permissive IMHO. My concern about "easily abusable ugly cases" still remains, and I think they should be banned instead of just discouraged in the PEP or in the doc. Since we spend way more time *reading* code rather than writing it, as a "reader" I would expect a more prudent approach to the problem. Proposal ======== 1) allow only one := assignment per line in "if" statements: >>> if x := val1 and y := val2: # SyntaxError or SyntaxWarning >>> if x == val1 and y := val2: # SyntaxError or SyntaxWarning >>> if x := val1 and y == val2: # SyntaxError or SyntaxWarning >>> if x := val1: # OK >>> if (x := val1): # OK 2) allow := in "while" statements, "if" statements and comprehensions only: >>> foo(x := 0) # SyntaxError >>> yield x := 3 # SyntaxError >>> assert y := 3 # SyntaxError 3) (debatable) disallow := if the variable is already defined: >>> x = 5 >>> if (x := val): # SyntaxError or SyntaxWarning 4) ban "a = (b := c)", "x = a := (b := (c := d))" and similar (they're just too ugly IMHO) Rationale 1 =========== In visual terms assignments in Python have always occurred at the BEGINNING of the line and always on the most LEFT side: >>> foo = fun1() >>> bar = fun2() >>> baz = fun3() That is where I naturally expect an assignment to be when reading code. My main concern with PEP-572 is that an assignments can now occur at *any point* in the line: >>> foo = fun1() >>> bar = fun2() >>> if foo == val1 and bar == val2 and baz := fun3(): ... ... That forces me to visually scan the whole line horizontally from left to right 'till its end, looking for possible variables being set. I'm concerned that I will miss := occurrences because visually they are very similar to == unless parentheses are made mandatory: >>> if foo == val1 and bar == val2 and (baz := fun3()): ... ... Also, in case of multi-line conditionals I have to visually scan the construct both horizontally AND vertically: >>> if (foo == val1 and \ ... bar == val2 and \ ... baz := val3): ... ... Again, that is not a place where I would expect to find or look for a variable assignment. I know I wouldn't like to read or review a code which does that and I suspect linters will likely end up wanting to emit a warning in that case (see: https://github.com/PyCQA/pylint/issues/2246). https://github.com/python/cpython/pull/8116/files avoids using multiple := per line and that's why the result appears readable enough IMO. Rationale 2 =========== PEP-572 states: > The := operator may be used directly in a positional function call argument That means allowing: >>> foo(x := 0) I honestly don't see why anyone would want to call a function AND assign a variable value at the same time (except in comprehensions). With this in place I not only have to guard against "if" statements assigning values at any point in the code, but also function calls, both horizontally and vertically e.g.: >>> foo(some_long_var_name, another_one, x := bar(), y := fun()) To me this looks like the perfect example of where this functionality can be abused. Also, I'm not clear what PEP-572 intend to do about "all other places". E.g. should these cases be allowed? (IMO no) >>> yield x := 3 >>> assert y := 3 -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Jul 7 09:13:47 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 7 Jul 2018 23:13:47 +1000 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Sat, Jul 7, 2018 at 11:09 PM, Giampaolo Rodola' wrote: > To me this looks like the perfect example of where this functionality can be > abused. Also, I'm not clear what PEP-572 intend to do about "all other > places". E.g. should these cases be allowed? (IMO no) > > >>> yield x := 3 > >>> assert y := 3 Special cases aren't special enough to break the rules. Can we stop trying to nerf this into complexity, please? ChrisA From ncoghlan at gmail.com Sat Jul 7 09:34:44 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 7 Jul 2018 23:34:44 +1000 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3E341E.70402@UGent.be> <1cb5933b3ce7418d90aa8b9488efc8e7@xmail103.UGent.be> <5B3F3B4E.6080508@UGent.be> Message-ID: On 7 July 2018 at 07:12, Guido van Rossum wrote: > On Fri, Jul 6, 2018 at 2:52 AM Jeroen Demeyer wrote: >> The Cython developers (in particular Stefan Behnel) certainly support my >> work. I have talked with them in person at a workshop and they posted a >> few emails to python-dev and they also gave me some personal comments >> about PEP 580. > > > And how do they feel about PEP 576? I'd like to see some actual debate of > the pros and cons of the details of PEP 576 vs. PEP 580. So far I mostly see > you and INADA Naoki disagreeing about process, which doesn't give me the > feeling that there is consensus. I think part of the confusion here stems from the fact that Jeroen's original PEP 575 tried to cover a lot of different things, with two of the most notable being: 1. Getting functions implemented in C to act more like their Python counterparts from an introspection and capability perspective (e.g. having access to their defining module regardless of whether they're a top level function or a method on a type definition) 2. Allowing third party compilers like Cython to route function calls through the CPython C API more efficiently than the existing public APIs that require building and deconstructing Python tuples and dicts Hence the request that the PEP be split up into an overview PEP describing the problem space (now available as https://www.python.org/dev/peps/pep-0579/ ), and then follow-up PEPs targeting specific sub-topics within that PEP. That's happened for PEP 580 (since Jeroen was working on both PEPs at the same time as an initial replacement for PEP 575), but PEP 576 hasn't been updated yet to specify which of the subtopics within PEP 579 it is aiming to address My current reading is that PEP 576 isn't really targeting the same aspects as PEP 580: PEP 580 aims to allow third party callable implementations to be as fast as native CPython internal ones regardless of which callable type they use (point 2 above), while PEP 576 has the more modest aim of eliminating some of the current reasons that third parties find it necessary to avoid using the CPython native callable types in the first place (point 1 above). That said, I think Inada-san's request for benchmarks that clearly demonstrate the challenges with the status quo and hence can be used to quantify the potential benefits is a sound one, as those same benchmarks can then be used to assess the complexity of adapting existing third party tools and libraries like Cython and NumPy to implement the proposals in order to produce updated benchmarking numbers (for PEP 580, add code to implement the new protocol method, for PEP 576, switch to inheriting from one of the two newly defined C level types). At a micro-benchmark level, that would probably involve just comparing mapping builtin functions and methods over a rangewith the performance of variants of those functions implemented using only the public C API (the specific functions and methods chosen for the benchmark will need to be those where the optimisations for particularly simple function signatures don't apply). At a macro-benchmark level, it would likely require first choosing or defining a language level computational performance benchmark that's based on real world code using libraries like NumPy/pandas/scikit-learn, akin to the domain specific benchmarks we already have for libraries like SQL Alchemy, dulwich, and various templating engines (django, mako, genshi). One possibility that may make sense could be to set up comparisons of https://github.com/numpy/numpy/tree/master/benchmarks numbers between a conventional NumPy and one with more optimised C level calls, and doing something similar for https://pandas.pydata.org/pandas-docs/stable/contributing.html#running-the-performance-test-suite. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mark at hotpy.org Sat Jul 7 09:38:39 2018 From: mark at hotpy.org (Mark Shannon) Date: Sat, 7 Jul 2018 14:38:39 +0100 Subject: [Python-Dev] PEP 575, 576, 579 and 580 Message-ID: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Hi, We seem to have a plethora of PEPs where we really ought to have one (or none?). Traditionally when writing a new piece of software, one gathered requirements before implementing the code. Let us return to that venerable tradition. IMO, mailing lists are a terrible way to do software design, but a good way to gather requirements as it makes less likely that someone will be forgotten. So, let us gather the requirements for a new calling API. Here are my starting suggestions: 1. The new API should be fully backwards compatible and shouldn't break the ABI 2. The new API should be used internally so that 3rd party extensions are not second class citizens in term of call performance. 3. The new API should not prevent 3rd party extensions having full introspection capabilities, supporting keyword arguments or another feature supported by Python functions. 4. The implementation should not exceed D lines of code delta and T lines of code in total size. I would suggest +200 and 1000 for D and T respectively (or is that too restrictive?). 5. It should speed up CPython for the standard benchmark suite. 6. It should be understandable. What am I missing? Comments from the maintainers of Cython and other similar tools would be appreciated. Cheers, Mark. From ncoghlan at gmail.com Sat Jul 7 10:14:13 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Jul 2018 00:14:13 +1000 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Message-ID: On 7 July 2018 at 23:38, Mark Shannon wrote: > Hi, > > We seem to have a plethora of PEPs where we really ought to have one (or > none?). > > Traditionally when writing a new piece of software, one gathered > requirements before implementing the code. Let us return to that venerable > tradition. > > IMO, mailing lists are a terrible way to do software design, but a good way > to gather requirements as it makes less likely that someone will be > forgotten. That's the purpose of PEP 579: gather the background information on the problems that folks want to solve such that the competing proposed solutions aren't defining the problem that needs to be solved in different ways. If PEP 579 isn't working as a problem specification from your perspective, then I'd suggest posting a PR that Jeroen could review (although I think this thread is a good idea as well). > So, let us gather the requirements for a new calling API. > Here are my starting suggestions: > > 1. The new API should be fully backwards compatible and shouldn't break the > ABI > 2. The new API should be used internally so that 3rd party extensions are > not second class citizens in term of call performance. > 3. The new API should not prevent 3rd party extensions having full > introspection capabilities, supporting keyword arguments or another feature > supported by Python functions. > 4. The implementation should not exceed D lines of code delta and T lines of > code in total size. I would suggest +200 and 1000 for D and T respectively > (or is that too restrictive?). > 5. It should speed up CPython for the standard benchmark suite. > 6. It should be understandable. I like points 1, 2, 3, and 6, but I think point 4 should be a design trade-off rather than a requirement, since minimising the delta in CPython becomes an anti-goal if the outcome of doing so is to make the change harder to adopt for third party projects (at the same time, a delta that's too large is unlikely to be accepted, reviewed and merged, which is what makes it a trade-off). I don't think point 5 is a goal here either, as the problem isn't that these calling optimisations don't exist, it's that they don't currently have a public API that third party projects can access (the most recent METH_FASTCALL thread covers that pretty well). My own additional concern that I think is also on the debatable border between "design requirement" and "design trade-off" is whether or not it's acceptable for us to require that existing third party projects change their parent CPython type in order to access the optimised calling conventions. Changing Python base types in an extension module can end up being an annoyingly intrusive change, since it changes the memory layout in your instances. Whether or not that's a problem depends on exactly what you're doing, but when the new calling convention is tied to a protocol that any type can implement (as PEP 580 proposes), the concern doesn't even arise. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan_ml at behnel.de Sat Jul 7 10:39:08 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 7 Jul 2018 16:39:08 +0200 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Message-ID: Nick Coghlan schrieb am 07.07.2018 um 16:14: > when the new calling > convention is tied to a protocol that any type can implement (as PEP > 580 proposes), the concern doesn't even arise. Nick, +1 to all of what you said in your reply, and I also really like the fact that this proposal is creating a new, general protocol that removes lots of type special casing from places where objects are being called "efficiently". We're essentially designing a Fast Duck Calling convention here. Stefan From steve.dower at python.org Sat Jul 7 10:46:20 2018 From: steve.dower at python.org (Steve Dower) Date: Sat, 7 Jul 2018 07:46:20 -0700 Subject: [Python-Dev] A "day of silence" on PEP 572? In-Reply-To: References: <20180706213935.62653fa0@fsol> <20180707000237.20cd23ff@fsol> <16471e3bda0.27a3.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: There has been off-list discussion with the authors, for sure. But most of the recent threads are disputes and not useful. At this point, if you're not helping clarify what?s in the PEP, you?re not helping by posting an opinion. But since we can?t possibly convince everyone not to post their opinions, perhaps all the rest of us (especially the PEP authors!) should abandon the list for a few days and let them shout it out :) Top-posted from my Windows 10 phone From: Steve Holden Sent: Friday, July 6, 2018 16:30 To: Ryan Gonzalez Cc: Antoine Pitrou; Python-Dev at Python. Org Subject: Re: [Python-Dev] A "day of silence" on PEP 572? On Sat, Jul 7, 2018 at 12:18 AM, Ryan Gonzalez wrote: On July 6, 2018 5:04:05 PM Antoine Pitrou wrote: (or contact the PEP's authors privately). Hoenstly, this feels like a recipe for a disaster... ?Many of the people who have strong opinions in this know the PEP authors from years of working together.? ? ?They might feel that personal channels are appropriate.? I'd agree it would be a bit presumptuous and spammy of others to use them. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Sat Jul 7 10:50:58 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 7 Jul 2018 16:50:58 +0200 Subject: [Python-Dev] PEP 575, 576, 579 and 580 References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Message-ID: <20180707165058.5b134c95@fsol> On Sun, 8 Jul 2018 00:14:13 +1000 Nick Coghlan wrote: > > > So, let us gather the requirements for a new calling API. > > > Here are my starting suggestions: > > > > 1. The new API should be fully backwards compatible and shouldn't break the > > ABI > > 2. The new API should be used internally so that 3rd party extensions are > > not second class citizens in term of call performance. > > 3. The new API should not prevent 3rd party extensions having full > > introspection capabilities, supporting keyword arguments or another feature > > supported by Python functions. > > 4. The implementation should not exceed D lines of code delta and T lines of > > code in total size. I would suggest +200 and 1000 for D and T respectively > > (or is that too restrictive?). > > 5. It should speed up CPython for the standard benchmark suite. > > 6. It should be understandable. > > I like points 1, 2, 3, and 6, but I think point 4 should be a design > trade-off rather than a requirement, since minimising the delta in > CPython becomes an anti-goal if the outcome of doing so is to make the > change harder to adopt for third party projects (at the same time, a > delta that's too large is unlikely to be accepted, reviewed and > merged, which is what makes it a trade-off). > > I don't think point 5 is a goal here either, as the problem isn't that > these calling optimisations don't exist, it's that they don't > currently have a public API that third party projects can access (the > most recent METH_FASTCALL thread covers that pretty well). Agreed. The goal is not to speed up CPython but to bring third-party extensions up to speed (both literally and figuratively). Regards Antoine. From solipsis at pitrou.net Sat Jul 7 10:51:40 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sat, 7 Jul 2018 16:51:40 +0200 Subject: [Python-Dev] PEP 575, 576, 579 and 580 References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Message-ID: <20180707165140.6610d8b4@fsol> On Sat, 7 Jul 2018 16:39:08 +0200 Stefan Behnel wrote: > Nick Coghlan schrieb am 07.07.2018 um 16:14: > > when the new calling > > convention is tied to a protocol that any type can implement (as PEP > > 580 proposes), the concern doesn't even arise. > > Nick, +1 to all of what you said in your reply, and I also really like the > fact that this proposal is creating a new, general protocol that removes > lots of type special casing from places where objects are being called > "efficiently". We're essentially designing a Fast Duck Calling convention here. The Quick Quack protocole? From songofacandy at gmail.com Sat Jul 7 11:16:33 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Sun, 8 Jul 2018 00:16:33 +0900 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Message-ID: > IMO, mailing lists are a terrible way to do software design, but a good > way to gather requirements as it makes less likely that someone will be > forgotten. > Agreed. There are several topics we should discuss for these PEPs. Mailing list is hard to follow. Can we have other communication channel? Dedicated Github repository? zulip? or discuss.python.org? > So, let us gather the requirements for a new calling API. > > Here are my starting suggestions: > > 1. The new API should be fully backwards compatible and shouldn't break > the ABI Agreed. We have chance to break ABI/API slightly at Python 4, although breakage should be very small compared with Python 3. Until then, we should keep backward compatibility as possible. > 2. The new API should be used internally so that 3rd party extensions > are not second class citizens in term of call performance. These PEPs proposes new public protocol which can be implemented by 3rd party extensions, especially Cython. In this meaning, it's not used only *internally*. > 3. The new API should not prevent 3rd party extensions having full > introspection capabilities, supporting keyword arguments or another > feature supported by Python functions. OK. > 4. The implementation should not exceed D lines of code delta and T > lines of code in total size. I would suggest +200 and 1000 for D and T > respectively (or is that too restrictive?). Hmm, I think this should be considered as (Frequency * Value) / (Complexity). Especially, if PEP 580 can removes 2000 lines of code, T>1000 seems OK. > 5. It should speed up CPython for the standard benchmark suite. I think it's impossible in short term. We have specialized optimization (FASTCALL and LOAD_METHOD/CALL_METHOD) already. These optimization makes simple method calls 30% faster. These PEPs makes 3rd party callable types can utilize these optimization. > 6. It should be understandable. > OK. While main audience is Cython, C extension writer should be able to use new protocols by handwritten extension. > What am I missing? Comments from the maintainers of Cython and other > similar tools would be appreciated. > > Cheers, > Mark. -- INADA Naoki From stefan_ml at behnel.de Sat Jul 7 11:29:31 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 7 Jul 2018 17:29:31 +0200 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Message-ID: INADA Naoki schrieb am 07.07.2018 um 17:16: >> 2. The new API should be used internally so that 3rd party extensions >> are not second class citizens in term of call performance. > > These PEPs proposes new public protocol which can be implemented > by 3rd party extensions, especially Cython. > In this meaning, it's not used only *internally*. I think Mark meant that the API should *also* be used internally, in the same way that external code uses it. Meaning, there shouldn't be a separate internal API. Stefan From songofacandy at gmail.com Sat Jul 7 11:36:24 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Sun, 8 Jul 2018 00:36:24 +0900 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: <20180707165058.5b134c95@fsol> References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> <20180707165058.5b134c95@fsol> Message-ID: > > > 5. It should speed up CPython for the standard benchmark suite. ... > > > > I don't think point 5 is a goal here either, as the problem isn't that > > these calling optimisations don't exist, it's that they don't > > currently have a public API that third party projects can access (the > > most recent METH_FASTCALL thread covers that pretty well). > > Agreed. The goal is not to speed up CPython but to bring third-party > extensions up to speed (both literally and figuratively). > For clarify, main goal is not just only 3rd party extension faster. Publicate some private APIs is enough for it. Goals of these PEP 576 (GitHub version) and 580 is making custom callable type (especially method-like object) faster. Because most functions and methods are defined with PyMethodDef and m_methods / tp_methods, these PEPs are not needed for them. I think main motivation of these PEPs are modern Python usages: Jupyter notebook + Cython. Unlike extension module writer, we shouldn't expect user knows difference between C and Python. That's why Cython want emulate normal Python function/methods as possible. Regards, -- INADA Naoki From guido at python.org Sat Jul 7 11:48:45 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 7 Jul 2018 08:48:45 -0700 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: This seems more suitable for a style guide. Enforcing such restrictions in the grammar would actually be complicated, due to nesting -- but even if it wasn't, I wouldn't want to, as I don't want to limit future generations to only the benefits of the new construct that we can now come up with. Orthogonality is a good thing in my mind, else we might never have had nested functions or conditional imports. As to why you might want to use := in a function call, I could imagine writing if validate(name := re.search(pattern, line).group(1)): return name The benefit of combining the assignment with the if would be more apparent if there was an if-elif...elif-else pattern, like here: https://github.com/python/peps/pull/695/files#diff-09a4f112ea673a2339f0bec6014ff47fR409 (click off the comments to see it better). On Sat, Jul 7, 2018 at 6:12 AM Giampaolo Rodola' wrote: > Sorry in advance for opening yet another topic about PEP-572. With PEP-572 > being officially accepted I know debating its inclusion in the language is > a useless exercise at this point, but since it's still in "draft" status I > would like to express my opinion as I think this is a feature which can > potentially be abused fairly easily. FWIW I initially found myself > disliking the idea as a whole but > https://github.com/python/cpython/pull/8122 made me (and others) > reconsider it quite a bit (see: > https://twitter.com/grodola/status/1015251302350245888). PR-8122 clearly > shows an improvement in expressiveness and compactness (many folks argue > this is too much) but PEP-572 as it currently stands is too permissive > IMHO. My concern about "easily abusable ugly cases" still remains, and I > think they should be banned instead of just discouraged in the PEP or in > the doc. Since we spend way more time *reading* code rather than writing > it, as a "reader" I would expect a more prudent approach to the problem. > > Proposal > ======== > > 1) allow only one := assignment per line in "if" statements: > >>> if x := val1 and y := val2: # SyntaxError or SyntaxWarning > >>> if x == val1 and y := val2: # SyntaxError or SyntaxWarning > >>> if x := val1 and y == val2: # SyntaxError or SyntaxWarning > >>> if x := val1: # OK > >>> if (x := val1): # OK > > 2) allow := in "while" statements, "if" statements and comprehensions only: > >>> foo(x := 0) # SyntaxError > >>> yield x := 3 # SyntaxError > >>> assert y := 3 # SyntaxError > > 3) (debatable) disallow := if the variable is already defined: > >>> x = 5 > >>> if (x := val): # SyntaxError or SyntaxWarning > > 4) ban "a = (b := c)", "x = a := (b := (c := d))" and similar (they're > just too ugly IMHO) > > Rationale 1 > =========== > > In visual terms assignments in Python have always occurred at the > BEGINNING of the line and always on the most LEFT side: > > >>> foo = fun1() > >>> bar = fun2() > >>> baz = fun3() > > That is where I naturally expect an assignment to be when reading code. My > main concern with PEP-572 is that an assignments can now occur at *any > point* in the line: > > >>> foo = fun1() > >>> bar = fun2() > >>> if foo == val1 and bar == val2 and baz := fun3(): > ... ... > > That forces me to visually scan the whole line horizontally from left to > right 'till its end, looking for possible variables being set. I'm > concerned that I will miss := occurrences because visually they are very > similar to == unless parentheses are made mandatory: > > >>> if foo == val1 and bar == val2 and (baz := fun3()): > ... ... > > Also, in case of multi-line conditionals I have to visually scan the > construct both horizontally AND vertically: > > >>> if (foo == val1 and \ > ... bar == val2 and \ > ... baz := val3): > ... ... > > Again, that is not a place where I would expect to find or look for a > variable assignment. I know I wouldn't like to read or review a code which > does that and I suspect linters will likely end up wanting to emit a > warning in that case (see: https://github.com/PyCQA/pylint/issues/2246). > https://github.com/python/cpython/pull/8116/files avoids using multiple > := per line and that's why the result appears readable enough IMO. > > Rationale 2 > =========== > > PEP-572 states: > > > The := operator may be used directly in a positional function call > argument > > That means allowing: > > >>> foo(x := 0) > > I honestly don't see why anyone would want to call a function AND assign a > variable value at the same time (except in comprehensions). With this in > place I not only have to guard against "if" statements assigning values at > any point in the code, but also function calls, both horizontally and > vertically e.g.: > > >>> foo(some_long_var_name, another_one, x := bar(), > y := fun()) > > To me this looks like the perfect example of where this functionality can > be abused. Also, I'm not clear what PEP-572 intend to do about "all other > places". E.g. should these cases be allowed? (IMO no) > > >>> yield x := 3 > >>> assert y := 3 > > -- > Giampaolo - http://grodola.blogspot.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sat Jul 7 12:53:14 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 7 Jul 2018 11:53:14 -0500 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: [Guido] > ... > As to why you might want to use := in a function call, I could imagine > writing > > if validate(name := re.search(pattern, line).group(1)): > return name > When I was staring at my code, I never mentioned the very first plausible use I bumped into (in code I was actively working on at the time): while not probable_prime(p := randrange(lo, hi)): pass # and now `p` is likely a random prime in range I never mentioned it because I expected it would annoy people on 3(!) counts: - assigning in a function call - reducing the loop body to `pass` - using the binding long after the loop ended Indeed, for those reasons it wasn't "an obvious" win to me - or an obvious loss. So I just moved on. However, after staring at hundreds of other cases, it does strike me as "a small win" today - my brain cells have rewired to recognize more ":=" patterns at a glance. Whether that's a good thing or not I don't know, but it is a real thing ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Jul 7 15:58:48 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 7 Jul 2018 15:58:48 -0400 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On 7/7/2018 12:53 PM, Tim Peters wrote: > [Guido] > > ... > As to why you might want to use := in a function call, I could > imagine writing > > ??? if validate(name := re.search(pattern, line).group(1)): > ??????? return name If name has to be non-blank to pass validate, one can avoid the assignment within the function call be adding a redundant pre-test. if name := re.search(pattern, line).group(1) and validate(name): return name Giampaolo would presumably prefer this, but I don't think such preference should be enforced on everyone. If name == '' is valid, then the alternative is the current one, using a separate assignment statement. name = re.search(pattern, line).group(1) if validate(name): return name > When I was staring at my code, I never mentioned the very first > plausible use I bumped into (in code I was actively working on at the time): > > while not probable_prime(p := randrange(lo, hi)): > ? ? ?pass > # and now `p` is likely a random prime in range As long as lo excludes 0: while p := randrange(lo, hi) and not probable_prime(p): continue I can see how someone might prefer this stylistically, but it is buggy. If this is contained in a function (very likely) and lo could be <= 0, because it is either passed in or calculated, 0 could be passed on a likely prime! > I never mentioned it because I expected it would annoy people on 3(!) > counts: > > - assigning in a function call This is a style preference that people can and will disagree on. In any case, I think correctness trumps beauty, just as it trumps speed. > - reducing the loop body to `pass` I fixed that ;-). 'continue' better expresses the 'try again' part of English versions, such as "While the trial value is not acceptable, try again." > - using the binding long after the loop ended The same is true for the current 4-line loop and a half. while True: p = randrange(lo, hi) if probable_prime(p): break # p used somewhere else > Indeed, for those reasons it wasn't "an obvious" win to me - or an > obvious loss.? So I just moved on. > > However, after staring at hundreds of other cases, it does strike me as > "a small win" today - my brain cells have rewired to recognize more ":=" > patterns at a glance. > > Whether that's a good thing or not I don't know, but it is a real thing ;-) I must admit that I too am already more comfortable with := now than I was originally. -- Terry Jan Reedy From brett at python.org Sat Jul 7 16:21:08 2018 From: brett at python.org (Brett Cannon) Date: Sat, 7 Jul 2018 13:21:08 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: On Fri, Jul 6, 2018, 16:32 Guido van Rossum, wrote: > On Fri, Jul 6, 2018 at 4:19 PM Terry Reedy wrote: > >> Since Guido, the first respondent, did not immediately shoot the idea >> down, I intend to flesh it out and make it more concrete. >> > > Maybe I should have shot it down. The term is entrenched in multiple > languages by now (e.g. https://en.wikipedia.org/wiki/List_comprehension). > Regarding "list builder" one could argue that it would just add more > confusion, since there's already an unrelated Builder Pattern ( > https://en.wikipedia.org/wiki/Builder_pattern) commonly used in Java. > (Though I worry about the presence of a Python example in that Wikipedia > page. :-) > > Also, "generator builder" is not much more expressive than "generator > expression", and the key observation that led to this idea was that it's > such a mouthful to say "comprehensions and generator expressions". Maybe > it's not too late to start calling the latter "generator comprehensions" so > that maybe by the year 2025 we can say "comprehensions" and everyone will > understand we mean all four types? > > FWIW more people should start using "list display" etc. for things like > [a, b, c]. > I can get behind "generator comprehension" and "list display". The builder idea isn't doing it for me. -Brett > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Sat Jul 7 17:11:44 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 7 Jul 2018 23:11:44 +0200 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: References: Message-ID: <5B412C90.5000905@UGent.be> On 2018-07-07 15:38, Mark Shannon wrote: > Hi, > > We seem to have a plethora of PEPs where we really ought to have one (or > none?). - PEP 575 has been withdrawn. - PEP 579 is an informational PEP with the bigger picture; it does contain some of the requirements that you want to discuss here. - PEP 580 and PEP 576 are two alternative implementations of a protocol to optimize callables implemented in C. > 5. It should speed up CPython for the standard benchmark suite. I'd like to replace this by: must *not slow down* the standard benchmark suite and preferable should not slow down anything. From J.Demeyer at UGent.be Sat Jul 7 18:30:53 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sun, 8 Jul 2018 00:30:53 +0200 Subject: [Python-Dev] Comparing PEP 576 and PEP 580 In-Reply-To: <4f022f79891c47e284dbb16a77418620@xmail103.UGent.be> References: <5B3B91ED.4060309@UGent.be> <917922e45d304ba3b9f94c874d2fd095@xmail101.UGent.be> <5B3CEEEE.30204@UGent.be> <2ab8f92f24af4c0ba47df26fb35fba02@xmail103.UGent.be> <5B3DE556.5040903@UGent.be> <2eba9cc56fac4a588e2ce2b27c41d7d9@xmail103.UGent.be> <5B3E080E.4030107@UGent.be> <5B3E341E.70402@UGent.be> <1cb5933b3ce7418d90aa8b9488efc8e7@xmail103.UGent.be> <5B3F3B4E.6080508@UGent.be> <5B3FF511.7060205@UGent.be> <4f022f79891c47e284dbb16a77418620@xmail103.UGent.be> Message-ID: <5B413F1D.6030408@UGent.be> On 2018-07-07 14:54, Mark Shannon wrote: > There is a minimal implementation and has been for a while. > There is a link at the bottom of the PEP. Yes, I saw that but the implementation does not correspond to the PEP. In particular, this sentence from the PEP has not been implemented: When binding a method_descriptor instance to an instance of its owning class, a bound_method will be created instead of a builtin_function_or_method It's not clear to me whether you still want to implement that or whether it should be dropped from the PEP. > PEP 576 adds a new calling convention which can be used by *any* object. > Seems quite extensible to me. Yes and no. Yes, it can do anything. But because it can do anything, callers cannot optimize certain special cases. For example, in PEP 576 you need an extra flag Py_TPFLAGS_FUNCTION_DESCRIPTOR because your protocol doesn't specify anything about __get__. Imagine that you want to support more optimizations like that in the future, how do you plan to do that? Of course, you can always add more stuff to PyTypeObject, but a separate structure like what I propose in PEP 580 might make more sense. Jeroen. From brett at python.org Sat Jul 7 20:55:56 2018 From: brett at python.org (Brett Cannon) Date: Sat, 7 Jul 2018 17:55:56 -0700 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Message-ID: On Sat, Jul 7, 2018, 08:17 INADA Naoki, wrote: > > IMO, mailing lists are a terrible way to do software design, but a good > > way to gather requirements as it makes less likely that someone will be > > forgotten. > > > > Agreed. There are several topics we should discuss for these PEPs. > Mailing list is hard to follow. > > Can we have other communication channel? Dedicated Github repository? > zulip? or discuss.python.org? > > > So, let us gather the requirements for a new calling API. > > > > Here are my starting suggestions: > > > > 1. The new API should be fully backwards compatible and shouldn't break > > the ABI > > Agreed. We have chance to break ABI/API slightly at Python 4, although > breakage should be very small compared with Python 3. > > Until then, we should keep backward compatibility as possible. > > > 2. The new API should be used internally so that 3rd party extensions > > are not second class citizens in term of call performance. > > These PEPs proposes new public protocol which can be implemented > by 3rd party extensions, especially Cython. > In this meaning, it's not used only *internally*. > > > 3. The new API should not prevent 3rd party extensions having full > > introspection capabilities, supporting keyword arguments or another > > feature supported by Python functions. > > OK. > > > 4. The implementation should not exceed D lines of code delta and T > > lines of code in total size. I would suggest +200 and 1000 for D and T > > respectively (or is that too restrictive?). > > Hmm, I think this should be considered as (Frequency * Value) / > (Complexity). > Especially, if PEP 580 can removes 2000 lines of code, T>1000 seems OK. > I don't think any concrete number is really going to be helpful. This is probably going to come down to subjective "will this be complicated and hard to maintain?" And that call will probably come down to the BDFL for the PEP. -Brett 0p > > 5. It should speed up CPython for the standard benchmark suite. > > I think it's impossible in short term. We have specialized optimization > (FASTCALL and LOAD_METHOD/CALL_METHOD) already. > These optimization makes simple method calls 30% faster. > These PEPs makes 3rd party callable types can utilize these optimization. > > > 6. It should be understandable. > > > > OK. > While main audience is Cython, C extension writer should be able to use > new protocols by handwritten extension. > > > What am I missing? Comments from the maintainers of Cython and other > > similar tools would be appreciated. > > > > Cheers, > > Mark. > > > -- > INADA Naoki > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Jul 7 22:44:33 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Jul 2018 12:44:33 +1000 Subject: [Python-Dev] Making the status of Provisional PEPs clearer Message-ID: Hi folks, After review from Barry & Guido, I've just merged an update to PEP 1 and the PEP index generator that separates out provisionally accepted PEPs to their own state in the PEP flow: https://github.com/python/peps/commit/307dda38d4e7a5760dd4979ae9978a4eb1e70589 To date, that status has been reported as "Accepted" both in the original PEPs and in the main PEP index, now it gets reported as Provisional in both places. Cheers, Nick. P.S. As part of this, I switched the flow diagram in PEP 1 from a PNG to an SVG, which seems to have confused python.org's image rendering: https://github.com/python/peps/issues/701 I'm already looking into it, but am open to tips from folks more familiar with the website's rendering machinery. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sat Jul 7 22:48:47 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 7 Jul 2018 19:48:47 -0700 Subject: [Python-Dev] Making the status of Provisional PEPs clearer In-Reply-To: References: Message-ID: Should we update some PEPs with the new status? E.g. PEP 484. On Sat, Jul 7, 2018 at 7:46 PM Nick Coghlan wrote: > Hi folks, > > After review from Barry & Guido, I've just merged an update to PEP 1 > and the PEP index generator that separates out provisionally accepted > PEPs to their own state in the PEP flow: > > https://github.com/python/peps/commit/307dda38d4e7a5760dd4979ae9978a4eb1e70589 > > To date, that status has been reported as "Accepted" both in the > original PEPs and in the main PEP index, now it gets reported as > Provisional in both places. > > Cheers, > Nick. > > P.S. As part of this, I switched the flow diagram in PEP 1 from a PNG > to an SVG, which seems to have confused python.org's image rendering: > https://github.com/python/peps/issues/701 > > I'm already looking into it, but am open to tips from folks more > familiar with the website's rendering machinery. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-dev at mgmiller.net Sat Jul 7 23:09:08 2018 From: python-dev at mgmiller.net (Mike Miller) Date: Sat, 7 Jul 2018 20:09:08 -0700 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On 2018-07-07 06:09, Giampaolo Rodola' wrote: > I initially found myself disliking the idea as a whole but > https://github.com/python/cpython/pull/8122 made me (and others) reconsider it > quite a bit (see: https://twitter.com/grodola/status/1015251302350245888). > PR-8122 clearly shows an improvement in expressiveness and compactness (many > folks argue this is too much) One of the requirements from the PEP (informing its design) is that there would be a significant need and opportunities to use it with multiple and/or compound conditions. While it may be a function of Victor's choices, from my reading of the pull request the vast majority of the improved lines has but a single, simple condition such as: while (data := f.readframes(1024)): I believe Giampaolo has a good point. These expressions are useful, however don't seem to have much value outside these examples. On the subject on imagining how they would be used, suppose we could look at JavaScript or other expression-oriented languages for best practices. Didn't find many admonitions however, other than keep it simple: https://stackoverflow.com/q/9726496/450917 -Mike From ncoghlan at gmail.com Sat Jul 7 23:29:39 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 8 Jul 2018 13:29:39 +1000 Subject: [Python-Dev] Making the status of Provisional PEPs clearer In-Reply-To: References: Message-ID: On 8 July 2018 at 12:48, Guido van Rossum wrote: > Should we update some PEPs with the new status? E.g. PEP 484. Aye, 3 PEPs were given the new status as part of the commit: - 484 (type hinting syntax) - 518 (pyproject.toml/[build-system].requires) - 517 (pyproject.toml/[build-system].backend) Checking the other Accepted PEPs, it looks like a few of them need to be marked Final now that 3.7 has been released, but I don't believe any of them are Provisional. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Sat Jul 7 23:33:18 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 7 Jul 2018 20:33:18 -0700 Subject: [Python-Dev] Making the status of Provisional PEPs clearer In-Reply-To: References: Message-ID: Thanks! On Sat, Jul 7, 2018 at 8:29 PM Nick Coghlan wrote: > On 8 July 2018 at 12:48, Guido van Rossum wrote: > > Should we update some PEPs with the new status? E.g. PEP 484. > > Aye, 3 PEPs were given the new status as part of the commit: > > - 484 (type hinting syntax) > - 518 (pyproject.toml/[build-system].requires) > - 517 (pyproject.toml/[build-system].backend) > > Checking the other Accepted PEPs, it looks like a few of them need to > be marked Final now that 3.7 has been released, but I don't believe > any of them are Provisional. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Jul 8 00:13:27 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 7 Jul 2018 21:13:27 -0700 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> References: <073a528f-b917-c574-576f-171c29a48518@hotpy.org> Message-ID: On Sat, Jul 7, 2018 at 6:38 AM, Mark Shannon wrote: > 1. The new API should be fully backwards compatible and shouldn't break the > ABI Which ABI? The stable ABI (PEP 384)? I don't think object layout is exposed there, though I'm not sure of the details. The regular ABI that almost everyone actually uses? That's already broken on every minor release, so you shouldn't spend any time worrying about preserving compatibility there. -n -- Nathaniel J. Smith -- https://vorpus.org From tim.peters at gmail.com Sun Jul 8 00:27:31 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 7 Jul 2018 23:27:31 -0500 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: > > [Tim] > > When I was staring at my code, I never mentioned the very first > > plausible use I bumped into (in code I was actively working on at the > time): > > > > while not probable_prime(p := randrange(lo, hi)): > > pass > > # and now `p` is likely a random prime in range > {Terry Reedy] > > As long as lo excludes 0: > > while p := randrange(lo, hi) and not probable_prime(p): > continue > I can see how someone might prefer this stylistically, but it is buggy. > If this is contained in a function (very likely) and lo could be <= 0, > because it is either passed in or calculated, 0 could be passed on a > likely prime! > I never write code that uses "and" relying on that context-specific data constraints "guarantee" the LHS is always true. That combines a delicate precondition with "a trick". Dreadful. I won't even write it this way, which keeps "the trick" but eliminates the hidden data assumption: while [p := randrange(lo, hi)] and not probable_prime(p): A singleton list is always truthy, so at least now it makes no assumptions about the value bound to `p`. I could be paid to write it this way, but most employers couldn't afford to pay enough to do it twice ;-) : while [(p := randrange(lo, hi)), not probable_prime(p)][-1]: That always works and doesn't rely on "a trick", but is ugly, obscure, and inefficient. > I never mentioned it because I expected it would annoy people on 3(!) > > counts: > > > > - assigning in a function call > > This is a style preference that people can and will disagree on. In any > case, I think correctness trumps beauty, just as it trumps speed. > All else being equal (and, yup, correctness is more equal than the others), I like to push assignments "to the left" as much as possible. > > - reducing the loop body to `pass` > > I fixed that ;-). 'continue' better expresses the 'try again' part of > English versions, such as "While the trial value is not acceptable, try > again." > Thanks! Now that you mention it (it had not occurred to me), I like `continue` much better than `pass` here too. > - using the binding long after the loop ended > The same is true for the current 4-line loop and a half. > > while True: > p = randrange(lo, hi) > if probable_prime(p): > break # p used somewhere else > Sure. But this PEP _started_ with a fancier model wherein the language would magically limit the scope of assignment targets in these block-opening tests. That was eventually removed, but I'm sure we'll see "style guides" demanding that it "should" never be used unless the target is in fact never referenced (at least not before re-binding) after the associated block ends. It's been mildly surprising to me to see how often that _is_ the case in real code. But, as in the example above, I won't be following such a rule rigidly. > Indeed, for those reasons it wasn't "an obvious" win to me - or an > > obvious loss. So I just moved on. > > > > However, after staring at hundreds of other cases, it does strike me as > > "a small win" today - my brain cells have rewired to recognize more ":=" > > patterns at a glance. > > > > Whether that's a good thing or not I don't know, but it is a real thing > ;-) > > I must admit that I too am already more comfortable with := now than I > was originally. The stories about its uncanny ability to destroy entire projects with a single use may have been exaggerated ;-) But, ya, I've tried playing with it much more than most so far, and my bar for "obvious little win" has lowered. Not much, though, and it seems to have bottomed out with that example. So, in the end, I expect I'll use it as sparingly - and gratefully! - as in all the other languages I've used with assignment expressions. Next project: rehabilitate the much-maligned GOTO statement ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Sun Jul 8 03:07:06 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sun, 8 Jul 2018 09:07:06 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: <6e5e107b5cb44137b7d41d8bd14898b0@xmail103.UGent.be> References: <5B3E30D1.3080706@UGent.be> <6e5e107b5cb44137b7d41d8bd14898b0@xmail103.UGent.be> Message-ID: <5B41B81A.6060301@UGent.be> On 2018-07-07 10:55, Serhiy Storchaka wrote: > The first part of > handling arguments can be made outside of the C function, by the calling > API. Sure, it could be done but I don't see the advantage. I don't think you will gain performance because you are just moving code from one place to another. And how do you plan to deal with *args and **kwds in your proposal? You'll need to make sure that this doesn't become slower. From stefan_ml at behnel.de Sun Jul 8 03:34:53 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 8 Jul 2018 09:34:53 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: <5B41B81A.6060301@UGent.be> References: <5B3E30D1.3080706@UGent.be> <6e5e107b5cb44137b7d41d8bd14898b0@xmail103.UGent.be> <5B41B81A.6060301@UGent.be> Message-ID: Jeroen Demeyer schrieb am 08.07.2018 um 09:07: > On 2018-07-07 10:55, Serhiy Storchaka wrote: >> The first part of >> handling arguments can be made outside of the C function, by the calling >> API. > > Sure, it could be done but I don't see the advantage. I don't think you > will gain performance because you are just moving code from one place to > another. You probably can, by allowing the caller to decide how to map the keyword arguments. Passing them as a flat array is the fastest way for the callee to evaluate them, so that's great. For the caller, they might already be available in that format or not, so the caller can save time if they are, and only needs to invest time if they are not. > And how do you plan to deal with *args and **kwds in your > proposal? You'll need to make sure that this doesn't become slower. That, on the other hand, is an actual concern. If we already have a tuple and dict, unpacking them for the call and then repacking them on the other side is a serious performance regression ? for this specific use case. The question is, how important is that use case compared to everything else? And, since we have more than one supported signature anyway, can we leave that case to a non-fastcall case? In the end, it's up to the callee to decide which protocol to support and use, and if the intention is to work with **kwargs, then maybe the callee should not use the fastcall protocol in the first place. Stefan From larry at hastings.org Sun Jul 8 03:45:43 2018 From: larry at hastings.org (Larry Hastings) Date: Sun, 8 Jul 2018 00:45:43 -0700 Subject: [Python-Dev] Time for 3.4.9 and 3.5.6 Message-ID: My six-month cadence means it's time for the next releases of 3.4 and 3.5.? There haven't been many changes since the last releases--two, to be exact.? These two security fixes were backported to both 3.4 and 3.5: * bpo-32981: Fix catastrophic backtracking vulns (GH-5955) * bpo-33001: Prevent buffer overrun in os.symlink (GH-5989) 3.5 also got some doc-only changes related to the online "version switcher" dropdown.? (They weren't backported to 3.4 because we don't list 3.4 in the version switcher dropdown anymore.) There are currently no PRs open for either 3.4 or 3.5, and they also have no open "release blocker" or "deferred blocker" bugs.? It seems things are pretty quiet in our two security-fixes-only branches--a good way to be! I therefore propose to cut the RCs in a week and a half, and the finals two weeks later.? So: Wednesday? July 18 2018 - 3.4.9rc1 and 3.5.6rc1 Wednesday August 1 2018 - 3.4.9 final and 3.5.6 final If anybody needs more time I'm totally happy to accommodate them--you can probably have all the time you need.? I'm trying to keep to my rough six-month cadence, but honestly that's pretty arbitrary. Thanks to all of you who keep making 3.4 and 3.5 better, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Sun Jul 8 05:41:27 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sun, 8 Jul 2018 11:41:27 +0200 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Sat, Jul 7, 2018 at 5:48 PM Guido van Rossum wrote: > Enforcing such restrictions in the grammar would actually be complicated, > due to nesting -- but even if it wasn't, I wouldn't want to, as I don't > want to limit future generations to only the benefits of the new construct > that we can now come up with. Orthogonality is a good thing in my mind, > else we might never have had nested functions or conditional imports. > Got it. As to why you might want to use := in a function call, I could imagine > writing > > if validate(name := re.search(pattern, line).group(1)): > return name > I meant this case: >>> foo(x := 1) # execute "x = 1" AND "foo(1)" I find that (space between the parentheses of a function call statement) too unnatural as a place where to put an assignment. It is not even "guarded" by a keyword like "if" or "while" which can help as indicators that an assignment may occur. Also, I think it's way too easy to confuse it with a keyword argument: >>> foo(x = 1) # keyword arg >>> foo(x := 1) # assignment + value passing As for "assert" what I'm concern about is the proliferation of things like this: class Foo: def __init__(self): assert self.x := fun1() assert self.y := fun2() assert self.z := fun3() When I look at that my brain tells me that the main subject of the line is "assert", not the assignment, but maybe it's just because I'm not used to it. That aside there's the question of what to do when "python -O" switch is used. With this in place "-O" would acquire a new meaning, as it would disable "assert" statements AND assignments. > On Sat, Jul 7, 2018 at 6:12 AM Giampaolo Rodola' > wrote: > >> Sorry in advance for opening yet another topic about PEP-572. With >> PEP-572 being officially accepted I know debating its inclusion in the >> language is a useless exercise at this point, but since it's still in >> "draft" status I would like to express my opinion as I think this is a >> feature which can potentially be abused fairly easily. FWIW I initially >> found myself disliking the idea as a whole but >> https://github.com/python/cpython/pull/8122 made me (and others) >> reconsider it quite a bit (see: >> https://twitter.com/grodola/status/1015251302350245888). PR-8122 clearly >> shows an improvement in expressiveness and compactness (many folks argue >> this is too much) but PEP-572 as it currently stands is too permissive >> IMHO. My concern about "easily abusable ugly cases" still remains, and I >> think they should be banned instead of just discouraged in the PEP or in >> the doc. Since we spend way more time *reading* code rather than writing >> it, as a "reader" I would expect a more prudent approach to the problem. >> >> Proposal >> ======== >> >> 1) allow only one := assignment per line in "if" statements: >> >>> if x := val1 and y := val2: # SyntaxError or SyntaxWarning >> >>> if x == val1 and y := val2: # SyntaxError or SyntaxWarning >> >>> if x := val1 and y == val2: # SyntaxError or SyntaxWarning >> >>> if x := val1: # OK >> >>> if (x := val1): # OK >> >> 2) allow := in "while" statements, "if" statements and comprehensions >> only: >> >>> foo(x := 0) # SyntaxError >> >>> yield x := 3 # SyntaxError >> >>> assert y := 3 # SyntaxError >> >> 3) (debatable) disallow := if the variable is already defined: >> >>> x = 5 >> >>> if (x := val): # SyntaxError or SyntaxWarning >> >> 4) ban "a = (b := c)", "x = a := (b := (c := d))" and similar (they're >> just too ugly IMHO) >> >> Rationale 1 >> =========== >> >> In visual terms assignments in Python have always occurred at the >> BEGINNING of the line and always on the most LEFT side: >> >> >>> foo = fun1() >> >>> bar = fun2() >> >>> baz = fun3() >> >> That is where I naturally expect an assignment to be when reading code. >> My main concern with PEP-572 is that an assignments can now occur at *any >> point* in the line: >> >> >>> foo = fun1() >> >>> bar = fun2() >> >>> if foo == val1 and bar == val2 and baz := fun3(): >> ... ... >> >> That forces me to visually scan the whole line horizontally from left to >> right 'till its end, looking for possible variables being set. I'm co # >> execute "foo(1)" AND "x = 1"ncerned that I will miss := occurrences because >> visually they are very similar to == unless parentheses are made mandatory: >> >> >>> if foo == val1 and bar == val2 and (baz := fun3()): >> ... ... >> >> Also, in case of multi-line conditionals I have to visually scan the >> construct both horizontally AND vertically: >> >> >>> if (foo == val1 and \ >> ... bar == val2 and \ >> ... baz := val3): >> ... ... >> >> Again, that is not a place where I would expect to find or look for a >> variable assignment. I know I wouldn't like to read or review a code which >> does that and I suspect linters will likely end up wanting to emit a >> warning in that case (see: https://github.com/PyCQA/pylint/issues/2246). >> https://github.com/python/cpython/pull/8116/files avoids using multiple >> := per line and that's why the result appears readable enough IMO. >> >> Rationale 2 >> =========== >> >> PEP-572 states: >> >> > The := operator may be used directly in a positional function call >> argument >> >> That means allowing: >> >> >>> foo(x := 0) >> >> I honestly don't see why anyone would want to call a function AND assign >> a variable value at the same time (except in comprehensions). With this in >> place I not only have to guard against "if" statements assigning values at >> any point in the code, but also function calls, both horizontally and >> vertically e.g.: >> >> >>> foo(some_long_var_name, another_one, x := bar(), >> y := fun()) >> >> To me this looks like the perfect example of where this functionality can >> be abused. Also, I'm not clear what PEP-572 intend to do about "all other >> places". E.g. should these cases be allowed? (IMO no) >> >> >>> yield x := 3 >> >>> assert y := 3 >> >> -- >> Giampaolo - http://grodola.blogspot.com >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> > > > -- > --Guido van Rossum (python.org/~guido) > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Sun Jul 8 07:03:32 2018 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 8 Jul 2018 07:03:32 -0400 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On 7/8/2018 5:41 AM, Giampaolo Rodola' wrote: > As for "assert" what I'm concern about is the proliferation of things > like this: > ? ? class Foo: > ? ? ? ? def __init__(self): > ? ? ? ? ? ? assert self.x := fun1() > ? ? ? ? ? ? assert self.y := fun2() > ? ? ? ? ? ? assert self.z := fun3() > > When I look at that my brain tells me that the main subject of the line > is "assert", not the assignment, but maybe it's just because I'm not > used to it. That aside there's the question of what to do when "python > -O" switch is used. With this in place "-O" would acquire a new meaning, > as it would disable "assert" statements AND assignments. It has always meant "disable the assert statement and therefore any side effects the expression has". It's just that now the side effects are more obvious, or maybe easier to create. Even pre-572 I've been bitten by this, I'm ashamed to admit. Eric From steve at holdenweb.com Sun Jul 8 12:45:24 2018 From: steve at holdenweb.com (Steve Holden) Date: Sun, 8 Jul 2018 17:45:24 +0100 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' wrote: > ?[...] > I find that (space between the parentheses of a function call statement) > too unnatural as a place where to put an assignment. It is not even > "guarded" by a keyword like "if" or "while" which can help as indicators > that an assignment may occur. Also, I think it's way too easy to confuse it > with a keyword argument: > > >>> foo(x = 1) # keyword arg > >>> foo(x := 1) # assignment + value passing > ?[...] > ?But the PEP 8 spellings are? foo(x=1) and f(x := 1). The extra spacing makes it obvious that this isn't a regular named argument. -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Sun Jul 8 13:05:24 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Sun, 8 Jul 2018 20:05:24 +0300 Subject: [Python-Dev] Time for 3.4.9 and 3.5.6 In-Reply-To: References: Message-ID: I'll use this opportunity to remind you that 3.4 build is broken -- it can't be built from start to installer with the instructions given because of outside factors (CPython has migrated from Hg to Git). https://bugs.python.org/issue31623 about this was ignored (see https://bugs.python.org/issue31623#msg303708 for supplemental fixes). If this isn't something considered needing a fix, the claim that 3.4 is supported in any shape and form is but a pretense -- if something can't be built, it can't be used. On 08.07.2018 10:45, Larry Hastings wrote: > > > My six-month cadence means it's time for the next releases of 3.4 and > 3.5.? There haven't been many changes since the last releases--two, to > be exact.? These two security fixes were backported to both 3.4 and 3.5: > > * bpo-32981: Fix catastrophic backtracking vulns (GH-5955) > * bpo-33001: Prevent buffer overrun in os.symlink (GH-5989) > > 3.5 also got some doc-only changes related to the online "version > switcher" dropdown.? (They weren't backported to 3.4 because we don't > list 3.4 in the version switcher dropdown anymore.) > > > There are currently no PRs open for either 3.4 or 3.5, and they also > have no open "release blocker" or "deferred blocker" bugs. It seems > things are pretty quiet in our two security-fixes-only branches--a > good way to be! > > I therefore propose to cut the RCs in a week and a half, and the > finals two weeks later.? So: > > Wednesday? July 18 2018 - 3.4.9rc1 and 3.5.6rc1 > Wednesday August 1 2018 - 3.4.9 final and 3.5.6 final > > If anybody needs more time I'm totally happy to accommodate them--you > can probably have all the time you need.? I'm trying to keep to my > rough six-month cadence, but honestly that's pretty arbitrary. > > Thanks to all of you who keep making 3.4 and 3.5 better, > > > //arry/ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Sun Jul 8 13:14:45 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sun, 8 Jul 2018 19:14:45 +0200 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Sun, Jul 8, 2018 at 6:45 PM Steve Holden wrote: > On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' > wrote: > >> ?[...] >> I find that (space between the parentheses of a function call statement) >> too unnatural as a place where to put an assignment. It is not even >> "guarded" by a keyword like "if" or "while" which can help as indicators >> that an assignment may occur. Also, I think it's way too easy to confuse it >> with a keyword argument: >> >> >>> foo(x = 1) # keyword arg >> >>> foo(x := 1) # assignment + value passing >> ?[...] >> > > ?But the PEP 8 spellings are? > > foo(x=1) > > and > > f(x := 1). > > The extra spacing makes it obvious that this isn't a regular named > argument. > What if the author of the code I'm reading didn't respect PEP-8? I don't think it's fair to invoke PEP-8 as a counter-measure to obviate a syntax which can clearly be mistaken with something else simply by omitting 2 spaces. Not to mention that I don't see why anyone would want to declare a variable in there in the first place. -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Jul 8 13:23:50 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 Jul 2018 03:23:50 +1000 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Mon, Jul 9, 2018 at 3:14 AM, Giampaolo Rodola' wrote: > > > On Sun, Jul 8, 2018 at 6:45 PM Steve Holden wrote: >> >> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' >> wrote: >>> >>> [...] >>> I find that (space between the parentheses of a function call statement) >>> too unnatural as a place where to put an assignment. It is not even >>> "guarded" by a keyword like "if" or "while" which can help as indicators >>> that an assignment may occur. Also, I think it's way too easy to confuse it >>> with a keyword argument: >>> >>> >>> foo(x = 1) # keyword arg >>> >>> foo(x := 1) # assignment + value passing >>> [...] >> >> >> But the PEP 8 spellings are >> >> foo(x=1) >> >> and >> >> f(x := 1). >> >> The extra spacing makes it obvious that this isn't a regular named >> argument. > > > What if the author of the code I'm reading didn't respect PEP-8? I don't > think it's fair to invoke PEP-8 as a counter-measure to obviate a syntax > which can clearly be mistaken with something else simply by omitting 2 > spaces. Not to mention that I don't see why anyone would want to declare a > variable in there in the first place. > It's not about why someone would want to assign inside a function call. It's about why it should be forbidden. Perhaps nobody has a good reason to use THlS_OBJECT as a variable name, and it's potentially very confusing; but should the grammar of Python forbid it? No. Because there is no value in forbidding it. Python's grammar has a number of weird edge cases due to necessity (for instance, "for x in a if cond else b:" works, but not in a comprehension), and when there's an actual conflict, sure, you can say "but nobody would ever want to do that, so we'll forbid it". In this case, there is no conflict. ChrisA From mertz at gnosis.cx Sun Jul 8 13:24:40 2018 From: mertz at gnosis.cx (David Mertz) Date: Sun, 8 Jul 2018 13:24:40 -0400 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: The case I find more reasonable is assignment in earlier arguments: z = something () w = myfun(x := get_data(), y=calculate(x, z)) I would probably recommend against that in code review, but it's not absurdly obfuscated. On Sun, Jul 8, 2018, 1:15 PM Giampaolo Rodola' wrote: > > > On Sun, Jul 8, 2018 at 6:45 PM Steve Holden wrote: > >> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' >> wrote: >> >>> ?[...] >>> I find that (space between the parentheses of a function call >>> statement) too unnatural as a place where to put an assignment. It is >>> not even "guarded" by a keyword like "if" or "while" which can help as >>> indicators that an assignment may occur. Also, I think it's way too easy to >>> confuse it with a keyword argument: >>> >>> >>> foo(x = 1) # keyword arg >>> >>> foo(x := 1) # assignment + value passing >>> ?[...] >>> >> >> ?But the PEP 8 spellings are? >> >> foo(x=1) >> >> and >> >> f(x := 1). >> >> The extra spacing makes it obvious that this isn't a regular named >> argument. >> > > What if the author of the code I'm reading didn't respect PEP-8? I don't > think it's fair to invoke PEP-8 as a counter-measure to obviate a syntax > which can clearly be mistaken with something else simply by omitting 2 > spaces. Not to mention that I don't see why anyone would want to declare a > variable in there in the first place. > > -- > Giampaolo - http://grodola.blogspot.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Sun Jul 8 13:55:34 2018 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 8 Jul 2018 13:55:34 -0400 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On 7/8/2018 1:23 PM, Chris Angelico wrote: > On Mon, Jul 9, 2018 at 3:14 AM, Giampaolo Rodola' wrote: >> >> >> On Sun, Jul 8, 2018 at 6:45 PM Steve Holden wrote: >>> But the PEP 8 spellings are >>> >>> foo(x=1) >>> >>> and >>> >>> f(x := 1). >>> >>> The extra spacing makes it obvious that this isn't a regular named >>> argument. >> >> >> What if the author of the code I'm reading didn't respect PEP-8? I don't >> think it's fair to invoke PEP-8 as a counter-measure to obviate a syntax >> which can clearly be mistaken with something else simply by omitting 2 >> spaces. Not to mention that I don't see why anyone would want to declare a >> variable in there in the first place. >> > > It's not about why someone would want to assign inside a function > call. It's about why it should be forbidden. Perhaps nobody has a good > reason to use THlS_OBJECT as a variable name, and it's potentially > very confusing; but should the grammar of Python forbid it? No. > Because there is no value in forbidding it. > > Python's grammar has a number of weird edge cases due to necessity > (for instance, "for x in a if cond else b:" works, but not in a > comprehension), and when there's an actual conflict, sure, you can say > "but nobody would ever want to do that, so we'll forbid it". In this > case, there is no conflict. I agree with Chris in this case. That said, there is at least one place where the grammar does forbid you from doing something that would otherwise make be allowable: decorators. >>> @lookup[0] File "", line 1 @lookup[0] ^ SyntaxError: invalid syntax But this works: >>> new_decorator = lookup[0] >>> @new_decorator ... def f(): pass Thus, the idea of restricting the type of expression that can be used in particular circumstances is not without precedent, and should not be dismissed at face value. That is, unless we want to remove the restriction on decorators, which I'm okay with, too. I have occasionally wanted to do something more complicated with a decorator, and used the workaround above. Eric From rosuav at gmail.com Sun Jul 8 13:59:10 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 Jul 2018 03:59:10 +1000 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Mon, Jul 9, 2018 at 3:55 AM, Eric V. Smith wrote: > I agree with Chris in this case. That said, there is at least one place > where the grammar does forbid you from doing something that would otherwise > make be allowable: decorators. > >>>> @lookup[0] > File "", line 1 > @lookup[0] > ^ > SyntaxError: invalid syntax > > But this works: > >>>> new_decorator = lookup[0] >>>> @new_decorator > ... def f(): pass > > Thus, the idea of restricting the type of expression that can be used in > particular circumstances is not without precedent, and should not be > dismissed at face value. That is, unless we want to remove the restriction > on decorators, which I'm okay with, too. I have occasionally wanted to do > something more complicated with a decorator, and used the workaround above. > This is true. I wasn't around when decorator syntax was discussed; what were the reasons for this being the way it is? It isn't simply "'@' test". ChrisA From eric at trueblade.com Sun Jul 8 14:20:54 2018 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 8 Jul 2018 14:20:54 -0400 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: <09a430e5-342e-058c-1a84-c7dae0092b1b@trueblade.com> On 7/8/2018 1:59 PM, Chris Angelico wrote: > On Mon, Jul 9, 2018 at 3:55 AM, Eric V. Smith wrote: >> I agree with Chris in this case. That said, there is at least one place >> where the grammar does forbid you from doing something that would otherwise >> make be allowable: decorators. >> >>>>> @lookup[0] >> File "", line 1 >> @lookup[0] >> ^ >> SyntaxError: invalid syntax >> >> But this works: >> >>>>> new_decorator = lookup[0] >>>>> @new_decorator >> ... def f(): pass >> >> Thus, the idea of restricting the type of expression that can be used in >> particular circumstances is not without precedent, and should not be >> dismissed at face value. That is, unless we want to remove the restriction >> on decorators, which I'm okay with, too. I have occasionally wanted to do >> something more complicated with a decorator, and used the workaround above. >> > > This is true. I wasn't around when decorator syntax was discussed; > what were the reasons for this being the way it is? It isn't simply > "'@' test". I was around, but I don't recall the exact reasoning. Something along the lines of YAGNI, I believe. The first reference I found to it is https://mail.python.org/pipermail/python-ideas/2009-September/005623.html, although surely there are older ones, and even this email references an older super-confusing use of lambdas in decorators. In any event, I see no reason to restrict where assignment expressions can be used. Eric From josiah.carlson at gmail.com Sun Jul 8 14:26:24 2018 From: josiah.carlson at gmail.com (Josiah Carlson) Date: Sun, 8 Jul 2018 11:26:24 -0700 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: I'm sure that only 1 or 2 people cares about my opinion on this, but I will say that PEP 572 is taking one of my least favorite features of C/C++ and adding it to Python. About the only good thing I can say about it is that it might make some things more convenient to write. Worse to read, worse to debug, and definitely less clear unless you've got a comment to explain. At least in my case, the places I use them in C are places I feel *bad* about later. Partly because I know I'm usually trying to over-optimize, but partly because inline assignments are a big enough source of enough security vulnerabilities to make me question *any* language that decides to add them (especially 25+ years after the fact). Like... an entire async syntax and asyncio was modified in 3.6 (coroutine trampolines are tough without syntax, I get you, I did threads instead of dealing with them back in '05 because I couldn't get them quite right), but inline assignments just now? I'm at a loss. On the Python side, I wouldn't accept any patches in any of my projects to include := assignments, because those colons look like monitor grit, and/or a mistake (like someone wanted != but got := ), and means that my 2.6-3.6 compatible code is now 3.7+ only. I get that language evolution is a thing (my failures to try move *this* language in a direction I thought useful is part of why I stopped posting here), but hiding a colon in the middle of a line is not particularly explicit. And it definitely doesn't help for maintaining any of the software that I'm personally responsible for maintaining across the range of Python versions. I know you all don't care about me and my libraries, but at least consider the thousands of other libraries written and maintained by other folks. I mean, unless you all want 3.7+ - only libraries. To me, it seems like a lot of movement to support a feature that a lot of folks straight up don't want. But I don't have my finger on the pulse of Python or Python-dev, so maybe there are a bunch of folks coming from Go, Pascal, Smalltalk, etc., who are jonesin' for a hit of that :=, or have use-cases where this is a huge productivity improvement, and not a glaring security hole waiting to get them. I just don't know. But I also don't see a burning need; I'm 19 years in on this Python train, and I've never needed := before. Reading the PEP, the PR for changes to the Python standard library, email threads in support of the feature, etc., it just doesn't feel compelling. My $2, because that was a bit more than 2 cents worth of opinion, - Josiah On Sun, Jul 8, 2018 at 10:14 AM, Giampaolo Rodola' wrote: > > > On Sun, Jul 8, 2018 at 6:45 PM Steve Holden wrote: > >> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' >> wrote: >> >>> ?[...] >>> I find that (space between the parentheses of a function call >>> statement) too unnatural as a place where to put an assignment. It is >>> not even "guarded" by a keyword like "if" or "while" which can help as >>> indicators that an assignment may occur. Also, I think it's way too easy to >>> confuse it with a keyword argument: >>> >>> >>> foo(x = 1) # keyword arg >>> >>> foo(x := 1) # assignment + value passing >>> ?[...] >>> >> >> ?But the PEP 8 spellings are? >> >> foo(x=1) >> >> and >> >> f(x := 1). >> >> The extra spacing makes it obvious that this isn't a regular named >> argument. >> > > What if the author of the code I'm reading didn't respect PEP-8? I don't > think it's fair to invoke PEP-8 as a counter-measure to obviate a syntax > which can clearly be mistaken with something else simply by omitting 2 > spaces. Not to mention that I don't see why anyone would want to declare a > variable in there in the first place. > > -- > Giampaolo - http://grodola.blogspot.com > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > josiah.carlson%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From julien at palard.fr Sun Jul 8 14:23:53 2018 From: julien at palard.fr (Julien Palard) Date: Sun, 08 Jul 2018 14:23:53 -0400 Subject: [Python-Dev] [python-committers] Time for 3.4.9 and 3.5.6 In-Reply-To: References: Message-ID: Hi, [Larry] > 3.5 also got some doc-only changes related to the online "version switcher" dropdown. About this I have a question: the switchers for english version of 3.4 and 3.5 are disabled (https://docs.python.org/3.5/) but not disabled for translations (https://docs.python.org/fr/3.5/). I don't see any mention of dropping them in PEP 101, and I don't think it's a good thing (UX point of view). Should I re-enable version and language switchers on 3.5? I think so and I can do, just give me the go (or the argument/pointers on why it's disabled). Bests, ?--? Julien Palard https://mdk.fr? From nad at python.org Sun Jul 8 14:50:08 2018 From: nad at python.org (Ned Deily) Date: Sun, 8 Jul 2018 14:50:08 -0400 Subject: [Python-Dev] [python-committers] Time for 3.4.9 and 3.5.6 In-Reply-To: References: Message-ID: <30A628C5-3A92-4970-9E9F-86615B556963@python.org> On Jul 8, 2018, at 14:23, Julien Palard via Python-Dev wrote: > [Larry] >> 3.5 also got some doc-only changes related to the online "version switcher" dropdown. > > About this I have a question: the switchers for english version of 3.4 and 3.5 are disabled (https://docs.python.org/3.5/) but not disabled for translations (https://docs.python.org/fr/3.5/). I don't see any mention of dropping them in PEP 101, and I don't think it's a good thing (UX point of view). > > Should I re-enable version and language switchers on 3.5? I think so and I can do, just give me the go (or the argument/pointers on why it's disabled). I'm not Larry but I believe the reason that the switchers are missing on the on-line versions of 3.4 and 3.5 docs is that we release managers manually build and update the doc sets for release branches that are in security-fix-only mode (and that have been taken out of the automatic docs-build script) and we're not clever enough to know to build them with the switchers enabled. If we can document that in our release process, that would be cool. -- Ned Deily nad at python.org -- [] From mark at hotpy.org Sun Jul 8 17:13:42 2018 From: mark at hotpy.org (Mark Shannon) Date: Sun, 8 Jul 2018 22:13:42 +0100 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: <5B412C90.5000905@UGent.be> References: <5B412C90.5000905@UGent.be> Message-ID: <6b97d953-9fb0-b7ab-424c-be03e855285e@hotpy.org> On 07/07/18 22:11, Jeroen Demeyer wrote: > On 2018-07-07 15:38, Mark Shannon wrote: >> Hi, >> >> We seem to have a plethora of PEPs where we really ought to have one (or >> none?). > > - PEP 575 has been withdrawn. > - PEP 579 is an informational PEP with the bigger picture; it does > contain some of the requirements that you want to discuss here. > - PEP 580 and PEP 576 are two alternative implementations of a protocol > to optimize callables implemented in C. > >> 5. It should speed up CPython for the standard benchmark suite. > > I'd like to replace this by: must *not slow down* the standard benchmark > suite and preferable should not slow down anything. I've added you suggestion, and everyone else's, to this github repo: https://github.com/markshannon/extended-calling-convention Feel free to comment on github, submit PRs or just email me directly if you have anything else you want to add. Cheers, Mark. From g.rodola at gmail.com Sun Jul 8 17:27:10 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Sun, 8 Jul 2018 23:27:10 +0200 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Sun, Jul 8, 2018 at 7:24 PM Chris Angelico wrote: > > On Mon, Jul 9, 2018 at 3:14 AM, Giampaolo Rodola' wrote: > > > > > > On Sun, Jul 8, 2018 at 6:45 PM Steve Holden wrote: > >> > >> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' > >> wrote: > >>> > >>> [...] > >>> I find that (space between the parentheses of a function call statement) > >>> too unnatural as a place where to put an assignment. It is not even > >>> "guarded" by a keyword like "if" or "while" which can help as indicators > >>> that an assignment may occur. Also, I think it's way too easy to confuse it > >>> with a keyword argument: > >>> > >>> >>> foo(x = 1) # keyword arg > >>> >>> foo(x := 1) # assignment + value passing > >>> [...] > >> > >> > >> But the PEP 8 spellings are > >> > >> foo(x=1) > >> > >> and > >> > >> f(x := 1). > >> > >> The extra spacing makes it obvious that this isn't a regular named > >> argument. > > > > > > What if the author of the code I'm reading didn't respect PEP-8? I don't > > think it's fair to invoke PEP-8 as a counter-measure to obviate a syntax > > which can clearly be mistaken with something else simply by omitting 2 > > spaces. Not to mention that I don't see why anyone would want to declare a > > variable in there in the first place. > > > > It's not about why someone would want to assign inside a function > call. It's about why it should be forbidden. Perhaps nobody has a good > reason to use THlS_OBJECT as a variable name, and it's potentially > very confusing; but should the grammar of Python forbid it? No. > Because there is no value in forbidding it. I'll try to give some reasons on why I think it should be forbidden. In order of importance, more or less: 1) Because of readability and because it's ambiguous. foo(x := 1) and foo(x = 1) are visually too similar and mean 2 completely different things. And no, I don't think PEP-8 compliant variants are much better: >>> foo(x := 1) >>> foo(x=1) Good luck explaining the difference between the two to a beginner including the introduction of PEP-8 concepts and why spaces are so important in this case. The fact that spaces are more important here than in other places alone makes me think this is unlikely a good idea. 2) It's an incentive to write more compact code at the expense of readability. I see := being used in "if" and "while" statements as different in this regard. They imply an indented code block will follow and the variables being set in the "if/while" statement will be used right after that, supposedly in the very next line and *in that block only*. This is what https://github.com/python/cpython/pull/8122/files is all about, really. There is no block in this case hence AFAICT this syntax is only meant for writing one-liners. 3) := assignments in {if, while, yield, assert} statements are more clearly noticeable because delimited by such keywords whereas a function call can appear anywhere in the code. The only way to easily track assignments such as foo(x := 1) is via linting. 4) IMO this is the main bit of PEP-572 which can be subject to serious abuse. The value of "explicit is better than implicit" is an advanced concept. We can't expect it can be grasped by everybody. Many folks may be tempted to write one-liners at the top of a function and even feel good about it because a bunch of lines were saved. Potentially *a lot* of lines can be saved so it even more advanced users may be tempted to use it. After all the language allows it + "it's brand new syntax so it must be good". 5) It has no keyword argument correspondence. If foo(x := 1) is allowed then why this one is not? >>> foo(x=(x := 1)) (I don't think it should BTW: it's not pretty) 6) Differently from := usage in {if, while, comprehensions} no valid use case which would justify its usage has been provided so far. AFAICT the only valid use case is for writing one-liners. 7) Defining something in there just feels wrong (to me at least) 8) I'm pretty sure any linter would emit a warning by default so why bother adding support for a brand new syntax which we already know it would be discouraged anyway? 9) It goes against half of the Python Zen. -- Giampaolo - http://grodola.blogspot.com From rosuav at gmail.com Sun Jul 8 17:32:58 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 9 Jul 2018 07:32:58 +1000 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Mon, Jul 9, 2018 at 7:27 AM, Giampaolo Rodola' wrote: > 5) It has no keyword argument correspondence. If foo(x := 1) is > allowed then why this one is not? > >>> foo(x=(x := 1)) > (I don't think it should BTW: it's not pretty) Actually it is. Nothing wrong with that. It assigns to 'x' in the local scope, and also passes that as a keyword parameter named 'x'. ChrisA From guido at python.org Sun Jul 8 18:06:41 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 8 Jul 2018 15:06:41 -0700 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: Since you CC'ed me explicitly I feel compelled to respond. I have read your reasoning, and I simply don't agree with it. A few months ago I would have happily explained why (there is a reason) but given the endless debate we've already seen I am simply too tired for another long email. Please give me the benefit of the doubt. The sky is not falling. On Sun, Jul 8, 2018 at 2:27 PM Giampaolo Rodola' wrote: > On Sun, Jul 8, 2018 at 7:24 PM Chris Angelico wrote: > > > > On Mon, Jul 9, 2018 at 3:14 AM, Giampaolo Rodola' > wrote: > > > > > > > > > On Sun, Jul 8, 2018 at 6:45 PM Steve Holden > wrote: > > >> > > >> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' < > g.rodola at gmail.com> > > >> wrote: > > >>> > > >>> [...] > > >>> I find that (space between the parentheses of a function call > statement) > > >>> too unnatural as a place where to put an assignment. It is not even > > >>> "guarded" by a keyword like "if" or "while" which can help as > indicators > > >>> that an assignment may occur. Also, I think it's way too easy to > confuse it > > >>> with a keyword argument: > > >>> > > >>> >>> foo(x = 1) # keyword arg > > >>> >>> foo(x := 1) # assignment + value passing > > >>> [...] > > >> > > >> > > >> But the PEP 8 spellings are > > >> > > >> foo(x=1) > > >> > > >> and > > >> > > >> f(x := 1). > > >> > > >> The extra spacing makes it obvious that this isn't a regular named > > >> argument. > > > > > > > > > What if the author of the code I'm reading didn't respect PEP-8? I > don't > > > think it's fair to invoke PEP-8 as a counter-measure to obviate a > syntax > > > which can clearly be mistaken with something else simply by omitting 2 > > > spaces. Not to mention that I don't see why anyone would want to > declare a > > > variable in there in the first place. > > > > > > > It's not about why someone would want to assign inside a function > > call. It's about why it should be forbidden. Perhaps nobody has a good > > reason to use THlS_OBJECT as a variable name, and it's potentially > > very confusing; but should the grammar of Python forbid it? No. > > Because there is no value in forbidding it. > > I'll try to give some reasons on why I think it should be forbidden. > In order of importance, more or less: > > 1) Because of readability and because it's ambiguous. foo(x := 1) and > foo(x = 1) are visually too similar and mean 2 completely different > things. And no, I don't think PEP-8 compliant variants are much > better: > >>> foo(x := 1) > >>> foo(x=1) > Good luck explaining the difference between the two to a beginner > including the introduction of PEP-8 concepts and why spaces are so > important in this case. The fact that spaces are more important here > than in other places alone makes me think this is unlikely a good > idea. > > 2) It's an incentive to write more compact code at the expense of > readability. I see := being used in "if" and "while" statements as > different in this regard. They imply an indented code block will > follow and the variables being set in the "if/while" statement will be > used right after that, supposedly in the very next line and *in that > block only*. This is what > https://github.com/python/cpython/pull/8122/files is all about, > really. There is no block in this case hence AFAICT this syntax is > only meant for writing one-liners. > > 3) := assignments in {if, while, yield, assert} statements are more > clearly noticeable because delimited by such keywords whereas a > function call can appear anywhere in the code. The only way to easily > track assignments such as foo(x := 1) is via linting. > > 4) IMO this is the main bit of PEP-572 which can be subject to serious > abuse. The value of "explicit is better than implicit" is an advanced > concept. We can't expect it can be grasped by everybody. Many folks > may be tempted to write one-liners at the top of a function and even > feel good about it because a bunch of lines were saved. Potentially *a > lot* of lines can be saved so it even more advanced users may be > tempted to use it. After all the language allows it + "it's brand new > syntax so it must be good". > > 5) It has no keyword argument correspondence. If foo(x := 1) is > allowed then why this one is not? > >>> foo(x=(x := 1)) > (I don't think it should BTW: it's not pretty) > > 6) Differently from := usage in {if, while, comprehensions} no valid > use case which would justify its usage has been provided so far. > AFAICT the only valid use case is for writing one-liners. > > 7) Defining something in there just feels wrong (to me at least) > > 8) I'm pretty sure any linter would emit a warning by default so why > bother adding support for a brand new syntax which we already know it > would be discouraged anyway? > > 9) It goes against half of the Python Zen. > > -- > Giampaolo - http://grodola.blogspot.com > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Jul 8 18:32:16 2018 From: larry at hastings.org (Larry Hastings) Date: Sun, 8 Jul 2018 15:32:16 -0700 Subject: [Python-Dev] Time for 3.4.9 and 3.5.6 In-Reply-To: References: Message-ID: <2342a468-ee46-7b09-2522-d665f67850b0@hastings.org> On 07/08/2018 10:05 AM, Ivan Pozdeev via Python-Dev wrote: > > I'll use this opportunity to remind you that 3.4 build is broken -- it > can't be built from start to installer with the instructions given > because of outside factors (CPython has migrated from Hg to Git). > https://bugs.python.org/issue31623 about this was ignored (see > https://bugs.python.org/issue31623#msg303708 for supplemental fixes). > > If this isn't something considered needing a fix, the claim that 3.4 > is supported in any shape and form is but a pretense -- if something > can't be built, it can't be used. > By "3.4 build is broken", you mean that building the installer is broken on Windows.? Sadly the maintainer of that installer is no longer part of the Python community, and as a Linux-only dev I have no way of testing any proposed change. More importantly, 3.4 is in security-fixes-only mode, which means that changes that aren't security fixes won't be accepted.? Fixing this would not be a security fix.? So even if the patch was clean and well-reviewed and worked perfectly I'm simply not going to merge it into 3.4.? The 3.4 tree is only going to be in security-fixes mode for another eight months anyway, after which I will retire as 3.4 release manager, and 3.4 will no longer be supported by the Python core development community at all. As pointed out in that bpo issue: if the problem is entirely due to switching from "git" to "hg", then you should have very little difficulty working around that.? You can use a git-to-hg bridge, or create a local-only hg repo from the 3.4 tree.? That should permit you to build your own installers.? I'm a little sad that the 3.4 Windows installers no longer build directly out-of-tree without such a workaround, but sometimes that's just what happens with a Python release three major releases out of date languishing in security-fixes-only mode. //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Sun Jul 8 18:41:27 2018 From: larry at hastings.org (Larry Hastings) Date: Sun, 8 Jul 2018 15:41:27 -0700 Subject: [Python-Dev] [python-committers] Time for 3.4.9 and 3.5.6 In-Reply-To: <30A628C5-3A92-4970-9E9F-86615B556963@python.org> References: <30A628C5-3A92-4970-9E9F-86615B556963@python.org> Message-ID: <59593e7f-07b6-051a-6dcb-6a67918809f6@hastings.org> On 07/08/2018 11:50 AM, Ned Deily wrote: > On Jul 8, 2018, at 14:23, Julien Palard via Python-Dev wrote: >> [Larry] >>> 3.5 also got some doc-only changes related to the online "version switcher" dropdown. >> About this I have a question: the switchers for english version of 3.4 and 3.5 are disabled (https://docs.python.org/3.5/) but not disabled for translations (https://docs.python.org/fr/3.5/). I don't see any mention of dropping them in PEP 101, and I don't think it's a good thing (UX point of view). >> >> Should I re-enable version and language switchers on 3.5? I think so and I can do, just give me the go (or the argument/pointers on why it's disabled). > I'm not Larry but I believe the reason that the switchers are missing on the on-line versions of 3.4 and 3.5 docs is that we release managers manually build and update the doc sets for release branches that are in security-fix-only mode (and that have been taken out of the automatic docs-build script) and we're not clever enough to know to build them with the switchers enabled. If we can document that in our release process, that would be cool. Yes, exactly!? The place to document the process would be PEP 101. (Or, if you wanted to volunteer to handle building and deploying the online docs /for/ the RMs, that works too!) I know there's an automated system that rebuilds the docs on a regular (daily? hourly?) basis for certain versions.? I suspect the RMs all have login credentials for the machine where that happens. If you could make it so we could manually kick off a doc rebuild for a specific version (and tell us how to do it) that would be perfect! //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From g.rodola at gmail.com Sun Jul 8 19:03:02 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Mon, 9 Jul 2018 01:03:02 +0200 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: Fair enough. I will. Sorry for the extra pressure at this particular stage. On Mon, 9 Jul 2018 at 00:06, Guido van Rossum wrote: > Since you CC'ed me explicitly I feel compelled to respond. I have read > your reasoning, and I simply don't agree with it. A few months ago I would > have happily explained why (there is a reason) but given the endless debate > we've already seen I am simply too tired for another long email. Please > give me the benefit of the doubt. The sky is not falling. > > On Sun, Jul 8, 2018 at 2:27 PM Giampaolo Rodola' > wrote: > >> On Sun, Jul 8, 2018 at 7:24 PM Chris Angelico wrote: >> > >> > On Mon, Jul 9, 2018 at 3:14 AM, Giampaolo Rodola' >> wrote: >> > > >> > > >> > > On Sun, Jul 8, 2018 at 6:45 PM Steve Holden >> wrote: >> > >> >> > >> On Sun, Jul 8, 2018 at 10:41 AM, Giampaolo Rodola' < >> g.rodola at gmail.com> >> > >> wrote: >> > >>> >> > >>> [...] >> > >>> I find that (space between the parentheses of a function call >> statement) >> > >>> too unnatural as a place where to put an assignment. It is not even >> > >>> "guarded" by a keyword like "if" or "while" which can help as >> indicators >> > >>> that an assignment may occur. Also, I think it's way too easy to >> confuse it >> > >>> with a keyword argument: >> > >>> >> > >>> >>> foo(x = 1) # keyword arg >> > >>> >>> foo(x := 1) # assignment + value passing >> > >>> [...] >> > >> >> > >> >> > >> But the PEP 8 spellings are >> > >> >> > >> foo(x=1) >> > >> >> > >> and >> > >> >> > >> f(x := 1). >> > >> >> > >> The extra spacing makes it obvious that this isn't a regular named >> > >> argument. >> > > >> > > >> > > What if the author of the code I'm reading didn't respect PEP-8? I >> don't >> > > think it's fair to invoke PEP-8 as a counter-measure to obviate a >> syntax >> > > which can clearly be mistaken with something else simply by omitting 2 >> > > spaces. Not to mention that I don't see why anyone would want to >> declare a >> > > variable in there in the first place. >> > > >> > >> > It's not about why someone would want to assign inside a function >> > call. It's about why it should be forbidden. Perhaps nobody has a good >> > reason to use THlS_OBJECT as a variable name, and it's potentially >> > very confusing; but should the grammar of Python forbid it? No. >> > Because there is no value in forbidding it. >> >> I'll try to give some reasons on why I think it should be forbidden. >> In order of importance, more or less: >> >> 1) Because of readability and because it's ambiguous. foo(x := 1) and >> foo(x = 1) are visually too similar and mean 2 completely different >> things. And no, I don't think PEP-8 compliant variants are much >> better: >> >>> foo(x := 1) >> >>> foo(x=1) >> Good luck explaining the difference between the two to a beginner >> including the introduction of PEP-8 concepts and why spaces are so >> important in this case. The fact that spaces are more important here >> than in other places alone makes me think this is unlikely a good >> idea. >> >> 2) It's an incentive to write more compact code at the expense of >> readability. I see := being used in "if" and "while" statements as >> different in this regard. They imply an indented code block will >> follow and the variables being set in the "if/while" statement will be >> used right after that, supposedly in the very next line and *in that >> block only*. This is what >> https://github.com/python/cpython/pull/8122/files is all about, >> really. There is no block in this case hence AFAICT this syntax is >> only meant for writing one-liners. >> >> 3) := assignments in {if, while, yield, assert} statements are more >> clearly noticeable because delimited by such keywords whereas a >> function call can appear anywhere in the code. The only way to easily >> track assignments such as foo(x := 1) is via linting. >> >> 4) IMO this is the main bit of PEP-572 which can be subject to serious >> abuse. The value of "explicit is better than implicit" is an advanced >> concept. We can't expect it can be grasped by everybody. Many folks >> may be tempted to write one-liners at the top of a function and even >> feel good about it because a bunch of lines were saved. Potentially *a >> lot* of lines can be saved so it even more advanced users may be >> tempted to use it. After all the language allows it + "it's brand new >> syntax so it must be good". >> >> 5) It has no keyword argument correspondence. If foo(x := 1) is >> allowed then why this one is not? >> >>> foo(x=(x := 1)) >> (I don't think it should BTW: it's not pretty) >> >> 6) Differently from := usage in {if, while, comprehensions} no valid >> use case which would justify its usage has been provided so far. >> AFAICT the only valid use case is for writing one-liners. >> >> 7) Defining something in there just feels wrong (to me at least) >> >> 8) I'm pretty sure any linter would emit a warning by default so why >> bother adding support for a brand new syntax which we already know it >> would be discouraged anyway? >> >> 9) It goes against half of the Python Zen. >> >> -- >> Giampaolo - http://grodola.blogspot.com >> > > > -- > --Guido van Rossum (python.org/~guido) > -- Giampaolo - http://grodola.blogspot.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Sun Jul 8 20:10:02 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Mon, 9 Jul 2018 03:10:02 +0300 Subject: [Python-Dev] Time for 3.4.9 and 3.5.6 In-Reply-To: <2342a468-ee46-7b09-2522-d665f67850b0@hastings.org> References: <2342a468-ee46-7b09-2522-d665f67850b0@hastings.org> Message-ID: On 09.07.2018 1:32, Larry Hastings wrote: > On 07/08/2018 10:05 AM, Ivan Pozdeev via Python-Dev wrote: >> >> I'll use this opportunity to remind you that 3.4 build is broken -- >> it can't be built from start to installer with the instructions given >> because of outside factors (CPython has migrated from Hg to Git). >> https://bugs.python.org/issue31623 about this was ignored (see >> https://bugs.python.org/issue31623#msg303708 for supplemental fixes). >> >> If this isn't something considered needing a fix, the claim that 3.4 >> is supported in any shape and form is but a pretense -- if something >> can't be built, it can't be used. >> > > By "3.4 build is broken", you mean that building the installer is > broken on Windows.? Sadly the maintainer of that installer is no > longer part of the Python community, and as a Linux-only dev I have no > way of testing any proposed change. > Not only that, building the binaries is also broken as per https://bugs.python.org/issue31645 (that's one of the aforementioned "supplemental fixes"). > More importantly, 3.4 is in security-fixes-only mode, which means that > changes that aren't security fixes won't be accepted.? Fixing this > would not be a security fix.? So even if the patch was clean and > well-reviewed and worked perfectly I'm simply not going to merge it > into 3.4.? The 3.4 tree is only going to be in security-fixes mode for > another eight months anyway, after which I will retire as 3.4 release > manager, and 3.4 will no longer be supported by the Python core > development community at all. I kinda don't see a point of claiming any kind of support and doing any work if the codebase is unusable. All that achieves is confused users and wasted time for everyone involved. If you "a Linux-only dev" and no-one is going to look at the Windows part, why not just say clearly that this version line is not supported outside Linux? I'm okay with that (what is and isn't supported is none of my business). At least, there won't be a nasty surprise when I rely on the team's claim that the code is workable, and it actually isn't -- and another one when I go for the trouble to provide a fix, and is told that I'm a troublemaker and has just massively wasted my and everybody else's time as a thanks. Besides, that'll be a reason to officially close all still-open tickets for 3.4/3.5 (there are about 2000 that are mentioning them) regardless of the topic (I've checked that none are currently marked as security issues). > As pointed out in that bpo issue: if the problem is entirely due to > switching from "git" to "hg", then you should have very little > difficulty working around that.? You can use a git-to-hg bridge, or > create a local-only hg repo from the 3.4 tree.? That should permit you > to build your own installers.? I'm a little sad that the 3.4 Windows > installers no longer build directly out-of-tree without such a > workaround, but sometimes that's just what happens with a Python > release three major releases out of date languishing in > security-fixes-only mode. > > > //arry/ > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sun Jul 8 20:35:24 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sun, 8 Jul 2018 20:35:24 -0400 Subject: [Python-Dev] Time for 3.4.9 and 3.5.6 In-Reply-To: References: Message-ID: On 7/8/2018 1:05 PM, Ivan Pozdeev via Python-Dev wrote: > I'll use this opportunity to remind you that 3.4 build is broken -- it > can't be built from start to installer with the instructions given > because of outside factors (CPython has migrated from Hg to Git). > https://bugs.python.org/issue31623 about this was ignored (see > https://bugs.python.org/issue31623#msg303708 for supplemental fixes). > > If this isn't something considered needing a fix, the claim that 3.4 is > supported in any shape and form is but a pretense Another wild exaggeration that inhibits me, and I suspect others, from attending to your legitimate issue. > -- if something can't be built, it can't be used. but 3.4 source security releases can be built and used on *nix. What is true is that we do not currently support building new releases on XP. We never did for 3.5, and can no longer test for 3.4. Partly as a consequence, we are not currently supporting (updating scripts for) building 3.4 on Windows. But Windows is not all systems. > On 08.07.2018 10:45, Larry Hastings wrote: >> >> My six-month cadence means it's time for the next releases of 3.4 and >> 3.5.? There haven't been many changes since the last releases--two, to >> be exact.? These two security fixes were backported to both 3.4 and 3.5: >> >> * bpo-32981: Fix catastrophic backtracking vulns (GH-5955) >> * bpo-33001: Prevent buffer overrun in os.symlink (GH-5989) >> >> 3.5 also got some doc-only changes related to the online "version >> switcher" dropdown.? (They weren't backported to 3.4 because we don't >> list 3.4 in the version switcher dropdown anymore.) >> >> There are currently no PRs open for either 3.4 or 3.5, I verified that https://bugs.python.org/issue31623 is open and marked for 3.4 and has been so since last September. Unless you think there is plausible chance that it might be applied before the end, I think you should reject and close it now. That said, searching for open 3.4 issues returns 1617 items, almost none of which are even possibly applicable. You cannot even begin to wade thru and fix the headers. Adding type 'security' gives 8 hits, none of which are the 2 above. 4 have patches attached, which need to be turned into PRs to proceed. You might look at these 4. >> and they also >> have no open "release blocker" or "deferred blocker" bugs. >> It seems >> things are pretty quiet in our two security-fixes-only branches--a >> good way to be! >> >> I therefore propose to cut the RCs in a week and a half, and the >> finals two weeks later.? So: >> >> Wednesday? July 18 2018 - 3.4.9rc1 and 3.5.6rc1 >> Wednesday August 1 2018 - 3.4.9 final and 3.5.6 final I presume that this will be the last before the wrap-up next March. -- Terry Jan Reedy From eric at trueblade.com Sun Jul 8 21:29:48 2018 From: eric at trueblade.com (Eric V. Smith) Date: Sun, 8 Jul 2018 21:29:48 -0400 Subject: [Python-Dev] Time for 3.4.9 and 3.5.6 In-Reply-To: References: Message-ID: <5f42ad7f-d018-8b49-24c2-50fa91e68b81@trueblade.com> On 7/8/2018 8:35 PM, Terry Reedy wrote: > On 7/8/2018 1:05 PM, Ivan Pozdeev via Python-Dev wrote: >> I'll use this opportunity to remind you that 3.4 build is broken -- it >> can't be built from start to installer with the instructions given >> because of outside factors (CPython has migrated from Hg to Git). >> https://bugs.python.org/issue31623 about this was ignored (see >> https://bugs.python.org/issue31623#msg303708 for supplemental fixes). >> >> If this isn't something considered needing a fix, the claim that 3.4 >> is supported in any shape and form is but a pretense > > Another wild exaggeration that inhibits me, and I suspect others, from > attending to your legitimate issue. Yes, thanks for writing this, Terry. Given Ivan's previous behavior on his "Drop/deprecate Tkinter?" thread, and combined with this thread, I'm unlikely to spend my free time on his particular issue here. Eric From brett at python.org Sun Jul 8 22:11:47 2018 From: brett at python.org (Brett Cannon) Date: Sun, 8 Jul 2018 19:11:47 -0700 Subject: [Python-Dev] Time for 3.4.9 and 3.5.6 In-Reply-To: <5f42ad7f-d018-8b49-24c2-50fa91e68b81@trueblade.com> References: <5f42ad7f-d018-8b49-24c2-50fa91e68b81@trueblade.com> Message-ID: On Sun, Jul 8, 2018, 18:30 Eric V. Smith, wrote: > On 7/8/2018 8:35 PM, Terry Reedy wrote: > > On 7/8/2018 1:05 PM, Ivan Pozdeev via Python-Dev wrote: > >> I'll use this opportunity to remind you that 3.4 build is broken -- it > >> can't be built from start to installer with the instructions given > >> because of outside factors (CPython has migrated from Hg to Git). > >> https://bugs.python.org/issue31623 about this was ignored (see > >> https://bugs.python.org/issue31623#msg303708 for supplemental fixes). > >> > >> If this isn't something considered needing a fix, the claim that 3.4 > >> is supported in any shape and form is but a pretense > > > > Another wild exaggeration that inhibits me, and I suspect others, from > > attending to your legitimate issue. > > Yes, thanks for writing this, Terry. Given Ivan's previous behavior on > his "Drop/deprecate Tkinter?" thread, and combined with this thread, I'm > unlikely to spend my free time on his particular issue here. > Ditto for this specific issue and in general. People forget that we are doing all of this as a kindness for the community since most of us probably don't benefit from another 3.4 release, so any negativity is at best treated with indifference and at worst as de-motivating to any effort into open source (I know for me I'm definitely no longer in the mood to spend my free time on open source today if this is how people are going to treat my hard, volunteer work). -Brett > Eric > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Mon Jul 9 01:41:47 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 09 Jul 2018 17:41:47 +1200 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: <5B42F59B.4050507@canterbury.ac.nz> Eric V. Smith wrote: > there is at least one place > where the grammar does forbid you from doing something that would > otherwise make be allowable: decorators. And that was a controversial issue at the time. I don't remember there being much of an objective argument for the restriction -- it was more or less a matter of "Guido wanted it that way". -- Greg From tim.peters at gmail.com Mon Jul 9 01:54:48 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 9 Jul 2018 00:54:48 -0500 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: <5B42F59B.4050507@canterbury.ac.nz> References: <5B42F59B.4050507@canterbury.ac.nz> Message-ID: [Eric V. Smith] > > there is at least one place > where the grammar does forbid you from doing something that would > > otherwise make be allowable: decorators. > [Greg Ewing] > And that was a controversial issue at the time. I don't remember there being much of an objective argument for the restriction -- > it was more or less a matter of "Guido wanted it that way". > Start here: https://mail.python.org/pipermail/python-dev/2004-August/046711.html My favorite in that thread was Michael Hudson vigorously arguing what a sad loss it would be if Python hadn't allowed class C(random.choice([dict, list])): pass ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Mon Jul 9 02:59:20 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 9 Jul 2018 08:59:20 +0200 Subject: [Python-Dev] PEP 575, 576, 579 and 580 In-Reply-To: <44c3472d1c534a688bb96ec341281370@xmail103.UGent.be> References: <5B412C90.5000905@UGent.be> <44c3472d1c534a688bb96ec341281370@xmail103.UGent.be> Message-ID: <5B4307C8.2020906@UGent.be> On 2018-07-08 23:13, Mark Shannon wrote: > I've added you suggestion, and everyone else's, to this github repo: > https://github.com/markshannon/extended-calling-convention > > Feel free to comment on github, submit PRs or just email me directly if > you have anything else you want to add. Do you agree to add this to PEP 579 instead? From mcepl at cepl.eu Mon Jul 9 03:26:33 2018 From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Mon, 09 Jul 2018 09:26:33 +0200 Subject: [Python-Dev] Call for prudence about PEP-572 References: Message-ID: On 2018-07-07, 15:48 GMT, Guido van Rossum wrote: > if validate(name := re.search(pattern, line).group(1)): > return name Except there is no error handling for situations when re.search() returns None, so one shouldn't use it anyway (most of the time). Which seems to me like another nice example why one should stay away from this style as much as possible. I am too lazy to be tempted into this nice-example-terrible-production-code world. Best, Mat?j -- https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 They that can give up essential liberty to obtain a little temporary safety deserve neither liberty nor safety. -- Benjamin Franklin, Historical Review of Pennsylvania, 1759. From berker.peksag at gmail.com Mon Jul 9 05:24:47 2018 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Mon, 9 Jul 2018 12:24:47 +0300 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: <5B42F59B.4050507@canterbury.ac.nz> Message-ID: On Mon, Jul 9, 2018 at 8:54 AM, Tim Peters wrote: > [Eric V. Smith] >> >> > there is at least one place >> >> > where the grammar does forbid you from doing something that would >> > otherwise make be allowable: decorators. > > > [Greg Ewing] >> >> And that was a controversial issue at the time. I don't remember >> >> there being much of an objective argument for the restriction -- >> it was more or less a matter of "Guido wanted it that way". > > > Start here: > > https://mail.python.org/pipermail/python-dev/2004-August/046711.html There is also an open issue about removing that restriction: https://bugs.python.org/issue19660 https://bugs.python.org/file32745/decorator-syntax.patch removes a test that raises SyntaxError for the following snippet: @x[3] def foo(): pass --Berker From mcepl at cepl.eu Mon Jul 9 07:45:50 2018 From: mcepl at cepl.eu (=?UTF-8?Q?Mat=C4=9Bj?= Cepl) Date: Mon, 09 Jul 2018 13:45:50 +0200 Subject: [Python-Dev] Time for 3.4.9 and 3.5.6 References: <2342a468-ee46-7b09-2522-d665f67850b0@hastings.org> Message-ID: On 2018-07-08, 22:32 GMT, Larry Hastings wrote: > More importantly, 3.4 is in security-fixes-only mode, which > means that changes that aren't security fixes won't be > accepted. So, why isn?t https://bugs.python.org/issue31623 closed as WONTFIX (or whatever is the equivalent in b.p.o)? If we don't close our bugs, we surely will drown in them even more. Best, Mat?j -- https://matej.ceplovi.cz/blog/, Jabber: mcepl at ceplovi.cz GPG Finger: 3C76 A027 CA45 AD70 98B5 BC1D 7920 5802 880B C9D8 Give your heartache to him. (1Pt 5,7; Mt 11:28-30) From cosimo at anthrotype.com Mon Jul 9 11:03:27 2018 From: cosimo at anthrotype.com (Cosimo Lupo) Date: Mon, 9 Jul 2018 16:03:27 +0100 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? Message-ID: If one goes to httWhps://www.python.org/downloads from a Windows browser, the default download URL is for the 32-bit installer instead of the 64-bit one. I wonder why is this still the case? Shouldn't we encourage new Windows users (who may not even know the distinction between the two architectures) to use the 64-bit version of Python, since most likely they can? If this is not the correct forum for this, please let me know where I can direct my question/feature request, thanks. Cosimo -- Cosimo Lupo -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve.dower at python.org Mon Jul 9 12:01:00 2018 From: steve.dower at python.org (Steve Dower) Date: Mon, 9 Jul 2018 09:01:00 -0700 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: References: Message-ID: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> On 09Jul2018 0803, Cosimo Lupo wrote: > If one goes to httWhps://www.python.org/downloads > ?from a Windows browser, the default > download URL is for the 32-bit installer instead of the 64-bit one. > I wonder why is this still the case? > Shouldn't we encourage new Windows users (who may not even know the > distinction between the two architectures) to use the 64-bit version of > Python, since most likely they can? The difficulty is that they *definitely* can use the 32-bit version, and those few who are on older machines or older installs of Windows may not understand why the link we provide didn't work for them. From the various telemetry I've seen (and I work at Microsoft, so I have better access than most :) ), there is still enough 32-bit Windows out there that I'm not confident enough with "most likely". I haven't checked any location data (not even sure if we've got it), but I'd guess that there's higher 32-bit usage among less privileged countries and people. I've thought a bit about making a single installer that can offer the option of 32-bit/64-bit at install time, but I don't actually think it's that big a problem to deserve that much effort as a solution. Perhaps we should add non-button text below the button saying "Get the 64-bit version"? Cheers, Steve From paddy3118 at gmail.com Mon Jul 9 12:20:44 2018 From: paddy3118 at gmail.com (Paddy McCarthy) Date: Mon, 9 Jul 2018 17:20:44 +0100 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Sat, 7 Jul 2018 at 16:50, Guido van Rossum wrote: > This seems more suitable for a style guide. > Which would be good enough, if the automatic checkers were improved to flag such poor use of ':='.. Would that be possible? -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Jul 9 12:22:12 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 9 Jul 2018 18:22:12 +0200 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> Message-ID: <20180709182212.138466e7@fsol> On Mon, 9 Jul 2018 09:01:00 -0700 Steve Dower wrote: > On 09Jul2018 0803, Cosimo Lupo wrote: > > If one goes to httWhps://www.python.org/downloads > > ?from a Windows browser, the default > > download URL is for the 32-bit installer instead of the 64-bit one. > > I wonder why is this still the case? > > Shouldn't we encourage new Windows users (who may not even know the > > distinction between the two architectures) to use the 64-bit version of > > Python, since most likely they can? > > The difficulty is that they *definitely* can use the 32-bit version, and > those few who are on older machines or older installs of Windows may not > understand why the link we provide didn't work for them. > > From the various telemetry I've seen (and I work at Microsoft, so I > have better access than most :) ), there is still enough 32-bit Windows > out there that I'm not confident enough with "most likely". I haven't > checked any location data (not even sure if we've got it), but I'd guess > that there's higher 32-bit usage among less privileged countries and people. > > I've thought a bit about making a single installer that can offer the > option of 32-bit/64-bit at install time, but I don't actually think it's > that big a problem to deserve that much effort as a solution. > > Perhaps we should add non-button text below the button saying "Get the > 64-bit version"? Or perhaps the 32-bit installer could detect a 64-bit system and add an info box at the beginning? Regards Antoine. From rosuav at gmail.com Mon Jul 9 12:25:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 10 Jul 2018 02:25:18 +1000 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: On Tue, Jul 10, 2018 at 2:20 AM, Paddy McCarthy wrote: > > > On Sat, 7 Jul 2018 at 16:50, Guido van Rossum wrote: >> >> This seems more suitable for a style guide. > > > Which would be good enough, if the automatic checkers were improved to flag > such poor use of ':='.. > > Would that be possible? > I would recommend letting it be for a while, and seeing what abuses actually happen. Then linters can be coded to flag the abuses. ChrisA From p.f.moore at gmail.com Mon Jul 9 12:27:04 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 9 Jul 2018 17:27:04 +0100 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> Message-ID: On 9 July 2018 at 17:01, Steve Dower wrote: > On 09Jul2018 0803, Cosimo Lupo wrote: >> >> If one goes to httWhps://www.python.org/downloads >> from a Windows browser, the default >> download URL is for the 32-bit installer instead of the 64-bit one. >> I wonder why is this still the case? >> Shouldn't we encourage new Windows users (who may not even know the >> distinction between the two architectures) to use the 64-bit version of >> Python, since most likely they can? I agree, I'd rather see the 64-bit version be more accessible. > The difficulty is that they *definitely* can use the 32-bit version, and > those few who are on older machines or older installs of Windows may not > understand why the link we provide didn't work for them. > > From the various telemetry I've seen (and I work at Microsoft, so I have > better access than most :) ), there is still enough 32-bit Windows out there > that I'm not confident enough with "most likely". I haven't checked any > location data (not even sure if we've got it), but I'd guess that there's > higher 32-bit usage among less privileged countries and people. I'm happy to go with your research on whether the 32-bit version should be the obvious choice, but I don't think that means we have to hide the 64-bit version as much as we do. > I've thought a bit about making a single installer that can offer the option > of 32-bit/64-bit at install time, but I don't actually think it's that big a > problem to deserve that much effort as a solution. I agree, having 2 versions of the installer seems fine to me. > Perhaps we should add non-button text below the button saying "Get the > 64-bit version"? Why not just have a second button, "Download Python 3.7.0 (64-bit)" alongside or below the "Download Python 3.7.0" button? People who don't know the difference will just ignore it, people who do will be able to choose straight from the main download page. Paul From vano at mail.mipt.ru Mon Jul 9 12:27:24 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Mon, 9 Jul 2018 19:27:24 +0300 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> Message-ID: <791dbaf9-11ab-3406-1afb-e6c7b86eca6f@mail.mipt.ru> On 09.07.2018 19:01, Steve Dower wrote: > On 09Jul2018 0803, Cosimo Lupo wrote: >> If one goes to httWhps://www.python.org/downloads >> ?from a Windows browser, the >> default download URL is for the 32-bit installer instead of the >> 64-bit one. >> I wonder why is this still the case? >> Shouldn't we encourage new Windows users (who may not even know the >> distinction between the two architectures) to use the 64-bit version >> of Python, since most likely they can? > > The difficulty is that they *definitely* can use the 32-bit version, > and those few who are on older machines or older installs of Windows > may not understand why the link we provide didn't work for them. > > From the various telemetry I've seen (and I work at Microsoft, so I > have better access than most :) ), there is still enough 32-bit > Windows out there that I'm not confident enough with "most likely". I > haven't checked any location data (not even sure if we've got it), but > I'd guess that there's higher 32-bit usage among less privileged > countries and people. > > I've thought a bit about making a single installer that can offer the > option of 32-bit/64-bit at install time, but I don't actually think > it's that big a problem to deserve that much effort as a solution. > > Perhaps we should add non-button text below the button saying "Get the > 64-bit version"? > Maybe infer the bitness from User-Agent instead. This seems to be the trend among official sites in general. > Cheers, > Steve > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru From steve.dower at python.org Mon Jul 9 13:18:24 2018 From: steve.dower at python.org (Steve Dower) Date: Mon, 9 Jul 2018 10:18:24 -0700 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: <791dbaf9-11ab-3406-1afb-e6c7b86eca6f@mail.mipt.ru> References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> <791dbaf9-11ab-3406-1afb-e6c7b86eca6f@mail.mipt.ru> Message-ID: <5a44c4c6-7d9c-f0a9-412c-480ccf815816@python.org> On 09Jul2018 0927, Ivan Pozdeev via Python-Dev wrote: > On 09.07.2018 19:01, Steve Dower wrote: >> Perhaps we should add non-button text below the button saying "Get the >> 64-bit version"? >> > > Maybe infer the bitness from User-Agent instead. This seems to be the > trend among official sites in general. I'm not into creating that much work for someone else, since I'm certainly not going to be the one making this change. Adding static text or a second button (as Paul suggested) is fine by me. But I don't want to start receiving bugs (or more likely, I don't want webmaster to start receiving emails) saying that the download is broken because someone who didn't think got the incompatible version. Cheers, Steve From steve.dower at python.org Mon Jul 9 13:20:53 2018 From: steve.dower at python.org (Steve Dower) Date: Mon, 9 Jul 2018 10:20:53 -0700 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: <20180709182212.138466e7@fsol> References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> <20180709182212.138466e7@fsol> Message-ID: On 09Jul2018 0922, Antoine Pitrou wrote: > On Mon, 9 Jul 2018 09:01:00 -0700 > Steve Dower wrote: >> I've thought a bit about making a single installer that can offer the >> option of 32-bit/64-bit at install time, but I don't actually think it's >> that big a problem to deserve that much effort as a solution. >> >> Perhaps we should add non-button text below the button saying "Get the >> 64-bit version"? > > Or perhaps the 32-bit installer could detect a 64-bit system and add > an info box at the beginning? That's not a bad idea. Needs a bpo issue, but shouldn't be too hard to do. And it should be safe to backport for 3.7.1, as there's no real behaviour change. Cheers, Steve From chris.barker at noaa.gov Mon Jul 9 15:04:09 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Mon, 9 Jul 2018 15:04:09 -0400 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: TL;DR- +1 on advocating the term ?generator comprehension? as a synonym for ?generator expression? in future documentation and instructional materials. The long version: If we were starting from scratch, maybe it would makes sense to use ? builder? or something else, rather than comprehension. But we?re not starting from scratch. And ?list comprehension? has become a pretty universal (or at least common) term in the python world. So we really can?t expect to re-name that. And despite the term having its roots in an esoteric corner of mathematics, I?ve never seen anyone confused by it ? it?s just a new word for a new thing to most folks. Which is actually why ?generator expression? is a bit confusing ? it seems to have more in common with comprehensions than with generators (As in: the things functions with a yield in them produce) In fact, the term ?generator? is a bit confusing itself ? due to it being a common English word, it seems to refer to the generic idea of ?a thing that generates values on the fly?, which, of course it can be in Python. But it actually is a specific type. And you can make a ?thing that generates values in the fly? with a regular class that follows the iterator protocol. So re-using the word in ?generator expression? just adds a bit to the confusion. (Even though it?s technically correct, it does ?make? an actual generator object, doesn?t it? (On a phone, so I can?t check) but that?s kind of an implementation detail. Also: if you google: ?python list comprehension? you get a LOT of hits to tutorials, etc. So it is WAY too late to change that name. When you google ?python generator expression? the top hit is the python docs, and the next few are about comparing generator expressions and list comprehensions. So I think we could start a new naming scheme for those. ?cause then we?d get lost of questions about the difference between generator expressions and generator comprehensions ;-) -CHB From guido at python.org Mon Jul 9 15:05:20 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Jul 2018 12:05:20 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: I think this is worth a try. On Mon, Jul 9, 2018 at 12:04 PM Chris Barker - NOAA Federal < chris.barker at noaa.gov> wrote: > TL;DR- > > +1 on advocating the term ?generator comprehension? as a synonym for > ?generator expression? in future documentation and instructional > materials. > > The long version: > > If we were starting from scratch, maybe it would makes sense to use ? > builder? or something else, rather than comprehension. But we?re not > starting from scratch. > > And ?list comprehension? has become a pretty universal (or at least > common) term in the python world. So we really can?t expect to re-name > that. > > And despite the term having its roots in an esoteric corner of > mathematics, I?ve never seen anyone confused by it ? it?s just a new > word for a new thing to most folks. > > Which is actually why ?generator expression? is a bit confusing ? it > seems to have more in common with comprehensions than with generators > > (As in: the things functions with a yield in them produce) > > In fact, the term ?generator? is a bit confusing itself ? due to it > being a common English word, it seems to refer to the generic idea of > ?a thing that generates values on the fly?, which, of course it can be > in Python. But it actually is a specific type. And you can make a > ?thing that generates values in the fly? with a regular class that > follows the iterator protocol. > > So re-using the word in ?generator expression? just adds a bit to the > confusion. (Even though it?s technically correct, it does ?make? an > actual generator object, doesn?t it? (On a phone, so I can?t check) > but that?s kind of an implementation detail. > > Also: if you google: ?python list comprehension? you get a LOT of hits > to tutorials, etc. > > So it is WAY too late to change that name. > > When you google ?python generator expression? the top hit is the > python docs, and the next few are about comparing generator > expressions and list comprehensions. > > So I think we could start a new naming scheme for those. > > ?cause then we?d get lost of questions about the difference between > generator expressions and generator comprehensions ;-) > > > -CHB > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From olivier.grisel at ensta.org Mon Jul 9 15:34:04 2018 From: olivier.grisel at ensta.org (Olivier Grisel) Date: Mon, 9 Jul 2018 21:34:04 +0200 Subject: [Python-Dev] PEP 574 -- Pickle protocol 5 with out-of-band data In-Reply-To: References: <20180328203951.3488ca41@fsol> <34eaf991-fb6c-c540-ea6d-e7b6267df34f@gmail.com> <20180328221939.10d3cd01@fsol> Message-ID: Hi Serhiy, Do you have any bug / issue to track the work you want to do to add native pickling support for locally defined function and classes by serializing the code objects like cloudpickle does? Is this work public on some git branch on GitHub or somewhere else? Cheers, -- Olivier ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jul 9 15:57:20 2018 From: brett at python.org (Brett Cannon) Date: Mon, 9 Jul 2018 12:57:20 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: On Mon, 9 Jul 2018 at 12:05 Guido van Rossum wrote: > I think this is worth a try. > How far do we want to go with this? Update the docs? Update the grammar and/or code? I think the former is probably good enough for now to see if it takes, and if it does then we can talk about updating code to not confuse people. > > On Mon, Jul 9, 2018 at 12:04 PM Chris Barker - NOAA Federal < > chris.barker at noaa.gov> wrote: > >> TL;DR- >> >> +1 on advocating the term ?generator comprehension? as a synonym for >> ?generator expression? in future documentation and instructional >> materials. >> >> The long version: >> >> If we were starting from scratch, maybe it would makes sense to use ? >> builder? or something else, rather than comprehension. But we?re not >> starting from scratch. >> >> And ?list comprehension? has become a pretty universal (or at least >> common) term in the python world. So we really can?t expect to re-name >> that. >> >> And despite the term having its roots in an esoteric corner of >> mathematics, I?ve never seen anyone confused by it ? it?s just a new >> word for a new thing to most folks. >> >> Which is actually why ?generator expression? is a bit confusing ? it >> seems to have more in common with comprehensions than with generators >> >> (As in: the things functions with a yield in them produce) >> >> In fact, the term ?generator? is a bit confusing itself ? due to it >> being a common English word, it seems to refer to the generic idea of >> ?a thing that generates values on the fly?, which, of course it can be >> in Python. But it actually is a specific type. And you can make a >> ?thing that generates values in the fly? with a regular class that >> follows the iterator protocol. >> >> So re-using the word in ?generator expression? just adds a bit to the >> confusion. (Even though it?s technically correct, it does ?make? an >> actual generator object, doesn?t it? (On a phone, so I can?t check) >> but that?s kind of an implementation detail. >> >> Also: if you google: ?python list comprehension? you get a LOT of hits >> to tutorials, etc. >> >> So it is WAY too late to change that name. >> >> When you google ?python generator expression? the top hit is the >> python docs, and the next few are about comparing generator >> expressions and list comprehensions. >> >> So I think we could start a new naming scheme for those. >> >> ?cause then we?d get lost of questions about the difference between >> generator expressions and generator comprehensions ;-) >> >> >> -CHB >> > > > -- > --Guido van Rossum (python.org/~guido) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Jul 9 16:18:07 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Jul 2018 13:18:07 -0700 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: Definitely docs first. And we should keep references to generator expressions too, if only to explain that they've been renamed. Perhaps someone should try it out in a 3rd party tutorial to see how it goes? I'm CC'ing Raymond Hettinger. On Mon, Jul 9, 2018 at 12:57 PM Brett Cannon wrote: > > > On Mon, 9 Jul 2018 at 12:05 Guido van Rossum wrote: > >> I think this is worth a try. >> > > How far do we want to go with this? Update the docs? Update the grammar > and/or code? I think the former is probably good enough for now to see if > it takes, and if it does then we can talk about updating code to not > confuse people. > > >> >> On Mon, Jul 9, 2018 at 12:04 PM Chris Barker - NOAA Federal < >> chris.barker at noaa.gov> wrote: >> >>> TL;DR- >>> >>> +1 on advocating the term ?generator comprehension? as a synonym for >>> ?generator expression? in future documentation and instructional >>> materials. >>> >>> The long version: >>> >>> If we were starting from scratch, maybe it would makes sense to use ? >>> builder? or something else, rather than comprehension. But we?re not >>> starting from scratch. >>> >>> And ?list comprehension? has become a pretty universal (or at least >>> common) term in the python world. So we really can?t expect to re-name >>> that. >>> >>> And despite the term having its roots in an esoteric corner of >>> mathematics, I?ve never seen anyone confused by it ? it?s just a new >>> word for a new thing to most folks. >>> >>> Which is actually why ?generator expression? is a bit confusing ? it >>> seems to have more in common with comprehensions than with generators >>> >>> (As in: the things functions with a yield in them produce) >>> >>> In fact, the term ?generator? is a bit confusing itself ? due to it >>> being a common English word, it seems to refer to the generic idea of >>> ?a thing that generates values on the fly?, which, of course it can be >>> in Python. But it actually is a specific type. And you can make a >>> ?thing that generates values in the fly? with a regular class that >>> follows the iterator protocol. >>> >>> So re-using the word in ?generator expression? just adds a bit to the >>> confusion. (Even though it?s technically correct, it does ?make? an >>> actual generator object, doesn?t it? (On a phone, so I can?t check) >>> but that?s kind of an implementation detail. >>> >>> Also: if you google: ?python list comprehension? you get a LOT of hits >>> to tutorials, etc. >>> >>> So it is WAY too late to change that name. >>> >>> When you google ?python generator expression? the top hit is the >>> python docs, and the next few are about comparing generator >>> expressions and list comprehensions. >>> >>> So I think we could start a new naming scheme for those. >>> >>> ?cause then we?d get lost of questions about the difference between >>> generator expressions and generator comprehensions ;-) >>> >>> >>> -CHB >>> >> >> >> -- >> --Guido van Rossum (python.org/~guido) >> > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Mon Jul 9 18:20:30 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 10 Jul 2018 00:20:30 +0200 Subject: [Python-Dev] Micro-benchmarks for function calls (PEP 576/579/580) Message-ID: <5B43DFAE.6060905@UGent.be> Here is an initial version of a micro-benchmark for C function calling: https://github.com/jdemeyer/callbench I don't have results yet, since I'm struggling to find the right options to "perf timeit" to get a stable result. If somebody knows how to do this, help is welcome. Jeroen. From anthony.flury at btinternet.com Mon Jul 9 18:26:25 2018 From: anthony.flury at btinternet.com (Anthony Flury) Date: Mon, 9 Jul 2018 23:26:25 +0100 Subject: [Python-Dev] Call for prudence about PEP-572 In-Reply-To: References: Message-ID: <6ec22f31-23e0-d1ce-fafd-a4365930ee31@btinternet.com> On 09/07/18 08:26, Mat?j Cepl wrote: > On 2018-07-07, 15:48 GMT, Guido van Rossum wrote: >> if validate(name := re.search(pattern, line).group(1)): >> return name > Except there is no error handling for situations when > re.search() returns None, so one shouldn't use it anyway (most > of the time). Which seems to me like another nice example why > one should stay away from this style as much as possible. I am > too lazy to be tempted into this > nice-example-terrible-production-code world. So wrap it in a try/except to capture the re.search returning None: ??? try: ??? ??? if validate(name := re.search(pattern, line).group(1)): ???? ??? ??? return name except TypeError: ? ??? ??? # No match case And this solution is even better if the validate function raises an exception rather than True/False which I think is much better than trying to do multiple steps : ?? match = re.search(pattern, line) ?? if match is not None: ??? ?? name = match.group(1) ?? ? ? if name and validate(name): ??? ?? ? ???? return name ?????? else: ?????????? # No valid case ?? else: ?? ??? # No Match case > Best, > > Mat?j -- -- Anthony Flury email : *Anthony.flury at btinternet.com* Twitter : *@TonyFlury * From guido at python.org Mon Jul 9 21:00:07 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Jul 2018 18:00:07 -0700 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft Message-ID: A lot has been said about PEP 572. I am planning to accept it soon, hopefully within a week. I realize I should have posted the draft from May 22 (when Tim and I were added as authors and it was significantly updated -- see https://github.com/python/peps/pull/654). For this I apologize. Since then we've been diligently updating various details, tightening the spec without really changing the intended design. I know it is inevitable that there will be replies attempting to convince us that a different syntax is better or that we shouldn't do this at all. Please save your breath. Iv'e seen every possible such response before, and they haven't convinced me. At this point we're just looking for feedback on the text of the document, e.g. pointing out ambiguities or requests for more clarity in some part of the spec, *not* for alternative designs or passionate appeals to PEP 20. Below is the latest draft. All three authors are happy with it. You can also read the PEP online at https://www.python.org/dev/peps/pep-0572/. We strongly prefer feedback in the form of Pull Requests to the peps repo (the file is at https://github.com/python/peps/blob/master/pep-0572.rst). PEP: 572 Title: Assignment Expressions Author: Chris Angelico , Tim Peters , Guido van Rossum Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018, 02-Mar-2018, 23-Mar-2018, 04-Apr-2018, 17-Apr-2018, 25-Apr-2018, 09-Jul-2018 Abstract ======== This is a proposal for creating a way to assign to variables within an expression using the notation ``NAME := expr``. Pending Acceptance ------------------ This PEP will be accepted, however it needs some editing for clarity and exact specification. A final draft will be posted to python-dev. Note that alternate syntax proposals ("EXPR as NAME", "EXPR given ...") are no longer under consideration. Hopefully the final draft will be posted well before the end of July 2018. Rationale ========= Naming the result of an expression is an important part of programming, allowing a descriptive name to be used in place of a longer expression, and permitting reuse. Currently, this feature is available only in statement form, making it unavailable in list comprehensions and other expression contexts. Additionally, naming sub-parts of a large expression can assist an interactive debugger, providing useful display hooks and partial results. Without a way to capture sub-expressions inline, this would require refactoring of the original code; with assignment expressions, this merely requires the insertion of a few ``name :=`` markers. Removing the need to refactor reduces the likelihood that the code be inadvertently changed as part of debugging (a common cause of Heisenbugs), and is easier to dictate to another programmer. The importance of real code --------------------------- During the development of this PEP many people (supporters and critics both) have had a tendency to focus on toy examples on the one hand, and on overly complex examples on the other. The danger of toy examples is twofold: they are often too abstract to make anyone go "ooh, that's compelling", and they are easily refuted with "I would never write it that way anyway". The danger of overly complex examples is that they provide a convenient strawman for critics of the proposal to shoot down ("that's obfuscated"). Yet there is some use for both extremely simple and extremely complex examples: they are helpful to clarify the intended semantics. Therefore there will be some of each below. However, in order to be *compelling*, examples should be rooted in real code, i.e. code that was written without any thought of this PEP, as part of a useful application, however large or small. Tim Peters has been extremely helpful by going over his own personal code repository and picking examples of code he had written that (in his view) would have been *clearer* if rewritten with (sparing) use of assignment expressions. His conclusion: the current proposal would have allowed a modest but clear improvement in quite a few bits of code. Another use of real code is to observe indirectly how much value programmers place on compactness. Guido van Rossum searched through a Dropbox code base and discovered some evidence that programmers value writing fewer lines over shorter lines. Case in point: Guido found several examples where a programmer repeated a subexpression, slowing down the program, in order to save one line of code, e.g. instead of writing:: match = re.match(data) group = match.group(1) if match else None they would write:: group = re.match(data).group(1) if re.match(data) else None Another example illustrates that programmers sometimes do more work to save an extra level of indentation:: match1 = pattern1.match(data) match2 = pattern2.match(data) if match1: return match1.group(1) elif match2: return match2.group(2) This code tries to match ``pattern2`` even if ``pattern1`` has a match (in which case the match on ``pattern2`` is never used). The more efficient rewrite would have been:: match1 = pattern1.match(data) if match1: return match1.group(1) else: match2 = pattern2.match(data) if match2: return match2.group(2) Syntax and semantics ==================== In most contexts where arbitrary Python expressions can be used, a **named expression** can appear. This is of the form ``NAME := expr`` where ``expr`` is any valid Python expression other than an unparenthesized tuple, and ``NAME`` is an identifier. The value of such a named expression is the same as the incorporated expression, with the additional side-effect that the target is assigned that value:: # Handle a matched regex if (match := pattern.search(data)) is not None: ... # A more explicit alternative to the 2-arg form of iter() invocation while (value := read_next_item()) is not None: ... # Share a subexpression between a comprehension filter clause and its output filtered_data = [y for x in data if (y := f(x)) is not None] Exceptional cases ----------------- There are a few places where assignment expressions are not allowed, in order to avoid ambiguities or user confusion: - Unparenthesized assignment expressions are prohibited at the top level of an expression statement. Example:: y := f(x) # INVALID (y := f(x)) # Valid, though not recommended This rule is included to simplify the choice for the user between an assignment statements and an assignment expression -- there is no syntactic position where both are valid. - Unparenthesized assignment expressions are prohibited at the top level of the right hand side of an assignment statement. Example:: y0 = y1 := f(x) # INVALID y0 = (y1 := f(x)) # Valid, though discouraged Again, this rule is included to avoid two visually similar ways of saying the same thing. - Unparenthesized assignment expressions are prohibited for the value of a keyword argument in a call. Example:: foo(x = y := f(x)) # INVALID foo(x=(y := f(x))) # Valid, though probably confusing This rule is included to disallow excessively confusing code, and because parsing keyword arguments is complex enough already. - Unparenthesized assignment expressions are prohibited at the top level of a function default value. Example:: def foo(answer = p := 42): # INVALID ... def foo(answer=(p := 42)): # Valid, though not great style ... This rule is included to discourage side effects in a position whose exact semantics are already confusing to many users (cf. the common style recommendation against mutable default values), and also to echo the similar prohibition in calls (the previous bullet). Scope of the target ------------------- An assignment expression does not introduce a new scope. In most cases the scope in which the target will be bound is self-explanatory: it is the current scope. If this scope contains a ``nonlocal`` or ``global`` declaration for the target, the assignment expression honors that. A lambda (being an explicit, if anonymous, function definition) counts as a scope for this purpose. There is one special case: an assignment expression occurring in a list, set or dict comprehension or in a generator expression (below collectively referred to as "comprehensions") binds the target in the containing scope, honoring a ``nonlocal`` or ``global`` declaration for the target in that scope, if one exists. For the purpose of this rule the containing scope of a nested comprehension is the scope that contains the outermost comprehension. A lambda counts as a containing scope. The motivation for this special case is twofold. First, it allows us to conveniently capture a "witness" for an ``any()`` expression, or a counterexample for ``all()``, for example:: if any((comment := line).startswith('#') for line in lines): print("First comment:", comment) else: print("There are no comments") if all((nonblank := line).strip() == '' for line in lines): print("All lines are blank") else: print("First non-blank line:", nonblank) Second, it allows a compact way of updating mutable state from a comprehension, for example:: # Compute partial sums in a list comprehension total = 0 partial_sums = [total := total + v for v in values] print("Total:", total) An exception to this special case applies when the target name is the same as a loop control variable for a comprehension containing it. This is invalid. This exception exists to rule out edge cases of the above scope rules as illustrated by ``[i := i+1 for i in range(5)]`` or ``[[(j := j) for i in range(5)] for j in range(5)]``. Note that this exception also applies to ``[i := 0 for i, j in stuff]``, as well as to cases like ``[i+1 for i in i := stuff]``. A further exception applies when an assignment expression occurrs in a comprehension whose containing scope is a class scope. If the rules above were to result in the target being assigned in that class's scope, the assignment expression is expressly invalid. (The reason for the latter exception is the implicit function created for comprehensions -- there is currently no runtime mechanism for a function to refer to a variable in the containing class scope, and we do not want to add such a mechanism. If this issue ever gets resolved this special case may be removed from the specification of assignment expressions. Note that the problem already exists for *using* a variable defined in the class scope from a comprehension.) See Appendix B for some examples of how the rules for targets in comprehensions translate to equivalent code. The two invalid cases listed above raise ``TargetScopeError``, a subclass of ``SyntaxError`` (with the same signature). Relative precedence of ``:=`` ----------------------------- The ``:=`` operator groups more tightly than a comma in all syntactic positions where it is legal, but less tightly than all operators, including ``or``, ``and`` and ``not``. As follows from section "Exceptional cases" above, it is never allowed at the same level as ``=``. In case a different grouping is desired, parentheses should be used. The ``:=`` operator may be used directly in a positional function call argument; however it is invalid directly in a keyword argument. Some examples to clarify what's technically valid or invalid:: # INVALID x := 0 # Valid alternative (x := 0) # INVALID x = y := 0 # Valid alternative x = (y := 0) # Valid len(lines := f.readlines()) # Valid foo(x := 3, cat='vector') # INVALID foo(cat=category := 'vector') # Valid alternative foo(cat=(category := 'vector')) Most of the "valid" examples above are not recommended, since human readers of Python source code who are quickly glancing at some code may miss the distinction. But simple cases are not objectionable:: # Valid if any(len(longline := line) >= 100 for line in lines): print("Extremely long line:", longline) This PEP recommends always putting spaces around ``:=``, similar to PEP 8's recommendation for ``=`` when used for assignment, whereas the latter disallows spaces around ``=`` used for keyword arguments.) Change to evaluation order -------------------------- In order to have precisely defined semantics, the proposal requires evaluation order to be well-defined. This is technically not a new requirement, as function calls may already have side effects. Python already has a rule that subexpressions are generally evaluated from left to right. However, assignment expressions make these side effects more visible, and we propose a single change to the current evaluation order: - In a dict comprehension ``{X: Y for ...}``, ``Y`` is currently evaluated before ``X``. We propose to change this so that ``X`` is evaluated before ``Y``. (In a dict display like ``{X: Y}}`` this is already the case, and also in ``dict((X, Y) for ...)`` which should clearly be equivalent to the dict comprehension.) Differences between assignment expressions and assignment statements --------------------------------------------------------------------- Most importantly, since ``:=`` is an expression, it can be used in contexts where statements are illegal, including lambda functions and comprehensions. Conversely, assignment expressions don't support the advanced features found in assignment statements: - Multiple targets are not directly supported:: x = y = z = 0 # Equivalent: (z := (y := (x := 0))) - Single assignment targets other than than a single ``NAME`` are not supported:: # No equivalent a[i] = x self.rest = [] - Priority around commas is different:: x = 1, 2 # Sets x to (1, 2) (x := 1, 2) # Sets x to 1 - Iterable packing and unpacking (both regular or extended forms) are not supported:: # Equivalent needs extra parentheses loc = x, y # Use (loc := (x, y)) info = name, phone, *rest # Use (info := (name, phone, *rest)) # No equivalent px, py, pz = position name, phone, email, *other_info = contact - Type annotations are not supported:: # No equivalent p: Optional[int] = None - Augmented assignment is not supported:: total += tax # Equivalent: (total := total + tax) Examples ======== Examples from the Python standard library ----------------------------------------- site.py ^^^^^^^ *env_base* is only used on these lines, putting its assignment on the if moves it as the "header" of the block. - Current:: env_base = os.environ.get("PYTHONUSERBASE", None) if env_base: return env_base - Improved:: if env_base := os.environ.get("PYTHONUSERBASE", None): return env_base _pydecimal.py ^^^^^^^^^^^^^ Avoid nested if and remove one indentation level. - Current:: if self._is_special: ans = self._check_nans(context=context) if ans: return ans - Improved:: if self._is_special and (ans := self._check_nans(context=context)): return ans copy.py ^^^^^^^ Code looks more regular and avoid multiple nested if. (See Appendix A for the origin of this example.) - Current:: reductor = dispatch_table.get(cls) if reductor: rv = reductor(x) else: reductor = getattr(x, "__reduce_ex__", None) if reductor: rv = reductor(4) else: reductor = getattr(x, "__reduce__", None) if reductor: rv = reductor() else: raise Error( "un(deep)copyable object of type %s" % cls) - Improved:: if reductor := dispatch_table.get(cls): rv = reductor(x) elif reductor := getattr(x, "__reduce_ex__", None): rv = reductor(4) elif reductor := getattr(x, "__reduce__", None): rv = reductor() else: raise Error("un(deep)copyable object of type %s" % cls) datetime.py ^^^^^^^^^^^ *tz* is only used for ``s += tz``, moving its assignment inside the if helps to show its scope. - Current:: s = _format_time(self._hour, self._minute, self._second, self._microsecond, timespec) tz = self._tzstr() if tz: s += tz return s - Improved:: s = _format_time(self._hour, self._minute, self._second, self._microsecond, timespec) if tz := self._tzstr(): s += tz return s sysconfig.py ^^^^^^^^^^^^ Calling ``fp.readline()`` in the ``while`` condition and calling ``.match()`` on the if lines make the code more compact without making it harder to understand. - Current:: while True: line = fp.readline() if not line: break m = define_rx.match(line) if m: n, v = m.group(1, 2) try: v = int(v) except ValueError: pass vars[n] = v else: m = undef_rx.match(line) if m: vars[m.group(1)] = 0 - Improved:: while line := fp.readline(): if m := define_rx.match(line): n, v = m.group(1, 2) try: v = int(v) except ValueError: pass vars[n] = v elif m := undef_rx.match(line): vars[m.group(1)] = 0 Simplifying list comprehensions ------------------------------- A list comprehension can map and filter efficiently by capturing the condition:: results = [(x, y, x/y) for x in input_data if (y := f(x)) > 0] Similarly, a subexpression can be reused within the main expression, by giving it a name on first use:: stuff = [[y := f(x), x/y] for x in range(5)] Note that in both cases the variable ``y`` is bound in the containing scope (i.e. at the same level as ``results`` or ``stuff``). Capturing condition values -------------------------- Assignment expressions can be used to good effect in the header of an ``if`` or ``while`` statement:: # Loop-and-a-half while (command := input("> ")) != "quit": print("You entered:", command) # Capturing regular expression match objects # See, for instance, Lib/pydoc.py, which uses a multiline spelling # of this effect if match := re.search(pat, text): print("Found:", match.group(0)) # The same syntax chains nicely into 'elif' statements, unlike the # equivalent using assignment statements. elif match := re.search(otherpat, text): print("Alternate found:", match.group(0)) elif match := re.search(third, text): print("Fallback found:", match.group(0)) # Reading socket data until an empty string is returned while data := sock.recv(): print("Received data:", data) Particularly with the ``while`` loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value. Fork ---- An example from the low-level UNIX world:: if pid := os.fork(): # Parent code else: # Child code Rejected alternative proposals ============================== Proposals broadly similar to this one have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above. Changing the scope rules for comprehensions ------------------------------------------- A previous version of this PEP proposed subtle changes to the scope rules for comprehensions, to make them more usable in class scope and to unify the scope of the "outermost iterable" and the rest of the comprehension. However, this part of the proposal would have caused backwards incompatibilities, and has been withdrawn so the PEP can focus on assignment expressions. Alternative spellings --------------------- Broadly the same semantics as the current proposal, but spelled differently. 1. ``EXPR as NAME``:: stuff = [[f(x) as y, x/y] for x in range(5)] Since ``EXPR as NAME`` already has meaning in ``import``, ``except`` and ``with`` statements (with different semantics), this would create unnecessary confusion or require special-casing (e.g. to forbid assignment within the headers of these statements). (Note that ``with EXPR as VAR`` does *not* simply assing the value of ``EXPR`` to ``VAR`` -- it calls ``EXPR.__enter__()`` and assigns the result of *that* to ``VAR``.) Additional reasons to prefer ``:=`` over this spelling include: - In ``if f(x) as y`` the assignment target doesn't jump out at you -- it just reads like ``if f x blah blah`` and it is too similar visually to ``if f(x) and y``. - In all other situations where an ``as`` clause is allowed, even readers with intermediary skills are led to anticipate that clause (however optional) by the keyword that starts the line, and the grammar ties that keyword closely to the as clause: - ``import foo as bar`` - ``except Exc as var`` - ``with ctxmgr() as var`` To the contrary, the assignment expression does not belong to the ``if`` or ``while`` that starts the line, and we intentionally allow assignment expressions in other contexts as well. - The parallel cadence between - ``NAME = EXPR`` - ``if NAME := EXPR`` reinforces the visual recognition of assignment expressions. 2. ``EXPR -> NAME``:: stuff = [[f(x) -> y, x/y] for x in range(5)] This syntax is inspired by languages such as R and Haskell, and some programmable calculators. (Note that a left-facing arrow ``y <- f(x)`` is not possible in Python, as it would be interpreted as less-than and unary minus.) This syntax has a slight advantage over 'as' in that it does not conflict with ``with``, ``except`` and ``import``, but otherwise is equivalent. But it is entirely unrelated to Python's other use of ``->`` (function return type annotations), and compared to ``:=`` (which dates back to Algol-58) it has a much weaker tradition. 3. Adorning statement-local names with a leading dot:: stuff = [[(f(x) as .y), x/.y] for x in range(5)] # with "as" stuff = [[(.y := f(x)), x/.y] for x in range(5)] # with ":=" This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable's scope is encoded into its name, making refactoring harder. 4. Adding a ``where:`` to any statement to create local name bindings:: value = x**2 + 2*x where: x = spam(1, 4, 7, q) Execution order is inverted (the indented body is performed first, followed by the "header"). This requires a new keyword, unless an existing keyword is repurposed (most likely ``with:``). See PEP 3150 for prior discussion on this subject (with the proposed keyword being ``given:``). 5. ``TARGET from EXPR``:: stuff = [[y from f(x), x/y] for x in range(5)] This syntax has fewer conflicts than ``as`` does (conflicting only with the ``raise Exc from Exc`` notation), but is otherwise comparable to it. Instead of paralleling ``with expr as target:`` (which can be useful but can also be confusing), this has no parallels, but is evocative. Special-casing conditional statements ------------------------------------- One of the most popular use-cases is ``if`` and ``while`` statements. Instead of a more general solution, this proposal enhances the syntax of these two statements to add a means of capturing the compared value:: if re.search(pat, text) as match: print("Found:", match.group(0)) This works beautifully if and ONLY if the desired condition is based on the truthiness of the captured value. It is thus effective for specific use-cases (regex matches, socket reads that return `''` when done), and completely useless in more complicated cases (eg where the condition is ``f(x) < 0`` and you want to capture the value of ``f(x)``). It also has no benefit to list comprehensions. Advantages: No syntactic ambiguities. Disadvantages: Answers only a fraction of possible use-cases, even in ``if``/``while`` statements. Special-casing comprehensions ----------------------------- Another common use-case is comprehensions (list/set/dict, and genexps). As above, proposals have been made for comprehension-specific solutions. 1. ``where``, ``let``, or ``given``:: stuff = [(y, x/y) where y = f(x) for x in range(5)] stuff = [(y, x/y) let y = f(x) for x in range(5)] stuff = [(y, x/y) given y = f(x) for x in range(5)] This brings the subexpression to a location in between the 'for' loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, ``where`` reads the most cleanly, but also has the greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` methods, as does ``tkinter.dnd.Icon`` in the standard library). 2. ``with NAME = EXPR``:: stuff = [(y, x/y) with y = f(x) for x in range(5)] As above, but reusing the `with` keyword. Doesn't read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into "longhand" for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. Would raise the question of why "with NAME = EXPR:" cannot be used as a statement on its own. 3. ``with EXPR as NAME``:: stuff = [(y, x/y) with f(x) as y for x in range(5)] As per option 2, but using ``as`` rather than an equals sign. Aligns syntactically with other uses of ``as`` for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of ``with`` inside a comprehension would be completely different from the meaning as a stand-alone statement, while retaining identical syntax. Regardless of the spelling chosen, this introduces a stark difference between comprehensions and the equivalent unrolled long-hand form of the loop. It is no longer possible to unwrap the loop into statement form without reworking any name bindings. The only keyword that can be repurposed to this task is ``with``, thus giving it sneakily different semantics in a comprehension than in a statement; alternatively, a new keyword is needed, with all the costs therein. Lowering operator precedence ---------------------------- There are two logical precedences for the ``:=`` operator. Either it should bind as loosely as possible, as does statement-assignment; or it should bind more tightly than comparison operators. Placing its precedence between the comparison and arithmetic operators (to be precise: just lower than bitwise OR) allows most uses inside ``while`` and ``if`` conditions to be spelled without parentheses, as it is most likely that you wish to capture the value of something, then perform a comparison on it:: pos = -1 while pos := buffer.find(search_term, pos + 1) >= 0: ... Once find() returns -1, the loop terminates. If ``:=`` binds as loosely as ``=`` does, this would capture the result of the comparison (generally either ``True`` or ``False``), which is less useful. While this behaviour would be convenient in many situations, it is also harder to explain than "the := operator behaves just like the assignment statement", and as such, the precedence for ``:=`` has been made as close as possible to that of ``=`` (with the exception that it binds tighter than comma). Allowing commas to the right ---------------------------- Some critics have claimed that the assignment expressions should allow unparenthesized tuples on the right, so that these two would be equivalent:: (point := (x, y)) (point := x, y) (With the current version of the proposal, the latter would be equivalent to ``((point := x), y)``.) However, adopting this stance would logically lead to the conclusion that when used in a function call, assignment expressions also bind less tight than comma, so we'd have the following confusing equivalence:: foo(x := 1, y) foo(x := (1, y)) The less confusing option is to make ``:=`` bind more tightly than comma. Always requiring parentheses ---------------------------- It's been proposed to just always require parenthesize around an assignment expression. This would resolve many ambiguities, and indeed parentheses will frequently be needed to extract the desired subexpression. But in the following cases the extra parentheses feel redundant:: # Top level in if if match := pattern.match(line): return match.group(1) # Short call len(lines := f.readlines()) Frequently Raised Objections ============================ Why not just turn existing assignment into an expression? --------------------------------------------------------- C and its derivatives define the ``=`` operator as an expression, rather than a statement as is Python's way. This allows assignments in more contexts, including contexts where comparisons are more common. The syntactic similarity between ``if (x == y)`` and ``if (x = y)`` belies their drastically different semantics. Thus this proposal uses ``:=`` to clarify the distinction. This could be used to create ugly code! --------------------------------------- So can anything else. This is a tool, and it is up to the programmer to use it where it makes sense, and not use it where superior constructs can be used. With assignment expressions, why bother with assignment statements? ------------------------------------------------------------------- The two forms have different flexibilities. The ``:=`` operator can be used inside a larger expression; the ``=`` statement can be augmented to ``+=`` and its friends, can be chained, and can assign to attributes and subscripts. Why not use a sublocal scope and prevent namespace pollution? ------------------------------------------------------------- Previous revisions of this proposal involved sublocal scope (restricted to a single statement), preventing name leakage and namespace pollution. While a definite advantage in a number of situations, this increases complexity in many others, and the costs are not justified by the benefits. In the interests of language simplicity, the name bindings created here are exactly equivalent to any other name bindings, including that usage at class or module scope will create externally-visible names. This is no different from ``for`` loops or other constructs, and can be solved the same way: ``del`` the name once it is no longer needed, or prefix it with an underscore. (The author wishes to thank Guido van Rossum and Christoph Groth for their suggestions to move the proposal in this direction. [2]_) Style guide recommendations =========================== As expression assignments can sometimes be used equivalently to statement assignments, the question of which should be preferred will arise. For the benefit of style guides such as PEP 8, two recommendations are suggested. 1. If either assignment statements or assignment expressions can be used, prefer statements; they are a clear declaration of intent. 2. If using assignment expressions would lead to ambiguity about execution order, restructure it to use statements instead. Acknowledgements ================ The authors wish to thank Nick Coghlan and Steven D'Aprano for their considerable contributions to this proposal, and members of the core-mentorship mailing list for assistance with implementation. Appendix A: Tim Peters's findings ================================= Here's a brief essay Tim Peters wrote on the topic. I dislike "busy" lines of code, and also dislike putting conceptually unrelated logic on a single line. So, for example, instead of:: i = j = count = nerrors = 0 I prefer:: i = j = 0 count = 0 nerrors = 0 instead. So I suspected I'd find few places I'd want to use assignment expressions. I didn't even consider them for lines already stretching halfway across the screen. In other cases, "unrelated" ruled:: mylast = mylast[1] yield mylast[0] is a vast improvment over the briefer:: yield (mylast := mylast[1])[0] The original two statements are doing entirely different conceptual things, and slamming them together is conceptually insane. In other cases, combining related logic made it harder to understand, such as rewriting:: while True: old = total total += term if old == total: return total term *= mx2 / (i*(i+1)) i += 2 as the briefer:: while total != (total := total + term): term *= mx2 / (i*(i+1)) i += 2 return total The ``while`` test there is too subtle, crucially relying on strict left-to-right evaluation in a non-short-circuiting or method-chaining context. My brain isn't wired that way. But cases like that were rare. Name binding is very frequent, and "sparse is better than dense" does not mean "almost empty is better than sparse". For example, I have many functions that return ``None`` or ``0`` to communicate "I have nothing useful to return in this case, but since that's expected often I'm not going to annoy you with an exception". This is essentially the same as regular expression search functions returning ``None`` when there is no match. So there was lots of code of the form:: result = solution(xs, n) if result: # use result I find that clearer, and certainly a bit less typing and pattern-matching reading, as:: if result := solution(xs, n): # use result It's also nice to trade away a small amount of horizontal whitespace to get another _line_ of surrounding code on screen. I didn't give much weight to this at first, but it was so very frequent it added up, and I soon enough became annoyed that I couldn't actually run the briefer code. That surprised me! There are other cases where assignment expressions really shine. Rather than pick another from my code, Kirill Balunov gave a lovely example from the standard library's ``copy()`` function in ``copy.py``:: reductor = dispatch_table.get(cls) if reductor: rv = reductor(x) else: reductor = getattr(x, "__reduce_ex__", None) if reductor: rv = reductor(4) else: reductor = getattr(x, "__reduce__", None) if reductor: rv = reductor() else: raise Error("un(shallow)copyable object of type %s" % cls) The ever-increasing indentation is semantically misleading: the logic is conceptually flat, "the first test that succeeds wins":: if reductor := dispatch_table.get(cls): rv = reductor(x) elif reductor := getattr(x, "__reduce_ex__", None): rv = reductor(4) elif reductor := getattr(x, "__reduce__", None): rv = reductor() else: raise Error("un(shallow)copyable object of type %s" % cls) Using easy assignment expressions allows the visual structure of the code to emphasize the conceptual flatness of the logic; ever-increasing indentation obscured it. A smaller example from my code delighted me, both allowing to put inherently related logic in a single line, and allowing to remove an annoying "artificial" indentation level:: diff = x - x_base if diff: g = gcd(diff, n) if g > 1: return g became:: if (diff := x - x_base) and (g := gcd(diff, n)) > 1: return g That ``if`` is about as long as I want my lines to get, but remains easy to follow. So, in all, in most lines binding a name, I wouldn't use assignment expressions, but because that construct is so very frequent, that leaves many places I would. In most of the latter, I found a small win that adds up due to how often it occurs, and in the rest I found a moderate to major win. I'd certainly use it more often than ternary ``if``, but significantly less often than augmented assignment. A numeric example ----------------- I have another example that quite impressed me at the time. Where all variables are positive integers, and a is at least as large as the n'th root of x, this algorithm returns the floor of the n'th root of x (and roughly doubling the number of accurate bits per iteration):: while a > (d := x // a**(n-1)): a = ((n-1)*a + d) // n return a It's not obvious why that works, but is no more obvious in the "loop and a half" form. It's hard to prove correctness without building on the right insight (the "arithmetic mean - geometric mean inequality"), and knowing some non-trivial things about how nested floor functions behave. That is, the challenges are in the math, not really in the coding. If you do know all that, then the assignment-expression form is easily read as "while the current guess is too large, get a smaller guess", where the "too large?" test and the new guess share an expensive sub-expression. To my eyes, the original form is harder to understand:: while True: d = x // a**(n-1) if a <= d: break a = ((n-1)*a + d) // n return a Appendix B: Rough code translations for comprehensions ====================================================== This appendix attempts to clarify (though not specify) the rules when a target occurs in a comprehension or in a generator expression. For a number of illustrative examples we show the original code, containing a comprehension, and the translation, where the comprehension has been replaced by an equivalent generator function plus some scaffolding. Since ``[x for ...]`` is equivalent to ``list(x for ...)`` these examples all use list comprehensions without loss of generality. And since these examples are meant to clarify edge cases of the rules, they aren't trying to look like real code. Note: comprehensions are already implemented via synthesizing nested generator functions like those in this appendix. The new part is adding appropriate declarations to establish the intended scope of assignment expression targets (the same scope they resolve to as if the assignment were performed in the block containing the outermost comprehension). For type inference purposes, these illustrative expansions do not imply that assignment expression targets are always Optional (but they do indicate the target binding scope). Let's start with a reminder of what code is generated for a generator expression without assignment expression. - Original code (EXPR usually references VAR):: def f(): a = [EXPR for VAR in ITERABLE] - Translation (let's not worry about name conflicts):: def f(): def genexpr(iterator): for VAR in iterator: yield EXPR a = list(genexpr(iter(ITERABLE))) Let's add a simple assignment expression. - Original code:: def f(): a = [TARGET := EXPR for VAR in ITERABLE] - Translation:: def f(): if False: TARGET = None # Dead code to ensure TARGET is a local variable def genexpr(iterator): nonlocal TARGET for VAR in iterator: TARGET = EXPR yield TARGET a = list(genexpr(iter(ITERABLE))) Let's add a ``global TARGET`` declaration in ``f()``. - Original code:: def f(): global TARGET a = [TARGET := EXPR for VAR in ITERABLE] - Translation:: def f(): global TARGET def genexpr(iterator): global TARGET for VAR in iterator: TARGET = EXPR yield TARGET a = list(genexpr(iter(ITERABLE))) Or instead let's add a ``nonlocal TARGET`` declaration in ``f()``. - Original code:: def g(): TARGET = ... def f(): nonlocal TARGET a = [TARGET := EXPR for VAR in ITERABLE] - Translation:: def g(): TARGET = ... def f(): nonlocal TARGET def genexpr(iterator): nonlocal TARGET for VAR in iterator: TARGET = EXPR yield TARGET a = list(genexpr(iter(ITERABLE))) Finally, let's nest two comprehensions. - Original code:: def f(): a = [[TARGET := i for i in range(3)] for j in range(2)] # I.e., a = [[0, 1, 2], [0, 1, 2]] print(TARGET) # prints 2 - Translation:: def f(): if False: TARGET = None def outer_genexpr(outer_iterator): nonlocal TARGET def inner_generator(inner_iterator): nonlocal TARGET for i in inner_iterator: TARGET = i yield i for j in outer_iterator: yield list(inner_generator(range(3))) a = list(outer_genexpr(range(2))) print(TARGET) Appendix C: No Changes to Scope Semantics ========================================= Because it has been a point of confusion, note that nothing about Python's scoping semantics is changed. Function-local scopes continue to be resolved at compile time, and to have indefinite temporal extent at run time ("full closures"). Example:: a = 42 def f(): # `a` is local to `f` yield ((a := i) for i in range(3)) yield lambda: a + 100 print("done") Then:: >>> results = list(f()) # [genexp, lambda] done # The execution frame for f no longer exists in CPython, # but f's locals live so long as they can still be referenced. >>> list(map(type, results)) [, ] >>> list(results[0]) [0, 1, 2] >>> results[1]() 102 >>> a 42 References ========== .. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/assignment-expressions) .. [2] Pivotal post regarding inline assignment semantics (https://mail.python.org/pipermail/python-ideas/2018-March/049409.html) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Mon Jul 9 21:53:47 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 10 Jul 2018 02:53:47 +0100 Subject: [Python-Dev] Encoding variable-length integers/counts in pickle Message-ID: <83f22260-f74d-6ea8-3464-6b3709f1b310@mrabarnett.plus.com> In the regex module I use an encoding scheme when pickling pattern objects which is based on the way MIDI encodes variable-length integers, and I think it might have a use in a pickle protocol. In the basic format, an integer is split up into 7-bit chunks, each chunk put into a byte, and the most-significant bit of the byte used to signal whether the value continues into the following byte. And integer must be encoded into the minimum number of bytes, so an encoded sequence of bytes would never start with 0x80. MIDI deviates from the basic idea in order to reduce the number of bytes, so as sequence of bytes in MIDI _can_ start with x080; this is fine for MIDI because it doesn't need to represent negative integers. The format I'm suggesting uses an initial 0x80 as a way of letting it encode negative integers. Here are a couple of Python functions that summarise the encoding and decoding (minus obvious optimisations for simplicity): def encode_varint(value: int) -> List[int]: negative = value < 0 encoded = [] if negative: final = -1 else: final = 0 while True: encoded.append(0x80 | (value & 0x7F)) value >>= 7 if value == final: break if negative: encoded.append(0x80) encoded.reverse() encoded[-1] &= 0x7F return encoded def decode_varint(encoded: Iterable[int]) -> int: byte = next(encoded) if byte == 0x80: value = -1 byte = next(encoded) else: value = 0 value = (value << 7) | (byte & 0x7F) while (byte & 0x80) != 0: byte = next(encoded) value = (value << 7) | (byte & 0x7F) return value The advantage of encoding integers in this way is that there's no limit to their size, so there's no need to add a new protocol to support larger counts. They can also make pickles smaller. Example: # Pickle (None, ) 0: \x80 PROTO 4 2: \x95 FRAME 4 11: N NONE 12: \x85 TUPLE1 13: \x94 MEMOIZE (as 0) 14: . STOP Here, FRAME takes an argument of 8 bytes. If you replaced FRAME with a version that accepted a variable-length count, you could reduce that argument to 1 byte. You also wouldn't need to have different fixed-length versions of an opcode, e.g. BINUNICODE and SHORT_BINUNICODE. Whether you do anything with this is entirely up to the core devs, I just thought someone might find it useful. From v+python at g.nevcal.com Mon Jul 9 22:40:49 2018 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 9 Jul 2018 19:40:49 -0700 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft In-Reply-To: References: Message-ID: On 7/9/2018 6:00 PM, Guido van Rossum wrote: > This rule is included to simplify the choice for the user between an > ? assignment statements and an assignment expression -- there is no "statements" should not be plural in the above line. > syntactic position where both are valid. > An exception to this special case applies when the target name is the > same as a loop control variable for a comprehension containing it. > This is invalid.? This exception exists to rule out edge cases of the > above scope rules as illustrated by ``[i := i+1 for i in range(5)]`` > or ``[[(j := j) for i in range(5)] for j in range(5)]``.? Note that > this exception also applies to ``[i := 0 for i, j in stuff]``, as well > as to cases like ``[i+1 for i in i := stuff]``. It is unclear whether exactly what is invalid. Is the use of the target name that is the same as (any of the nested) loop control variable invalid? I think, from discussions, that that is what is meant. But this paragraph could be interpreted as meaning the special case doesn't apply, meaning that the target name would be in a "sublocal" scope. > The ``:=`` operator groups more tightly than a comma in all syntactic > positions where it is legal, but less tightly than all operators, If comma is considered an operator, this sentence is inconsistent, would need to be "all other operators".? Even if comma is not considered an operator, the sentence would be more clear with "other" added, since ":=" is an operator. Sorry, I haven't learned pull requests. Glenn -------------- next part -------------- An HTML attachment was scrubbed... URL: From tritium-list at sdamon.com Mon Jul 9 22:54:37 2018 From: tritium-list at sdamon.com (Alex Walters) Date: Mon, 9 Jul 2018 22:54:37 -0400 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> Message-ID: > -----Original Message----- > From: Python-Dev list=sdamon.com at python.org> On Behalf Of Paul Moore > Why not just have a second button, "Download Python 3.7.0 (64-bit)" > alongside or below the "Download Python 3.7.0" button? People who > don't know the difference will just ignore it, people who do will be > able to choose straight from the main download page. > I think this is the solution. * If you don't know your architecture, 32-bit will mostly work so should remain the default (I say most, not all, since there are windows versions that run on ARM, but I think you can only install software through the store on those anyways.) * It's not exposed in the download drop down at all that x64 editions exist for those who know they want it. The drop down is wide enough to support a second download button. > Paul > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > list%40sdamon.com From rob.cliffe at btinternet.com Mon Jul 9 23:24:19 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Tue, 10 Jul 2018 04:24:19 +0100 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft In-Reply-To: References: Message-ID: I apologise for not replying in the form of a Pull Request - I don't know how to make one. On 10/07/2018 02:00, Guido van Rossum wrote: > Rationale > ========= > > Naming the result of an expression is an important part of programming, > allowing a descriptive name to be used in place of a longer expression, > and permitting reuse.? Currently, this feature is available only in > statement form, making it unavailable in list comprehensions and other > expression contexts. I think the wording of the last sentence gives too much importance to list comprehensions (IMO influenced by the amount of discussion of the PEP that was related to list comprehensions, understandably since that was the case where the semantics were most debatable).? I would suggest "... making it unavailable in expression contexts." or maybe "... making it unavailable in expression contexts (including list comprehension)." > Another example illustrates that programmers sometimes do more work to > save an extra level of indentation:: > > ??? match1 = pattern1.match(data) > ??? match2 = pattern2.match(data) > ??? if match1: > ??????? return match1.group(1) > ??? elif match2: > ??????? return match2.group(2) > > This code tries to match ``pattern2`` even if ``pattern1`` has a match > (in which case the match on ``pattern2`` is never used). The more > efficient rewrite would have been:: > > ??? match1 = pattern1.match(data) > ??? if match1: > ??????? return match1.group(1) > ??? else: > ??????? match2 = pattern2.match(data) > ??????? if match2: > ??????????? return match2.group(2) I suggest ... The more efficient rewrite would have been: ??? match1 = pattern1.match(data) ??? if match1: ??????? return match1.group(1) ??? match2 = pattern2.match(data) ??? if match2: ??????? return match2.group(2) (a more natural way to write it which avoids cascading indentation). > ??? # Handle a matched regex > ??? if (match := pattern.search(data)) is not None: > ??????? ... I suggest ??? # Handle a matched regex ??? if (match := pattern.search(data)) is not None: ??????? # do something with match I think it is really important to make clear the benefit of the PEP here: that "match" is bound to a value and can be used subsequently. > ??? # A more explicit alternative to the 2-arg form of iter() invocation > ??? while (value := read_next_item()) is not None: > ??????? ... As the 2-arg form of iter() is not that well known, I suggest that the iter version is spelled out for contrast.? (Sorry, I can't quite work it what it would be.) > ??? # Share a subexpression between a comprehension filter clause and > its output > ??? filtered_data = [y for x in data if (y := f(x)) is not None] That's fine, but what about also having an example that illustrates, simply, the "permitting reuse" in an expression part of the Rationale, e.g. ??? ??? powers = [ (y := x+1), y**2, y**3, y**4 ] (I appreciate that this sort of overlaps with the section "Simplifying list comprehensions", but it seems to me to be such an important part of the Rationale that it should be illustrated here.) > Relative precedence of ``:=`` > ----------------------------- > > The ``:=`` operator groups more tightly than a comma in all syntactic > positions where it is legal, but less tightly than all operators, > including ``or``, ``and`` and ``not``. and presumably including "if" and "else", as in ??? x := y if y else -1 Might it be worth adding "if" and "else" to the list? > - Single assignment targets other than than a single ``NAME`` are > ? not supported:: > > ??? # No equivalent > ??? a[i] = x > ??? self.rest = [] > > [snip] > > - Iterable packing and unpacking (both regular or extended forms) are > ? not supported:: > > ??? # Equivalent needs extra parentheses > ??? loc = x, y? # Use (loc := (x, y)) > ??? info = name, phone, *rest? # Use (info := (name, phone, *rest)) > > ??? # No equivalent > ??? px, py, pz = position > ??? name, phone, email, *other_info = contact > > [snip] > ??? total += tax? # Equivalent: (total := total + tax) Is it conceivable that some of these restrictions might be removed in a future version of Python?? If so, the PEP might include a note to this effect. Oh, and what I think are typos: (Note that ``with EXPR as VAR`` does *not* simply assing the value ?? of ``EXPR`` to ``VAR`` -- it calls ``EXPR.__enter__()`` and assigns ?? the result of *that* to ``VAR``.) ??? assing -> assign (eg where the condition is ``f(x) < 0`` ??? eg -> e.g. members of the core-mentorship mailing list ??? core-mentorship -> core mentorship is a vast improvment over the briefer:: ??? improvment -> improvement Best wishes Rob Cliffe From guido at python.org Mon Jul 9 23:43:14 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Jul 2018 20:43:14 -0700 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft In-Reply-To: References: Message-ID: On Mon, Jul 9, 2018 at 7:40 PM, Glenn Linderman wrote: > On 7/9/2018 6:00 PM, Guido van Rossum wrote: > > This rule is included to simplify the choice for the user between an > assignment statements and an assignment expression -- there is no > > > "statements" should not be plural in the above line. > Will fix. syntactic position where both are valid. > > > > An exception to this special case applies when the target name is the > same as a loop control variable for a comprehension containing it. > This is invalid. This exception exists to rule out edge cases of the > above scope rules as illustrated by ``[i := i+1 for i in range(5)]`` > or ``[[(j := j) for i in range(5)] for j in range(5)]``. Note that > this exception also applies to ``[i := 0 for i, j in stuff]``, as well > as to cases like ``[i+1 for i in i := stuff]``. > > > > It is unclear whether exactly what is invalid. Is the use of the target > name that is the same as (any of the nested) loop control variable invalid? > I think, from discussions, that that is what is meant. But this paragraph > could be interpreted as meaning the special case doesn't apply, meaning > that the target name would be in a "sublocal" scope. > Really? If it didn't say "this is invalid" I could see that "exception to the special case" might be interpreted as "the special case doesn't apply". But with "This is invalid" explicitly added I don't see how that interpretation could be valid. Is it clearer if I changed that to "Such code is invalid"? Or perhaps I should move "This is invalid" to the end of the paragraph? > The ``:=`` operator groups more tightly than a comma in all syntactic > positions where it is legal, but less tightly than all operators, > > > If comma is considered an operator, this sentence is inconsistent, would > need to be "all other operators". Even if comma is not considered an > operator, the sentence would be more clear with "other" added, since ":=" > is an operator. > Makes sense. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From v+python at g.nevcal.com Tue Jul 10 00:06:10 2018 From: v+python at g.nevcal.com (Glenn Linderman) Date: Mon, 9 Jul 2018 21:06:10 -0700 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft In-Reply-To: References: Message-ID: <7a4636dd-ba46-2326-6010-d588ae5cc0cd@g.nevcal.com> On 7/9/2018 8:43 PM, Guido van Rossum wrote: > >> An exception to this special case applies when the target name is the >> same as a loop control variable for a comprehension containing it. >> This is invalid.? This exception exists to rule out edge cases of the >> above scope rules as illustrated by ``[i := i+1 for i in range(5)]`` >> or ``[[(j := j) for i in range(5)] for j in range(5)]``. Note that >> this exception also applies to ``[i := 0 for i, j in stuff]``, as >> well >> as to cases like ``[i+1 for i in i := stuff]``. > > > It is unclear whether exactly what is invalid. Is the use of the > target name that is the same as (any of the nested) loop control > variable invalid? I think, from discussions, that that is what is > meant. But this paragraph could be interpreted as meaning the > special case doesn't apply, meaning that the target name would be > in a "sublocal" scope. > > > Really? If it didn't say "this is invalid" I could see that "exception > to the special case" might be interpreted as "the special case doesn't > apply". But with "This is invalid" explicitly added I don't see how > that interpretation could be valid. Is it clearer if I changed that to > "Such code is invalid"? Or perhaps I should move "This is invalid" to > the end of the paragraph? That's better; even better might be to say what is invalid... instead of used pronoun. "Use of any of the comprension loop control variables as a target name in an assignment expression is invalid." It isn't really an exception to the special case, it is an exception to the general rule than any old name can be used as an assignment expression target, IIUC. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jul 10 00:15:37 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 9 Jul 2018 21:15:37 -0700 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft In-Reply-To: References: Message-ID: On Mon, Jul 9, 2018 at 8:24 PM, Rob Cliffe via Python-Dev < python-dev at python.org> wrote: > I apologise for not replying in the form of a Pull Request - I don't know > how to make one. > > > On 10/07/2018 02:00, Guido van Rossum wrote: > >> Rationale >> ========= >> >> Naming the result of an expression is an important part of programming, >> allowing a descriptive name to be used in place of a longer expression, >> and permitting reuse. Currently, this feature is available only in >> statement form, making it unavailable in list comprehensions and other >> expression contexts. >> > > I think the wording of the last sentence gives too much importance to list > comprehensions (IMO influenced by the amount of discussion of the PEP that > was related to list comprehensions, understandably since that was the case > where the semantics were most debatable). I would suggest > "... making it unavailable in expression contexts." > or maybe > "... making it unavailable in expression contexts (including list > comprehension)." > Hm, I don't think it's worth tweaking the text here. The PEP has some choice things to say about comprehensions, and I don't mind calling them out especially here. Also, when you need a temporary variable outside a comprehension it's easy enough to rewrite the code a bit -- but for a comprehension you'd have to refactor the whole comprehension away (into a regular for-loop), which is a bit more effort. > Another example illustrates that programmers sometimes do more work to >> save an extra level of indentation:: >> >> match1 = pattern1.match(data) >> match2 = pattern2.match(data) >> if match1: >> return match1.group(1) >> elif match2: >> return match2.group(2) >> >> This code tries to match ``pattern2`` even if ``pattern1`` has a match >> (in which case the match on ``pattern2`` is never used). The more >> efficient rewrite would have been:: >> >> match1 = pattern1.match(data) >> if match1: >> return match1.group(1) >> else: >> match2 = pattern2.match(data) >> if match2: >> return match2.group(2) >> > I suggest > ... The more efficient rewrite would have been: > match1 = pattern1.match(data) > if match1: > return match1.group(1) > match2 = pattern2.match(data) > if match2: > return match2.group(2) > (a more natural way to write it which avoids cascading indentation). > Hm, I think I might have simplified this a bit from the code I actually found -- there might not have been a `return` in the original. I don't want to break the `if...elif` pattern in the rewrite. I guess I'll rewrite it using assignment to a variable instead of `return`. > # Handle a matched regex >> if (match := pattern.search(data)) is not None: >> ... >> > I suggest > # Handle a matched regex > if (match := pattern.search(data)) is not None: > # do something with match > I think it is really important to make clear the benefit of the PEP here: > that "match" is bound to a value and can be used subsequently. > > # A more explicit alternative to the 2-arg form of iter() invocation >> while (value := read_next_item()) is not None: >> ... >> > As the 2-arg form of iter() is not that well known, I suggest that the > iter version is spelled out for contrast. (Sorry, I can't quite work it > what it would be.) > Hm, you have a point there. I was referring to for value in iter(read_next_item, None): ... The example would be more convincing if there was an additional argument to read_next_value(), since then the 2-arg iter() version would have required a lambda. Maybe I shouldn't mention 2-arg iter here at all. > # Share a subexpression between a comprehension filter clause and its >> output >> filtered_data = [y for x in data if (y := f(x)) is not None] >> > That's fine, but what about also having an example that illustrates, > simply, the "permitting reuse" in an expression part of the Rationale, e.g. > powers = [ (y := x+1), y**2, y**3, y**4 ] > (I appreciate that this sort of overlaps with the section "Simplifying > list comprehensions", but it seems to me to be such an important part of > the Rationale that it should be illustrated here.) > I could add this (before the filtered_data example): # Reuse a value that's expensive to compute [y := f(x), y**2, y**3] > Relative precedence of ``:=`` >> ----------------------------- >> >> The ``:=`` operator groups more tightly than a comma in all syntactic >> positions where it is legal, but less tightly than all operators, >> including ``or``, ``and`` and ``not``. >> > and presumably including "if" and "else", as in > x := y if y else -1 > Might it be worth adding "if" and "else" to the list? > Good call, I'll add conditional expressions to the list. > - Single assignment targets other than than a single ``NAME`` are >> not supported:: >> >> # No equivalent >> a[i] = x >> self.rest = [] >> >> [snip] >> >> - Iterable packing and unpacking (both regular or extended forms) are >> not supported:: >> >> # Equivalent needs extra parentheses >> loc = x, y # Use (loc := (x, y)) >> info = name, phone, *rest # Use (info := (name, phone, *rest)) >> >> # No equivalent >> px, py, pz = position >> name, phone, email, *other_info = contact >> >> [snip] >> total += tax # Equivalent: (total := total + tax) >> > Is it conceivable that some of these restrictions might be removed in a > future version of Python? If so, the PEP might include a note to this > effect. > I wouldn't hold my breath. > Oh, and what I think are typos: > > (Note that ``with EXPR as VAR`` does *not* simply assing the value > of ``EXPR`` to ``VAR`` -- it calls ``EXPR.__enter__()`` and assigns > the result of *that* to ``VAR``.) > > assing -> assign > Fixed already. (eg where the condition is ``f(x) < 0`` > > eg -> e.g. > Thanks, will fix. > members of the core-mentorship mailing list > > core-mentorship -> core mentorship > Why? The list is named core-mentorship at python.org. > is a vast improvment over the briefer:: > > improvment -> improvement > Will fix. > Best wishes > Rob Cliffe > https://github.com/python/peps/pull/719 -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Tue Jul 10 00:19:10 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 10 Jul 2018 13:19:10 +0900 Subject: [Python-Dev] Micro-benchmarks for function calls (PEP 576/579/580) In-Reply-To: <5B43DFAE.6060905@UGent.be> References: <5B43DFAE.6060905@UGent.be> Message-ID: On Tue, Jul 10, 2018 at 7:23 AM Jeroen Demeyer wrote: > > Here is an initial version of a micro-benchmark for C function calling: > > https://github.com/jdemeyer/callbench > > I don't have results yet, since I'm struggling to find the right options > to "perf timeit" to get a stable result. If somebody knows how to do > this, help is welcome. > I suggest `--duplicate 10` option. While it is good for start point, please don't forget we need "application" benchmark. Even if some function call overhead can be 3x faster, if it takes only 3% of application execution time, total execution time only 1% faster. It's too small to accept PEP 580 complexity. Realistic application benchmark demonstrates not only "how much faster", but also "how important it is". Regards, > > Jeroen. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com -- INADA Naoki From tjreedy at udel.edu Tue Jul 10 01:58:43 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Tue, 10 Jul 2018 01:58:43 -0400 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft In-Reply-To: References: Message-ID: On 7/9/2018 9:00 PM, Guido van Rossum wrote: > We strongly prefer feedback in the form of Pull Requests to the peps > repo (the file is at > https://github.com/python/peps/blob/master/pep-0572.rst > ). I couple of people have said they don't know how to make pull requests against a PEP. I believe the following works with the web editor. 1. Go to url above. 2. Click upper right pencil icon. I don't know just who can do this. 3. Edit on the web page. It might be easier to read the html and then use browser find to change anything seen. 4. Note that there are two scrollbars -- one for webpage and one for editing box. 5. At the bottom of the webpage, click 'Create a new branch for this commit and start a PR'. This should unclick 'Commit directly to the master branch.' 6. Click Green [Propose file change] button. 7. I expect one will end up on the PR diff page. If not, click 'files changed to get there. Add comments explaining changes that need such. The problem I see with this is that I don't know of any way to accept or reject the different proposed changes within the PR. If there is, I would like to know how. -- Terry Jan Reedy From jcgoble3 at gmail.com Tue Jul 10 02:10:17 2018 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Tue, 10 Jul 2018 02:10:17 -0400 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft In-Reply-To: References: Message-ID: On Tue, Jul 10, 2018 at 2:00 AM Terry Reedy wrote: > On 7/9/2018 9:00 PM, Guido van Rossum wrote: > > We strongly prefer feedback in the form of Pull Requests to the peps > > repo (the file is at > > https://github.com/python/peps/blob/master/pep-0572.rst > > ). > > I couple of people have said they don't know how to make pull requests > against a PEP. I believe the following works with the web editor. > > 1. Go to url above. > 2. Click upper right pencil icon. I don't know just who can do this. > 3. Edit on the web page. It might be easier to read the html and then > use browser find to change anything seen. > 4. Note that there are two scrollbars -- one for webpage and one for > editing box. > 5. At the bottom of the webpage, click 'Create a new branch for this > commit and start a PR'. This should unclick 'Commit directly to the > master branch.' > 6. Click Green [Propose file change] button. > 7. I expect one will end up on the PR diff page. If not, click 'files > changed to get there. Add comments explaining changes that need such. > This is almost precisely how I, a newbie to contributing, fixed a typo in this PEP tonight. At step 2, anyone who is logged in can click the pencil icon; if you don't have write access to the repo, GitHub automatically forks the repo and commits your change to a new branch within your fork, eliminating step 5. The problem I see with this is that I don't know of any way to accept or > reject the different proposed changes within the PR. If there is, I > would like to know how. > I don't believe there is a way to merge part of a PR. The rejected changes would have to be reverted with a new commit prior to merging. -- > Terry Jan Reedy > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/jcgoble3%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Jul 10 02:55:29 2018 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 9 Jul 2018 23:55:29 -0700 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> <20180709182212.138466e7@fsol> Message-ID: On Mon, Jul 9, 2018 at 10:20 AM, Steve Dower wrote: > On 09Jul2018 0922, Antoine Pitrou wrote: >> >> On Mon, 9 Jul 2018 09:01:00 -0700 >> Steve Dower wrote: >>> >>> I've thought a bit about making a single installer that can offer the >>> option of 32-bit/64-bit at install time, but I don't actually think it's >>> that big a problem to deserve that much effort as a solution. >>> >>> Perhaps we should add non-button text below the button saying "Get the >>> 64-bit version"? >> >> >> Or perhaps the 32-bit installer could detect a 64-bit system and add >> an info box at the beginning? > > > That's not a bad idea. Needs a bpo issue, but shouldn't be too hard to do. > And it should be safe to backport for 3.7.1, as there's no real behaviour > change. And if the 64-bit installer could detect 32-bit systems and explain to users that they got the wrong version and direct them to the correct one, that would help anyone who does get confused by the 64-bit installer becoming more prominent. -n -- Nathaniel J. Smith -- https://vorpus.org From lists at andros.org.uk Tue Jul 10 06:03:03 2018 From: lists at andros.org.uk (Andrew McLean) Date: Tue, 10 Jul 2018 11:03:03 +0100 Subject: [Python-Dev] Encoding variable-length integers/counts in pickle In-Reply-To: <83f22260-f74d-6ea8-3464-6b3709f1b310@mrabarnett.plus.com> References: <83f22260-f74d-6ea8-3464-6b3709f1b310@mrabarnett.plus.com> Message-ID: Google Protocol Buffers use something similar, which they call " Base 128 Varints" https://developers.google.com/protocol-buffers/docs/encoding#varints I prefer the way this handles negative numbers. - Andrew On 10 July 2018 at 02:53, MRAB wrote: > In the regex module I use an encoding scheme when pickling pattern objects > which is based on the way MIDI encodes variable-length integers, and I > think it might have a use in a pickle protocol. > > In the basic format, an integer is split up into 7-bit chunks, each chunk > put into a byte, and the most-significant bit of the byte used to signal > whether the value continues into the following byte. > > And integer must be encoded into the minimum number of bytes, so an > encoded sequence of bytes would never start with 0x80. > > MIDI deviates from the basic idea in order to reduce the number of bytes, > so as sequence of bytes in MIDI _can_ start with x080; this is fine for > MIDI because it doesn't need to represent negative integers. > > The format I'm suggesting uses an initial 0x80 as a way of letting it > encode negative integers. > > Here are a couple of Python functions that summarise the encoding and > decoding (minus obvious optimisations for simplicity): > > > def encode_varint(value: int) -> List[int]: > negative = value < 0 > encoded = [] > > if negative: > final = -1 > else: > final = 0 > > while True: > encoded.append(0x80 | (value & 0x7F)) > value >>= 7 > > if value == final: > break > > if negative: > encoded.append(0x80) > > encoded.reverse() > > encoded[-1] &= 0x7F > > return encoded > > > def decode_varint(encoded: Iterable[int]) -> int: > byte = next(encoded) > > if byte == 0x80: > value = -1 > byte = next(encoded) > else: > value = 0 > > value = (value << 7) | (byte & 0x7F) > > while (byte & 0x80) != 0: > byte = next(encoded) > value = (value << 7) | (byte & 0x7F) > > return value > > > The advantage of encoding integers in this way is that there's no limit to > their size, so there's no need to add a new protocol to support larger > counts. > > They can also make pickles smaller. > > Example: > > # Pickle (None, ) > 0: \x80 PROTO 4 > 2: \x95 FRAME 4 > 11: N NONE > 12: \x85 TUPLE1 > 13: \x94 MEMOIZE (as 0) > 14: . STOP > > Here, FRAME takes an argument of 8 bytes. If you replaced FRAME with a > version that accepted a variable-length count, you could reduce that > argument to 1 byte. > > You also wouldn't need to have different fixed-length versions of an > opcode, e.g. BINUNICODE and SHORT_BINUNICODE. > > Whether you do anything with this is entirely up to the core devs, I just > thought someone might find it useful. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/lists% > 40andros.org.uk > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Jul 10 06:10:08 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Jul 2018 12:10:08 +0200 Subject: [Python-Dev] Encoding variable-length integers/counts in pickle References: <83f22260-f74d-6ea8-3464-6b3709f1b310@mrabarnett.plus.com> Message-ID: <20180710121008.30261bbe@fsol> On Tue, 10 Jul 2018 02:53:47 +0100 MRAB wrote: > In the regex module I use an encoding scheme when pickling pattern > objects which is based on the way MIDI encodes variable-length integers, > and I think it might have a use in a pickle protocol. > > In the basic format, an integer is split up into 7-bit chunks, each > chunk put into a byte, and the most-significant bit of the byte used to > signal whether the value continues into the following byte. > > And integer must be encoded into the minimum number of bytes, so an > encoded sequence of bytes would never start with 0x80. The problem with variable-length encoding is that you need more individual reads to fetch a datum. The whole point of pickle framing is to replace many small reads with a few large reads, and variable-length encoding is adversial to that goal. Regards Antoine. From J.Demeyer at UGent.be Tue Jul 10 07:53:01 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 10 Jul 2018 13:53:01 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> Message-ID: <5B449E1D.7000507@UGent.be> OK, I tried with --duplicate 200 and you can see the results at https://gist.github.com/jdemeyer/f0d63be8f30dc34cc989cd11d43df248 In short, the timings with and without PEP 580 are roughly the same (which is to be expected). Interestingly, a small but significant improvement can be seen when calling *unbound* methods. The real improvement comes from supporting a new calling protocol: formerly custom classes could only implement tp_call, but now they can use FASTCALL just like built-in functions/methods. For this, there is an improvement of roughly a factor 1.2 for calls without arguments, 1.6 for calls with positional arguments and 2.8 for calls with keywords. From songofacandy at gmail.com Tue Jul 10 08:59:28 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 10 Jul 2018 21:59:28 +0900 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <5B449E1D.7000507@UGent.be> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> Message-ID: On Tue, Jul 10, 2018 at 8:55 PM Jeroen Demeyer wrote: > > OK, I tried with --duplicate 200 and you can see the results at > https://gist.github.com/jdemeyer/f0d63be8f30dc34cc989cd11d43df248 > > In short, the timings with and without PEP 580 are roughly the same > (which is to be expected). Interestingly, a small but significant > improvement can be seen when calling *unbound* methods. > > The real improvement comes from supporting a new calling protocol: > formerly custom classes could only implement tp_call, but now they can > use FASTCALL just like built-in functions/methods. For this, there is an > improvement of roughly a factor 1.2 for calls without arguments, 1.6 for > calls with positional arguments and 2.8 for calls with keywords. We know it when we introduced FASTCALL. What I want know is "how often" tp_call in custom type is called in real application. Does it boost real application performance significantly? 5%? 10%? If it's not significant enough, I want to wait make FASTCALL public until more evolutionary optimization happened. There are some remaining possible optimizations. For example, let's assume cfunction like this: static PyObject* myfunc_impl(PyObject *self, Py_ssize_t i) { ... } static PyObject* myfunc(PyObject *self, PyObject *arg) { Py_ssize_t i; if (!PyArg_Parse(arg, "n;myfunc", &i)) { return NULL; } return myfunc_impl(self, i); } Then, the function is called from another C extension like this: PyObject_CallFunction(func, "n", 42); Currently, we create temporary long object for passing argument. If there is protocol for exposeing format used by PyArg_Parse*, we can bypass temporal Python object and call myfunc_impl directly. I think optimization like this idea can boost application performance using Cython heavily. But in Python and stdlib, there are no enough "call C function from C function" scenarios, compared with Cython based applications. We really need help from Cython world for this area. Regards, -- INADA Naoki From solipsis at pitrou.net Tue Jul 10 09:20:21 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 10 Jul 2018 15:20:21 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> Message-ID: <20180710152021.17d5c22d@fsol> On Tue, 10 Jul 2018 21:59:28 +0900 INADA Naoki wrote: > > Then, the function is called from another C extension like this: > > PyObject_CallFunction(func, "n", 42); > > Currently, we create temporary long object for passing argument. > If there is protocol for exposeing format used by PyArg_Parse*, we can > bypass temporal Python object and call myfunc_impl directly. This is another can of worms to open. If you're worried about the added complexity of PEP 580, what you're proposing is one or two orders of magnitude more complicated. > I think optimization like this idea can boost application performance > using Cython heavily. You can already define the C signature of a function in Cython and intra-Cython calls will benefit from it where possible. Cooperation from core Python is not necessary for that. The main point of PEP 580 is to make Cython functions faster when called from pure Python, not from other Cython functions. Regards Antoine. From J.Demeyer at UGent.be Tue Jul 10 09:29:41 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 10 Jul 2018 15:29:41 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <651a2fc283384d31872e225364dd983e@xmail103.UGent.be> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <651a2fc283384d31872e225364dd983e@xmail103.UGent.be> Message-ID: <5B44B4C5.70403@UGent.be> On 2018-07-10 14:59, INADA Naoki wrote: > Currently, we create temporary long object for passing argument. > If there is protocol for exposeing format used by PyArg_Parse*, we can > bypass temporal Python object and call myfunc_impl directly. Indeed, I already mentioned this at https://www.python.org/dev/peps/pep-0579/#allowing-native-c-arguments The way I see it, these kind of improvements should be done on top of PEP 580. Maybe I didn't emphasize this enough, but one of the goals of PEP 580 is to create an *extensible* protocol where such features can easily be added later in a compatible way. Jeroen. From ncoghlan at gmail.com Tue Jul 10 11:20:19 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 11 Jul 2018 01:20:19 +1000 Subject: [Python-Dev] PEP 572: Assignment Expressions -- intention to accept, near-final draft In-Reply-To: References: Message-ID: On 10 July 2018 at 11:00, Guido van Rossum wrote: > A lot has been said about PEP 572. I am planning to accept it soon, > hopefully within a week. I realize I should have posted the draft from May > 22 (when Tim and I were added as authors and it was significantly updated -- > see https://github.com/python/peps/pull/654). For this I apologize. Since > then we've been diligently updating various details, tightening the spec > without really changing the intended design. Thanks for the updates, especially the new Appendix B that walks through the comprehension scoping cases with worked examples. I've submitted a couple of minor PRs, but this version of the PEP looks solid to me, and makes the case for change well (I'm still only a +0 personally, but I can also see myself some day saying "Yeah, that turned out to be worth it in the long run") Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From vstinner at redhat.com Tue Jul 10 18:41:43 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 11 Jul 2018 00:41:43 +0200 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> Message-ID: 2018-07-09 18:01 GMT+02:00 Steve Dower : > The difficulty is that they *definitely* can use the 32-bit version, and > those few who are on older machines or older installs of Windows may not > understand why the link we provide didn't work for them. Let's say that only 10% of users still use 32-bit version. If they download a default 64-bit binary, I'm quite sure that running the binary will emit an error no? Such users should be used to such error, and be able to get the 64-bit version, no? Victor From vstinner at redhat.com Tue Jul 10 18:47:08 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 11 Jul 2018 00:47:08 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> Message-ID: 2018-07-10 14:59 GMT+02:00 INADA Naoki : > PyObject_CallFunction(func, "n", 42); > > Currently, we create temporary long object for passing argument. > If there is protocol for exposeing format used by PyArg_Parse*, we can > bypass temporal Python object and call myfunc_impl directly. I'm not sure that it's worth it. It seems complex to implement. I proposed something simpler, but nobody tried to implement it. Instead of calling the long and complex PyArg_Parse...() functions, why not generating C code to parse arguments instead? The idea looks like "inlining" PyArg_Parse...() in its caller, but technically it means that Argument Clinic generates C code to parse arguments. PyArg_Parse...() is cheap and has been optimized, but on very fast functions (less than 100 ns), it might be significant. Well, to be sure, someone should run a benchmark :-) Victor From vstinner at redhat.com Tue Jul 10 18:48:13 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 11 Jul 2018 00:48:13 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <5B449E1D.7000507@UGent.be> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> Message-ID: About your benchmark results: "FASTCALL unbound method(obj, 1, two=2): Mean +- std dev: 42.6 ns +- 29.6 ns" That's a very big standard deviation :-( Are you using CPU pinning and other technics explaining in my doc? http://perf.readthedocs.io/en/latest/run_benchmark.html#how-to-get-reproductible-benchmark-results Victor 2018-07-10 13:53 GMT+02:00 Jeroen Demeyer : > OK, I tried with --duplicate 200 and you can see the results at > https://gist.github.com/jdemeyer/f0d63be8f30dc34cc989cd11d43df248 > > In short, the timings with and without PEP 580 are roughly the same (which > is to be expected). Interestingly, a small but significant improvement can > be seen when calling *unbound* methods. > > The real improvement comes from supporting a new calling protocol: formerly > custom classes could only implement tp_call, but now they can use FASTCALL > just like built-in functions/methods. For this, there is an improvement of > roughly a factor 1.2 for calls without arguments, 1.6 for calls with > positional arguments and 2.8 for calls with keywords. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com From vstinner at redhat.com Tue Jul 10 18:50:58 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 11 Jul 2018 00:50:58 +0200 Subject: [Python-Dev] Micro-benchmarks for function calls (PEP 576/579/580) In-Reply-To: <5B43DFAE.6060905@UGent.be> References: <5B43DFAE.6060905@UGent.be> Message-ID: The pyperformance benchmark suite had micro benchmarks on function calls, but I removed them because they were sending the wrong signal. A function call by itself doesn't matter to compare two versions of CPython, or CPython to PyPy. It's also very hard to measure the cost of a function call when you are using a JIT compiler which is able to inline the code into the caller... So I removed all these stupid "micro benchmarks" to a dedicated Git repository: https://github.com/vstinner/pymicrobench Sometimes, I add new micro benchmarks when I work on one specific micro optimization. But more generally, I suggest you to not run micro benchmarks and avoid micro optimizations :-) Victor 2018-07-10 0:20 GMT+02:00 Jeroen Demeyer : > Here is an initial version of a micro-benchmark for C function calling: > > https://github.com/jdemeyer/callbench > > I don't have results yet, since I'm struggling to find the right options to > "perf timeit" to get a stable result. If somebody knows how to do this, help > is welcome. > > > Jeroen. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com From vstinner at redhat.com Tue Jul 10 19:14:42 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 11 Jul 2018 01:14:42 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: References: <5B3E30D1.3080706@UGent.be> Message-ID: 2018-07-07 0:26 GMT+02:00 Victor Stinner : > I designed FASTCALL with the help of Serhiy for keywords. I prepared a long > email reply, but I found an opportunity for optimisation on **kwargs and I > need time to see how to optimize it. I just created: "Python function call optimization: avoid temporary tuple to pass **kwargs" https://bugs.python.org/issue34090 Victor From vstinner at redhat.com Tue Jul 10 19:20:12 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 11 Jul 2018 01:20:12 +0200 Subject: [Python-Dev] On the METH_FASTCALL calling convention In-Reply-To: References: <5B3E30D1.3080706@UGent.be> Message-ID: 2018-07-07 10:55 GMT+02:00 Serhiy Storchaka : > There is my idea. Split every of keyword argument parsing functions on two > parts. The first part linearize keyword arguments, it converts positional > and keyword arguments (in whatever form they were presented) into a linear > array of PyObject* (with NULLs for not specified optional arguments). The > second part is common and similar to _PyArg_ParseStack(), but supports > NULLs. It converts an array of PyObject* to a sequence of C values. I tried > to implement this idea, is is not simple, and results were mixed, but I > don't loss a hope. > > And here we return to METH_FASTCALL|METH_KEYWORDS. The first part of > handling arguments can be made outside of the C function, by the calling > API. Then the signature of the C function can be simpler, the same as for > METH_FASTCALL. But we need to expose the list of keyword parameter names as > an attribute of CFunction. > > I don't know whether this ides is vital or dead, but I' going to try it. And > implementing it will change the METH_FASTCALL|METH_KEYWORDS calling > convention a lot. I don't understand why changing the implementation of PyArg_Parse*() would require to require the FASTCALL calling convention? Victor From songofacandy at gmail.com Tue Jul 10 20:01:47 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Wed, 11 Jul 2018 09:01:47 +0900 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> Message-ID: On Wed, Jul 11, 2018 at 7:47 AM Victor Stinner wrote: > > 2018-07-10 14:59 GMT+02:00 INADA Naoki : > > PyObject_CallFunction(func, "n", 42); > > > > Currently, we create temporary long object for passing argument. > > If there is protocol for exposeing format used by PyArg_Parse*, we can > > bypass temporal Python object and call myfunc_impl directly. > > I'm not sure that it's worth it. It seems complex to implement. > Both of my idea and PEP 580 are complicated. For Python stdlibs, I expect no significant benefit. We already bypass Python function calling by typecheck + concrete function call idiom. But for Cython users, especially people using Cython on Jupyter, I expect there are many extension-to-extension calls. Both of this idea and PEP 580 is complicated. And we don't have realistic example to demonstrate real world benefit of them. Without application benchmark, I think both of idea and PEP 580 shouldn't be happened. That's why I requested application benchmark again and again. PEP 576 seems minimalistic, straightforward way to allow FASTCALL for Cython and other 3rd party libraries. But if we accept PEP 576, it makes harder to allow more optimization in the future. I expect best balance is between PEP 576 and 580. Maybe, adding new slot as struct pointer with some flags, but don't add per-instance data. But I'm not sure because I'm not data scientist. I don't know what's the typical usage and where is main bottleneck of their application. Jeroen seems want we to discuss on PEP 576 and 580. So I explained to him why we need example application first. > I proposed something simpler, but nobody tried to implement it. > Instead of calling the long and complex PyArg_Parse...() functions, > why not generating C code to parse arguments instead? The idea looks > like "inlining" PyArg_Parse...() in its caller, but technically it > means that Argument Clinic generates C code to parse arguments. > I believe Cython did it already. But I have worrying about it. If we do it for all function, it makes Python binary fatter, consume more CPU cache. Once CPU cache is start stashing, application performance got slower quickly. And benchmarking CPU cache efficiency is very difficult. Current Python benchmark is too small. We benchmarked HTTP server, SQLAlchemy, JSON, template engine individually. But real application do all of them in loop. And many processes share L3 cache. Even L1 cache is shared by several processes by HyperThreading and context switch. > PyArg_Parse...() is cheap and has been optimized, but on very fast > functions (less than 100 ns), it might be significant. Well, to be > sure, someone should run a benchmark :-) > > Victor -- INADA Naoki From songofacandy at gmail.com Tue Jul 10 20:12:11 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Wed, 11 Jul 2018 09:12:11 +0900 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <20180710152021.17d5c22d@fsol> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> Message-ID: On Tue, Jul 10, 2018 at 10:20 PM Antoine Pitrou wrote: > > On Tue, 10 Jul 2018 21:59:28 +0900 > INADA Naoki wrote: > > > > Then, the function is called from another C extension like this: > > > > PyObject_CallFunction(func, "n", 42); > > > > Currently, we create temporary long object for passing argument. > > If there is protocol for exposeing format used by PyArg_Parse*, we can > > bypass temporal Python object and call myfunc_impl directly. > > This is another can of worms to open. If you're worried about the > added complexity of PEP 580, what you're proposing is one or two orders > of magnitude more complicated. This is just an example of possible optimization, to explain why I want example application first. I know Cython is important for data scientists. But I don't know how it used typically. If my idea has 50% gain and current PEP 580 has only 5% gain, why we should accept PEP 580? But no one know real gain, because there are no realistic application which bottleneck is calling overhead. Without example application, I can't consider PEP 580 seriously. Microbenchemarks doesn't attract me. And PEP 576 seems much simpler and straightforward way to expose FASTCALL. > > > I think optimization like this idea can boost application performance > > using Cython heavily. > > You can already define the C signature of a function in Cython and > intra-Cython calls will benefit from it where possible. Cooperation > from core Python is not necessary for that. Why not allow it for extensions written in C, without Cython? > > The main point of PEP 580 is to make Cython functions faster when > called from pure Python, not from other Cython functions. Really? If so, I prefer PEP 576 to PEP 580. PEP 580 is too complicated. Anyway, I want more example applications which uses Cython heavily for further discussion. Regards, -- INADA Naoki From vano at mail.mipt.ru Tue Jul 10 22:14:34 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 11 Jul 2018 05:14:34 +0300 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> Message-ID: On 11.07.2018 1:41, Victor Stinner wrote: > 2018-07-09 18:01 GMT+02:00 Steve Dower : >> The difficulty is that they *definitely* can use the 32-bit version, and >> those few who are on older machines or older installs of Windows may not >> understand why the link we provide didn't work for them. > Let's say that only 10% of users still use 32-bit version. If they > download a default 64-bit binary, I'm quite sure that running the > binary will emit an error no? Such users should be used to such error, > and be able to get the 64-bit version, no? Attached the image of what happens. The message is: "One or more issues caused the setup to fail. Please fix the issues and the retry setup. For more information see the log file . 0x80070661 - This installation package is not supported by this processor type. Contact your product vendor." Pretty descriptive in my book. > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -------------- next part -------------- A non-text attachment was scrubbed... Name: x64.png Type: image/png Size: 103485 bytes Desc: not available URL: From stefan_ml at behnel.de Wed Jul 11 01:34:27 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Jul 2018 07:34:27 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> Message-ID: INADA Naoki schrieb am 11.07.2018 um 02:12: > On Tue, Jul 10, 2018 at 10:20 PM Antoine Pitrou wrote: >> On Tue, 10 Jul 2018 21:59:28 +0900 >> INADA Naoki wrote: >>> >>> Then, the function is called from another C extension like this: >>> >>> PyObject_CallFunction(func, "n", 42); >>> >>> Currently, we create temporary long object for passing argument. >>> If there is protocol for exposeing format used by PyArg_Parse*, we can >>> bypass temporal Python object and call myfunc_impl directly. Note that this is not fast at all. It actually has to parse the format description at runtime. For really fast calls, this should be avoided, and it can be avoided by using a str object for the signature description and interning it. That relies on signature normalisation, which requires a proper formal specification of C/C++ signature strings, which ... is pretty much the can of worms that Antoine mentioned. >> This is another can of worms to open. If you're worried about the >> added complexity of PEP 580, what you're proposing is one or two orders >> of magnitude more complicated. > > This is just an example of possible optimization, to explain why I want > example application first. > I know Cython is important for data scientists. But I don't know how > it used typically. > > If my idea has 50% gain and current PEP 580 has only 5% gain, > why we should accept PEP 580? Because PEP 580 is also meant as a preparation for a fast C-to-C call interface in Python. Unpacking C callables is quite an involved protocol, and we have been pushing the idea around and away in the Cython project for some seven years. It's about time to consider it more seriously now, and there are plans to implement it on top of PEP 580 (which is also mentioned in the PEP). > And PEP 576 seems much simpler and straightforward way to expose > FASTCALL. Except that it does not get us one step forward on the path towards what you proposed. So, why would *you* prefer it over PEP 580? >>> I think optimization like this idea can boost application performance >>> using Cython heavily. >> >> You can already define the C signature of a function in Cython and >> intra-Cython calls will benefit from it where possible. Cooperation >> from core Python is not necessary for that. > > Why not allow it for extensions written in C, without Cython? It should be. They just need a simpler protocol, which is PEP 580. Stefan From steve at pearwood.info Wed Jul 11 01:39:50 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 11 Jul 2018 15:39:50 +1000 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> Message-ID: <20180711053949.GQ7318@ando.pearwood.info> On Wed, Jul 11, 2018 at 05:14:34AM +0300, Ivan Pozdeev via Python-Dev wrote: > On 11.07.2018 1:41, Victor Stinner wrote: > >2018-07-09 18:01 GMT+02:00 Steve Dower : > >>The difficulty is that they *definitely* can use the 32-bit version, and > >>those few who are on older machines or older installs of Windows may not > >>understand why the link we provide didn't work for them. I think Steve's comment is right on the money. Although professional programmers should be a bit more technically competent than the average user, many are just hobbiest programmers, or school kids who are just as clueless as the average user since they *are* average users. > >Let's say that only 10% of users still use 32-bit version. If they > >download a default 64-bit binary, I'm quite sure that running the > >binary will emit an error no? Such users should be used to such error, > >and be able to get the 64-bit version, no? That's a lot of assumptions there. Here are a few things which might break those assumptions: - inexperienced users who haven't installed much software; - or who don't know that 32- versus 64-bit is a thing; - conservative users who don't install much software aside from using their vendor's official packages; - users who install from app stores which automatically detect the right version of the installer so they don't need to think about it (how do app stores handle this issue?); - or those who expect the default installer to work by default, so long as they pick the right OS (Windows or Mac). I don't remember what CPU my PC has, and when I can't be bothered to look it up, I always go for the default install option expecting that it ought to work regardless of whether I have a 32- or 64-bit OS. So far that's a strategy that has never done me wrong :-) > Attached the image of what happens. The message is: > > "One or more issues caused the setup to fail. Please fix the issues and > the retry setup. For more information see the log file . > > 0x80070661 - This installation package is not supported by this > processor type. Contact your product vendor." > > Pretty descriptive in my book. Are you being sarcastic? I would expect that "this processor type" refers to incompatible chip sets like ARM versus Intel, not the 32- versus 64-bitness of the operating system. And I certainly wouldn't associate the problem: "I downloaded and ran the wrong installer" with the appropriate solution: "I need to hunt for a 32-bit installer, rather than using the default" given that error message. -- Steve From stefan_ml at behnel.de Wed Jul 11 01:39:05 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Wed, 11 Jul 2018 07:39:05 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> Message-ID: INADA Naoki schrieb am 11.07.2018 um 02:01: > On Wed, Jul 11, 2018 at 7:47 AM Victor Stinner wrote: >> I proposed something simpler, but nobody tried to implement it. >> Instead of calling the long and complex PyArg_Parse...() functions, >> why not generating C code to parse arguments instead? The idea looks >> like "inlining" PyArg_Parse...() in its caller, but technically it >> means that Argument Clinic generates C code to parse arguments. > > But I have worrying about it. If we do it for all function, it makes Python > binary fatter, consume more CPU cache. Once CPU cache is start > stashing, application performance got slower quickly. Now, I'd like to see benchmark numbers for that before I believe it. Macro benchmarks, not micro benchmarks! *wink* Stefan From turnbull.stephen.fw at u.tsukuba.ac.jp Wed Jul 11 02:14:12 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Wed, 11 Jul 2018 15:14:12 +0900 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> Message-ID: <23365.41012.710196.224983@turnbull.sk.tsukuba.ac.jp> Ivan Pozdeev via Python-Dev writes: > "One or more issues caused the setup to fail. Please fix the issues and > the retry setup. For more information see the log file . > > 0x80070661 - This installation package is not supported by this > processor type. Contact your product vendor." > > Pretty descriptive in my book. Experience shows that's definitely not descriptive enough for my university's students (and starting from AY 2021 we're moving to Python 3 as the university-wide programming course language, yay!) They have no idea that "processor type" means "word size", or what alternative package to look for. Sometimes they take the "contact vendor" wording to mean "package is broken". I don't think the Japanese or Chinese students will have 32-bit machines (haven't seen one among my advisees since March 2016), but we do get some students from less wealthy countries who may be using older machines. So I think it would be really nice if the installer detects the wordsize mismatch, and issues an explicit message like This package is intended for a 64-it machine, but yours is a 32-bit machine. Please download and install the package specifically for 32-bit machines instead. From v+python at g.nevcal.com Wed Jul 11 02:28:44 2018 From: v+python at g.nevcal.com (Glenn Linderman) Date: Tue, 10 Jul 2018 23:28:44 -0700 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: <23365.41012.710196.224983@turnbull.sk.tsukuba.ac.jp> References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> <23365.41012.710196.224983@turnbull.sk.tsukuba.ac.jp> Message-ID: <53bafa5f-b8bc-903a-d2dc-6d6b7524a743@g.nevcal.com> On 7/10/2018 11:14 PM, Stephen J. Turnbull wrote: > Ivan Pozdeev via Python-Dev writes: > > > "One or more issues caused the setup to fail. Please fix the issues and > > the retry setup. For more information see the log file . > > > > 0x80070661 - This installation package is not supported by this > > processor type. Contact your product vendor." > > > > Pretty descriptive in my book. > > Experience shows that's definitely not descriptive enough for my > university's students (and starting from AY 2021 we're moving to > Python 3 as the university-wide programming course language, yay!) > They have no idea that "processor type" means "word size", or what > alternative package to look for. Sometimes they take the "contact > vendor" wording to mean "package is broken". I don't think the > Japanese or Chinese students will have 32-bit machines (haven't seen > one among my advisees since March 2016), but we do get some students > from less wealthy countries who may be using older machines. > > So I think it would be really nice if the installer detects the > wordsize mismatch, and issues an explicit message like > > This package is intended for a 64-it machine, but yours is a 32-bit > machine. > > Please download and install the package specifically for 32-bit > machines instead. Which would be far, far better, regardless of which bitness(es) of installer is(are) displayed (prominently) on the web site. -------------- next part -------------- An HTML attachment was scrubbed... URL: From agriff at tin.it Wed Jul 11 03:19:44 2018 From: agriff at tin.it (Andrea Griffini) Date: Wed, 11 Jul 2018 09:19:44 +0200 Subject: [Python-Dev] Micro-benchmarks for function calls (PEP 576/579/580) In-Reply-To: References: <5B43DFAE.6060905@UGent.be> Message-ID: May be is something obvious but I find myself forgetting often about the fact that most modern CPUs can change speed (and energy consumption) depending on a moving average of CPU load. If you don't disable this "green" feature and the benchmarks are quick then the result can have huge variations depending on exactly when and if the CPU switches to fast mode. On Wed, Jul 11, 2018 at 12:53 AM Victor Stinner wrote: > The pyperformance benchmark suite had micro benchmarks on function > calls, but I removed them because they were sending the wrong signal. > A function call by itself doesn't matter to compare two versions of > CPython, or CPython to PyPy. It's also very hard to measure the cost > of a function call when you are using a JIT compiler which is able to > inline the code into the caller... So I removed all these stupid > "micro benchmarks" to a dedicated Git repository: > https://github.com/vstinner/pymicrobench > > Sometimes, I add new micro benchmarks when I work on one specific > micro optimization. > > But more generally, I suggest you to not run micro benchmarks and > avoid micro optimizations :-) > > Victor > > 2018-07-10 0:20 GMT+02:00 Jeroen Demeyer : > > Here is an initial version of a micro-benchmark for C function calling: > > > > https://github.com/jdemeyer/callbench > > > > I don't have results yet, since I'm struggling to find the right options > to > > "perf timeit" to get a stable result. If somebody knows how to do this, > help > > is welcome. > > > > > > Jeroen. > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/agriff%40tin.it > -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Wed Jul 11 03:53:58 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Wed, 11 Jul 2018 16:53:58 +0900 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> Message-ID: First of all, please don't be so defensive. I just say I need example target application. I don't against to PEP 580. Actually speaking, I lean to PEP 580 than PEP 576, although I wonder if some part of PEP 580 can be simplified or postponed. But PEP 580 is very complicated. I need enough evidence what PEP 580 provides before voting to PEP 580. I know Python is important for data scientists and Cython is important for them too. But I don't have any examples target applications because I'm not data scientist and I don't use Jupyter, numpy, etc. Python's performance test suite doesn't contain such applications too. So we can't measure or estimate benefits. That's why I requested real world application sample again and again. In my experience, I need Cython for making hotspot faster. Calling overhead is much smaller than the hotspot. I haven't focused on inter extension calling performance because `cdef` or `cpdef` is enough. So I really don't have any target application which bottleneck is calling performance of Cython. > >>> Currently, we create temporary long object for passing argument. > >>> If there is protocol for exposeing format used by PyArg_Parse*, we can > >>> bypass temporal Python object and call myfunc_impl directly. > > Note that this is not fast at all. It actually has to parse the format > description at runtime. For really fast calls, this should be avoided, and > it can be avoided by using a str object for the signature description and > interning it. That relies on signature normalisation, which requires a > proper formal specification of C/C++ signature strings, which ... is pretty > much the can of worms that Antoine mentioned. > Please don't start discussion about detail. This is just an example of possible optimization of future. (And I'm happy about hearing Cython will tackling this). > > If my idea has 50% gain and current PEP 580 has only 5% gain, > > why we should accept PEP 580? > > Because PEP 580 is also meant as a preparation for a fast C-to-C call > interface in Python. > > Unpacking C callables is quite an involved protocol, and we have been > pushing the idea around and away in the Cython project for some seven > years. It's about time to consider it more seriously now, and there are > plans to implement it on top of PEP 580 (which is also mentioned in the PEP). > I want to see it before accepting PEP 580. > > > And PEP 576 seems much simpler and straightforward way to expose > > FASTCALL. > > Except that it does not get us one step forward on the path towards what > you proposed. So, why would *you* prefer it over PEP 580? > I prefer PEP 580! I just don't have enough rational to accept PEP 580 complexities. >> >> But I have worrying about it. If we do it for all function, it makes Python >> binary fatter, consume more CPU cache. Once CPU cache is start >> stashing, application performance got slower quickly. > > Now, I'd like to see benchmark numbers for that before I believe it. Macro > benchmarks, not micro benchmarks! *wink* Yes, when I try inlining argument parsing or other optimization having significant memory overhead, I'll try to macro benchmark of cache efficiency. But for now, I'm working on making Python memory footprint smaller, not fatter. Regards, -- INADA Naoki From p.f.moore at gmail.com Wed Jul 11 04:08:21 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 11 Jul 2018 09:08:21 +0100 Subject: [Python-Dev] why is not 64-bit installer the default download link for Windows? In-Reply-To: <20180711053949.GQ7318@ando.pearwood.info> References: <96bc6294-89a1-baaa-caad-5d4800074435@python.org> <20180711053949.GQ7318@ando.pearwood.info> Message-ID: On 11 July 2018 at 06:39, Steven D'Aprano wrote: > On Wed, Jul 11, 2018 at 05:14:34AM +0300, Ivan Pozdeev via Python-Dev wrote: >> On 11.07.2018 1:41, Victor Stinner wrote: >> >2018-07-09 18:01 GMT+02:00 Steve Dower : >> >>The difficulty is that they *definitely* can use the 32-bit version, and >> >>those few who are on older machines or older installs of Windows may not >> >>understand why the link we provide didn't work for them. > > I think Steve's comment is right on the money. > > Although professional programmers should be a bit more technically > competent than the average user, many are just hobbiest programmers, or > school kids who are just as clueless as the average user since they > *are* average users. I'm perfectly happy for the default installer that you get from the obvious first choice button (the one that says "Python 3.7.0") on the "Downloads" drop-down to be the 32-bit installer[1]. But making people who know they want the 64-bit installer click through "View the full list of downloads" -> "Release 3.7.0", scroll down to the bottom of a page that looks more like a release note if you just glance at the top, and find "Windows x86-64 executable installer" is a bit much. And the convoluted route is a nightmare for people like me to explain when I'm trying to tell people who I know should be getting the 64-bit version, how to do so. Which is why I'd like to see a bit more choice on that initial dropdown. Just a second button for the 64-bit version is enough - for the full lists the set of links to the left of the dropdown is fine. Paul [1] Although I strongly dislike the fact that there's no indication at all in that dropdown that what you're getting *is* the 32 bit version, short of hovering over the link and knowing the naming convention of the installers :-(. From solipsis at pitrou.net Wed Jul 11 04:27:40 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 11 Jul 2018 10:27:40 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> Message-ID: <20180711102740.520211f7@fsol> On Wed, 11 Jul 2018 09:12:11 +0900 INADA Naoki wrote: > > Without example application, I can't consider PEP 580 seriously. > Microbenchemarks doesn't attract me. > And PEP 576 seems much simpler and straightforward way to expose > FASTCALL. I agree PEP 580 is extremely complicated and it's not obvious what the maintenance burden will be in the long term. Regards Antoine. From vstinner at redhat.com Wed Jul 11 04:42:10 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 11 Jul 2018 10:42:10 +0200 Subject: [Python-Dev] Micro-benchmarks for function calls (PEP 576/579/580) In-Reply-To: References: <5B43DFAE.6060905@UGent.be> Message-ID: 2018-07-11 9:19 GMT+02:00 Andrea Griffini : > May be is something obvious but I find myself forgetting often about > the fact that most modern CPUs can change speed (and energy consumption) > depending on a moving average of CPU load. > > If you don't disable this "green" feature and the benchmarks are quick then > the > result can have huge variations depending on exactly when and if the CPU > switches to fast mode. If you use "sudo python3 -m perf system tune": Turbo Boost is disabled and the CPU frequency is fixed. More into at: http://perf.readthedocs.io/en/latest/system.html Victor From vstinner at redhat.com Wed Jul 11 04:50:15 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 11 Jul 2018 10:50:15 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> Message-ID: 2018-07-11 2:12 GMT+02:00 INADA Naoki : > If my idea has 50% gain and current PEP 580 has only 5% gain, > why we should accept PEP 580? > But no one know real gain, because there are no realistic application > which bottleneck is calling overhead. I'm skeptical about "50% gain": I want to see a working implementation and reproduce benchmarks myself to believe that :-) As you wrote, the cost of function costs is unlikely the bottleneck of application. Sorry, I didn't read all these PEPs about function calls, but IMHO a minor speedup on micro benchmarks must not drive a PEP. If someone wants to work on CPython performance, I would suggest to think bigger and target 2x speedup on applications. To get to this point, the bottleneck is the C API and so we have to fix our C API first. http://vstinner.readthedocs.io/python_new_stable_api.html Victor From ncoghlan at gmail.com Wed Jul 11 08:56:57 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 11 Jul 2018 22:56:57 +1000 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> Message-ID: On 11 July 2018 at 18:50, Victor Stinner wrote: > I'm skeptical about "50% gain": I want to see a working implementation > and reproduce benchmarks myself to believe that :-) As you wrote, the > cost of function costs is unlikely the bottleneck of application. > > Sorry, I didn't read all these PEPs about function calls, but IMHO a > minor speedup on micro benchmarks must not drive a PEP. If someone > wants to work on CPython performance, I would suggest to think bigger > and target 2x speedup on applications. To get to this point, the > bottleneck is the C API and so we have to fix our C API first. Folks, I'd advise against focusing too heavily on CPython performance when reviewing PEP 580, as PEP 580 is *not primarily about CPython performance*. The key optimisations it enables have already been implemented in the form of FASTCALL, so nothing it does is going to make CPython faster. Instead, we're being approached in our role as the *central standards management group for the Python ecosystem*, similar to the way we were involved in the establishment of PEP 3118 as the conventional mechanism for zero-copy data sharing. While Stefan Krah eventually brought memoryview up to speed as a first class PEP 3118 buffer exporter and consumer, adding the memoryview builtin wasn't the *point* of that PEP - the point of the PEP was to let libraries like NumPy and PIL share the same backing memory across different Python objects without needing to know about each other directly. The request being made is a similar case of ecosystem enablement - it's less about the performance of any specific case (although that's certainly a significant intended benefit), and more about providing participants in the Python ecosystem more freedom to architect their projects in the way that makes the most sense from an ongoing maintenance perspective, without there being a concrete and measurable performance *penalty* in breaking a large monolithic extension module up into smaller independently updatable components. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From desmoulinmichel at gmail.com Wed Jul 11 09:13:02 2018 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Wed, 11 Jul 2018 15:13:02 +0200 Subject: [Python-Dev] A more flexible task creation In-Reply-To: References: Message-ID: > To be honest, I see "async with" being abused everywhere in asyncio, > lately.? I like to have objects with start() and stop() methods, but > everywhere I see async context managers.> > Fine, add nursery or whatever, but please also have a simple start() / > stop() public API. > > "async with" is only good for functional programming.? If you want to go > more of an object-oriented style, you tend to have start() and stop() > methods in your classes, which will call start() & stop() (or close()) > methods recursively on nested resources.? So of the libraries (aiopg, > I'm looking at you) don't support start/stop or open/close well. Wouldn't calling __enter__ and __exit__ manually works for you ? I started coding begin() and stop(), but I removed them, as I couldn't find a use case for them. And what exactly is the use case that doesn't work with `async with` ? The whole point is to spot the boundaries of the tasks execution easily. If you start()/stop() randomly, it kinda defeat the purpose. It's a genuine question though. I can totally accept I overlooked a valid use case. > > I tend to slightly agree, but OTOH if asyncio had been designed to not > schedule tasks automatically on __init__ I bet there would have been > other users complaining that "why didn't task XX run?", or "why do tasks > need a start() method, that is clunky!".? You can't please everyone... Well, ensure_future([schedule_immediatly=True]) and asyncio.create_task([schedule_immediatly=True] would take care of that. They are the entry point for the task creation and scheduling. > > Also, in > ?? ? ? ? ? ? task_list = run.all(foo(), foo(), foo()) > > As soon as you call?foo(),?you are instantiating a coroutine, which > consumes memory, while the task may not even be scheduled for a long > time (if you have 5000 potential tasks but only execute 10 at a time, > for example). Yes but this has the benefit of accepting any awaitable, not just coroutine. You don't have to wonder what to pass, or which form. It's always the same. Too many APi are hard to understand because you never know if it accept a callback, a coroutine function, a coroutine, a task, a future... For the same reason, request.get() create and destroys a session every time. It's inefficient, but way easier to understand, and fits the majority of the use cases. > > But if you do as Yuri suggested, you'll instead accept a function > reference, foo, which is a singleton, you can have many foo references > to the function, but they will only create coroutine objects when the > task is actually about to be scheduled, so it's more efficient in terms > of memory. I made some test, and the memory consumption is indeed radically smaller if you just store references if you just compare it to the same unique raw coroutine. However, this is a rare case. It assumes that: - you have a lot of tasks - you have a max concurrency - the max concurrency is very small - most tasks reuse a similar combination of callables and parameters It's a very specific narrow case. Also, everything you store on the scope will be wrapped into a Future object no matter if it's scheduled or not, so that you can cancel it later. So the scale of the memory consumption is not as much. I didn't want to compromise the quality of the current API for the general case for an edge case optimization. On the other hand, this is a low hanging fruit and on plateforms such as raspi where asyncio has a lot to offer, it can make a big difference to shave up 20 of memory consumption of a specific workload. So I listened and implemented an escape hatch: import random import asyncio import ayo async def zzz(seconds): await asyncio.sleep(seconds) print(f'Slept for {seconds} seconds') @ayo.run_as_main() async def main(run_in_top): async with ayo.scope(max_concurrency=10) as run: for _ in range(10000): run.from_callable(zzz, 0.005) # or run.asap(zzz(0.005)) This would only lazily create the awaitable (here the coroutine) on scheduling. I see a 15% of memory saving for the WHOLE program if using `from_callable()`. So definitly a good feature to have, thank you. But again, and I hope Yuri is reading this because he will implement that for uvloop, and this will trickles down to asyncio, I think we should not compromise the main API for this. asyncio is hard enough to grok, and too many concepts fly around. The average Python programmer has been experienced way easier things from past Python encounter. If we want, one day, that asyncio is consider the clean AND easy way to do async, we need to work on the API. asyncio.run() is a step in the right direction (although again I wish we implemented that 2 years ago when I talked about it instead of telling me no). Now if we add nurseries, it should hide the rest of the complexity. Not add to it. From J.Demeyer at UGent.be Wed Jul 11 15:28:42 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 11 Jul 2018 21:28:42 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <51fd727c8cbd4a19bfa167b328dee762@xmail103.UGent.be> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <51fd727c8cbd4a19bfa167b328dee762@xmail103.UGent.be> Message-ID: <5B465A6A.4050405@UGent.be> On 2018-07-11 00:48, Victor Stinner wrote: > About your benchmark results: > > "FASTCALL unbound method(obj, 1, two=2): Mean +- std dev: 42.6 ns +- 29.6 ns" > > That's a very big standard deviation :-( Yes, I know. My CPU was overheating and was slowed down. But that seemed to have happened for a small number of benchmarks only. But given that you find these benchmarks stupid anyway, I assume that you don't really care. Jeroen. From J.Demeyer at UGent.be Wed Jul 11 15:45:25 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 11 Jul 2018 21:45:25 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <3c98f25afa6547b9a93c66dfd875f2c4@xmail103.UGent.be> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> <3c98f25afa6547b9a93c66dfd875f2c4@xmail103.UGent.be> Message-ID: <5B465E55.2090301@UGent.be> On 2018-07-11 10:27, Antoine Pitrou wrote: > I agree PEP 580 is extremely complicated and it's not obvious what the > maintenance burden will be in the long term. But the status quo is also very complicated! If somebody would write a PEP describing the existing implementation of builtin_function_or_method and method_descriptor with all its optimizations, probably you would also find it complicated. Have you actually looked at the existing implementation in Python/ceval.c and Object/call.c for calling objects? One of the things that PEP 580 offers is replacing 5 (yes, five!) functions _PyCFunction_FastCallKeywords, _PyCFunction_FastCallDict, _PyMethodDescr_FastCallKeywords, _PyMethodDef_RawFastCallKeywords, _PyMethodDef_RawFastCallDict by a single function PyCCall_FASTCALL. Anyway, it would help if you could say why you (and others) think that it's complicated. Sure, there are many details to be considered (for example, the section about Descriptor behavior), but those are not essential to understand what the PEP does. I wrote the PEP as a complete specification, give full details. Maybe I should add a section just explaining the core ideas without details? Jeroen. From J.Demeyer at UGent.be Wed Jul 11 15:54:52 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Wed, 11 Jul 2018 21:54:52 +0200 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <3ec875cdb550457d869094869bce5178@xmail103.UGent.be> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> <3ec875cdb550457d869094869bce5178@xmail103.UGent.be> Message-ID: <5B46608C.7010000@UGent.be> On 2018-07-11 10:50, Victor Stinner wrote: > As you wrote, the > cost of function costs is unlikely the bottleneck of application. With that idea, METH_FASTCALL is not needed either. I still find it very strange that nobody seems to question all the crazy existing optimizations for function calls in CPython, yet claiming at the same time that those are just stupid micro-optimizations which are surely not important for real applications. Anyway, I'm thinking about real-life benchmarks but that's quite hard. One issue is that PEP 580 by itself does not make existing faster, but allows faster code to be written in the future. A second issue is that Cython (my main application) already contains optimizations for Cython-to-Cython calls. So, to see the actual impact of PEP 580, I should disable those. Jeroen. From guido at python.org Wed Jul 11 20:10:42 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 11 Jul 2018 17:10:42 -0700 Subject: [Python-Dev] Accepting PEP 572, Assignment Expressions Message-ID: As anticippated, after a final round of feedback I am hereby accepting PEP 572, Assignment Expressions: https://www.python.org/dev/peps/pep-0572/ Thanks to everyone who participated in the discussion or sent a PR. Below is a list of changes since the last post (https://mail.python.org/ pipermail/python-dev/2018-July/154557.html) -- they are mostly cosmetic so I won't post the doc again, but if you want to go over them in detail, here's the history of the file on GitHub: https://github.com/python/ peps/commits/master/pep-0572.rst, and here's a diff since the last posting: https://github.com/python/peps/compare/26e6f61f...master (sadly it's repo-wide -- you can click on Files changed and then navigate to pep-0572.rst). - Tweaked the example at line 95-100 to use result = ... rather than return ... so as to make a different rewrite less feasible - Replaced the weak "2-arg iter" example with Giampaolo Roloda's while chunk := file.read(8192): process(chunk) - *Added prohibition of unparenthesized assignment expressions in annotations and lambdas* - Clarified that TargetScopeError is a *new* subclass of SyntaxError - Clarified the text forbidding assignment to comprehension loop control variables - Clarified that the prohibition on := with annotation applies to *inline* annotation (i.e. they cannot be syntactically combined in the same expression) - Added conditional expressions to the things := binds less tightly than - Dropped section "This could be used to create ugly code" - Clarified the example in Appendix C Now on to the implementation work! (Maybe I'll sprint on this at the core-dev sprint in September.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Thu Jul 12 00:59:38 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 12 Jul 2018 13:59:38 +0900 Subject: [Python-Dev] Micro-benchmarks for PEP 580 In-Reply-To: <5B46608C.7010000@UGent.be> References: <2b42715eed9f4120945cf6f6f58bfeaa@xmail103.UGent.be> <5B449E1D.7000507@UGent.be> <20180710152021.17d5c22d@fsol> <3ec875cdb550457d869094869bce5178@xmail103.UGent.be> <5B46608C.7010000@UGent.be> Message-ID: On Thu, Jul 12, 2018 at 4:54 AM Jeroen Demeyer wrote: > > On 2018-07-11 10:50, Victor Stinner wrote: > > As you wrote, the > > cost of function costs is unlikely the bottleneck of application. > > With that idea, METH_FASTCALL is not needed either. I still find it very > strange that nobody seems to question all the crazy existing > optimizations for function calls in CPython, yet claiming at the same > time that those are just stupid micro-optimizations which are surely not > important for real applications. METH_FASTCALL for pyfunction and builtin cfunction made application significantly faster. It is proven by application benchmark, not only micro benchmark. On the other hand, calling 3rd party extension is much less frequently. Our benchmark suite contains some extension call, but it is not so frequent and I can't find any significant boost when using METH_FASTCALL on it. That's why METH_FASTCALL benefit is proven for builtins, but not for 3rd parties. If you want to prove it, you can add benchmark heavily using extension call. With it, we can measure impact of using METH_FASTCALL in 3rd party extensions. --- But for now, I'm +1 to enable FASTCALL in custom type in 3.8. Cython author confirmed they really want to use custom method type and lack of FASTCALL support will block them. So my current point is: should we go PEP 576 or 580, or middle of them? > > Anyway, I'm thinking about real-life benchmarks but that's quite hard. > One issue is that PEP 580 by itself does not make existing faster, but > allows faster code to be written in the future. At this time, no need to show performance difference. I need some application benchmark which are our target for optimize. We can understand concretely how PEP 580 (and possible future optimization based on PEP 580) can boost some type of applications by these target sample application benchmarks. We can estimate performance impact using these benchmarks too. We can write PoC to measure performance impact too. But for now, we don't have any target application in our benchmark suite. I think It is showstopper for us. > A second issue is that > Cython (my main application) already contains optimizations for > Cython-to-Cython calls. So, to see the actual impact of PEP 580, I > should disable those. > Could you more concrete? Which optimization do you refer? direct call of cdef/cpdef? METH_FASTCALL + LOAD_METHOD? Or more futher optimization PEP 580 enables? Switching "bidning=True" and "binding=False" is not enough for it? I expect we can have several Cython modules which calls each other. I feel it's right way to simulate "Cython (not Python) as a glue language" workload. Anyway, I don't request you to show "performance impact". I request only "target application we want to optimize with PEP 580 and future optimization based on PEP 580" for now. Regards, -- INADA Naoki From songofacandy at gmail.com Thu Jul 12 01:43:02 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 12 Jul 2018 14:43:02 +0900 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? Message-ID: I'm working on making pyc stable, via stablizing marshal.dumps() https://bugs.python.org/issue34093 Sadly, it makes marshal.dumps() 40% slower. Luckily, this overhead is small (only 4%) for dumps(compile(source)) case. So my question is: May I remove unstable but faster code? Or should I make this optional and we maintain two complex code? If so, should this option enabled by default or not? For example, xmlrpc uses marshal. But xmlrpc has significant overhead other than marshaling, like dumps(compile(source)) case. So I expect marshal.dumps() performance is not critical for it too. Is there any real application which marshal.dumps() performance is critical? -- INADA Naoki From storchaka at gmail.com Thu Jul 12 02:21:55 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 12 Jul 2018 09:21:55 +0300 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: References: Message-ID: 12.07.18 08:43, INADA Naoki ????: > I'm working on making pyc stable, via stablizing marshal.dumps() > https://bugs.python.org/issue34093 This is not enough for making pyc stable. The order in frozesets still is arbitrary. > Sadly, it makes marshal.dumps() 40% slower. > Luckily, this overhead is small (only 4%) for dumps(compile(source)) case. What about the memory consumption? > So my question is: May I remove unstable but faster code? > > Or should I make this optional and we maintain two complex code? > If so, should this option enabled by default or not? My concern is that even if not make it optional, this will complicate the code. > For example, xmlrpc uses marshal. But xmlrpc has significant overhead > other than marshaling, like dumps(compile(source)) case. So I expect > marshal.dumps() performance is not critical for it too. xmlrpc doesn't use the marshal module. It uses terms marshalling and unmarshalling, but in different meaning. > Is there any real application which marshal.dumps() performance is critical? EVE Online is a well known example. What if write a script which loads .pyc files and stabilize them? This could solve the problem for applications which need stable .pyc files, with zero impact on common use. From vstinner at redhat.com Thu Jul 12 04:00:36 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 12 Jul 2018 10:00:36 +0200 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: References: Message-ID: 2018-07-12 8:21 GMT+02:00 Serhiy Storchaka : >> Is there any real application which marshal.dumps() performance is >> critical? > > EVE Online is a well known example. EVE Online has been created in 2003. I guess that it still uses Python 2.7. I'm not sure that a video game would pick marshal in 2018. Victor From songofacandy at gmail.com Thu Jul 12 04:15:39 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 12 Jul 2018 17:15:39 +0900 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: References: Message-ID: On Thu, Jul 12, 2018 at 3:22 PM Serhiy Storchaka wrote: > > 12.07.18 08:43, INADA Naoki ????: > > I'm working on making pyc stable, via stablizing marshal.dumps() > > https://bugs.python.org/issue34093 > > This is not enough for making pyc stable. The order in frozesets still > is arbitrary. But we can use PYTHONHASHSEED to make pyc stable. Currently, refcnt is the only known issue for reproducible pyc build. > > > Sadly, it makes marshal.dumps() 40% slower. > > Luckily, this overhead is small (only 4%) for dumps(compile(source)) case. > > What about the memory consumption? No overhead, because we already used same hashtable for w_ref. I just make it two-pass, instead of one-pass. > > > So my question is: May I remove unstable but faster code? > > > > Or should I make this optional and we maintain two complex code? > > If so, should this option enabled by default or not? > > My concern is that even if not make it optional, this will complicate > the code. When it's not optional, it makes almost duplicate of w_object for reference counting in object tree. https://github.com/python/cpython/pull/8226/commits/e170116e80dfd27f923c88fc11e42f0d6f687a00 > > > For example, xmlrpc uses marshal. But xmlrpc has significant overhead > > other than marshaling, like dumps(compile(source)) case. So I expect > > marshal.dumps() performance is not critical for it too. > > xmlrpc doesn't use the marshal module. It uses terms marshalling and > unmarshalling, but in different meaning. > Oh, I just grepped and misunderstood. > > Is there any real application which marshal.dumps() performance is critical? > EVE Online is a well known example. > Do they use version>=3? In version 3, FLAG_REF is introduced and it made significant runtime overhead already. If marshaling speed is very important, version<2 should be used. > What if write a script which loads .pyc files and stabilize them? This > could solve the problem for applications which need stable .pyc files, > with zero impact on common use. > Hmm, do you mean which?: * Adding marshal.dump_stable_pyc() and use it like `marshal.dump_stable_pyc(marshal.loads(code))` * Implementing pure Python marshal.dumps in distutils -- INADA Naoki From tritium-list at sdamon.com Thu Jul 12 04:55:02 2018 From: tritium-list at sdamon.com (Alex Walters) Date: Thu, 12 Jul 2018 04:55:02 -0400 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: References: Message-ID: > -----Original Message----- > From: Python-Dev list=sdamon.com at python.org> On Behalf Of Victor Stinner > Sent: Thursday, July 12, 2018 4:01 AM > To: Serhiy Storchaka > Cc: python-dev > Subject: Re: [Python-Dev] Can I make marshal.dumps() slower but stabler? > > 2018-07-12 8:21 GMT+02:00 Serhiy Storchaka : > >> Is there any real application which marshal.dumps() performance is > >> critical? > > > > EVE Online is a well known example. > > EVE Online has been created in 2003. I guess that it still uses Python 2.7. > > I'm not sure that a video game would pick marshal in 2018. > EVE doesn't use stock CPython, IIRC. They use a version of stackless 2, with their own patches. If a company is willing to patch python itself, I don't think their practices should be cited without more context about what they actually modified. > Victor > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > list%40sdamon.com From chris.jerdonek at gmail.com Thu Jul 12 06:06:32 2018 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 12 Jul 2018 03:06:32 -0700 Subject: [Python-Dev] Accepting PEP 572, Assignment Expressions In-Reply-To: References: Message-ID: (status := "Accepted") and "Congratulations!" ;-) (hope I did that right, but I can't try it yet!) Thanks for hanging in there, Guido, and for your patience with everyone during the discussions. I'm glad you're still with us! --Chris On Wed, Jul 11, 2018 at 5:10 PM, Guido van Rossum wrote: > As anticippated, after a final round of feedback I am hereby accepting PEP > 572, Assignment Expressions: https://www.python.org/dev/peps/pep-0572/ > > Thanks to everyone who participated in the discussion or sent a PR. > > Below is a list of changes since the last post ( > https://mail.python.org/pipermail/python-dev/2018-July/154557.html) -- > they are mostly cosmetic so I won't post the doc again, but if you want to > go over them in detail, here's the history of the file on GitHub: > https://github.com/python/peps/commits/master/pep-0572.rst, and here's a > diff since the last posting: https://github.com/python/peps > /compare/26e6f61f...master (sadly it's repo-wide -- you can click on > Files changed and then navigate to pep-0572.rst). > > - Tweaked the example at line 95-100 to use result = ... rather than return > ... so as to make a different rewrite less feasible > - Replaced the weak "2-arg iter" example with Giampaolo Roloda's while > chunk := file.read(8192): process(chunk) > - *Added prohibition of unparenthesized assignment expressions in > annotations and lambdas* > - Clarified that TargetScopeError is a *new* subclass of SyntaxError > - Clarified the text forbidding assignment to comprehension loop > control variables > - Clarified that the prohibition on := with annotation applies to > *inline* annotation (i.e. they cannot be syntactically combined in the > same expression) > - Added conditional expressions to the things := binds less tightly > than > - Dropped section "This could be used to create ugly code" > - Clarified the example in Appendix C > > Now on to the implementation work! (Maybe I'll sprint on this at the > core-dev sprint in September.) > > -- > --Guido van Rossum (python.org/~guido) > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > chris.jerdonek%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Thu Jul 12 06:07:18 2018 From: steve at holdenweb.com (Steve Holden) Date: Thu, 12 Jul 2018 11:07:18 +0100 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: References: Message-ID: Eve is indeed based on stackless 2, and are well capable of ignoring changes they don't think they need (or were when I was working with them). At one point I seem to remember they optimised their interpreter to use singleton floating-point values, saving large quantities of memory by having only one floating-point zero. Steve Holden On Thu, Jul 12, 2018 at 9:55 AM, Alex Walters wrote: > > > > -----Original Message----- > > From: Python-Dev > list=sdamon.com at python.org> On Behalf Of Victor Stinner > > Sent: Thursday, July 12, 2018 4:01 AM > > To: Serhiy Storchaka > > Cc: python-dev > > Subject: Re: [Python-Dev] Can I make marshal.dumps() slower but stabler? > > > > 2018-07-12 8:21 GMT+02:00 Serhiy Storchaka : > > >> Is there any real application which marshal.dumps() performance is > > >> critical? > > > > > > EVE Online is a well known example. > > > > EVE Online has been created in 2003. I guess that it still uses Python > 2.7. > > > > I'm not sure that a video game would pick marshal in 2018. > > > > EVE doesn't use stock CPython, IIRC. They use a version of stackless 2, > with their own patches. If a company is willing to patch python itself, I > don't think their practices should be cited without more context about what > they actually modified. > > > Victor > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tritium- > > list%40sdamon.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > steve%40holdenweb.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Thu Jul 12 07:29:17 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Jul 2018 13:29:17 +0200 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? References: Message-ID: <20180712132917.083722bb@fsol> On Thu, 12 Jul 2018 09:21:55 +0300 Serhiy Storchaka wrote: > > > Is there any real application which marshal.dumps() performance is critical? > EVE Online is a well known example. > > What if write a script which loads .pyc files and stabilize them? This > could solve the problem for applications which need stable .pyc files, > with zero impact on common use. Should python-dev maintain that script? If yes, it sounds better to make marshal itself deterministic. Regards Antoine. From arj.python at gmail.com Thu Jul 12 11:14:24 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Thu, 12 Jul 2018 19:14:24 +0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: sorry for reviving the dead but community acceptance, a fundamental pep principle has not been respected for 572 also 29 core devs dislike vs 3 like maybe there are special cases where the BDFL can pin issues also, maybe there are two aspects, one disliking := and one the actual expression assignment as for me i don't like the := symbol Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Thu Jul 12 11:41:57 2018 From: vstinner at redhat.com (Victor Stinner) Date: Thu, 12 Jul 2018 17:41:57 +0200 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: 2018-07-12 17:14 GMT+02:00 Abdur-Rahmaan Janhangeer : > sorry for reviving the dead but community acceptance, a fundamental pep > principle has not been respected for 572 > > also 29 core devs dislike vs 3 like You are referring to a *poll* that I ran in May. I don't see any community issue, the PEP process was always the same: Guido van Rossum takes the final decision. IMHO since the poll has be done, some core devs changed their mind, *as the PEP evolved* in the meanwhile. For example, I'm more on the +1 side now (I was on the strong -1 side when I ran the poll). Anyway, it no longer matters since the PEP has been approved. It's now time to celebrate! > maybe there are special cases where the BDFL can pin issues D in BDFL stands for Dictator. Guido has super power :-) I'm fine with that. Victor From zhuoql at yahoo.com Thu Jul 12 03:37:02 2018 From: zhuoql at yahoo.com (ZHUO QL (KDr2)) Date: Thu, 12 Jul 2018 07:37:02 +0000 (UTC) Subject: [Python-Dev] Accepting PEP 572, Assignment Expressions In-Reply-To: References: Message-ID: <439700696.3163611.1531381022831@mail.yahoo.com> Hooray! I could be a happy python oneliner now! Greetings. ZHUO QL (KDr2, http://kdr2.com) On Thursday, July 12, 2018, 8:12:54 AM GMT+8, Guido van Rossum wrote: As anticippated, after a final round of feedback I am hereby accepting PEP 572, Assignment Expressions: https://www.python.org/dev/ peps/pep-0572/ Thanks to everyone who participated in the discussion or sent a PR. Below is a list of changes since the last post (https://mail.python.org/ pipermail/python-dev/2018- July/154557.html) -- they are mostly cosmetic so I won't post the doc again, but if you want to go over them in detail, here's the history of the file on GitHub: https://github.com/python/ peps/commits/master/pep-0572. rst, and here's a diff since the last posting: https://github.com/python/ peps/compare/26e6f61f...master (sadly it's repo-wide -- you can click on Files changed and then navigate to pep-0572.rst). - Tweaked the example at line 95-100 to use result = ... rather than return ... so as to make a different rewrite less feasible - Replaced the weak "2-arg iter" example with Giampaolo Roloda's while chunk := file.read(8192): process(chunk) - Added prohibition of unparenthesized assignment expressions in annotations and lambdas - Clarified that TargetScopeError is a new subclass of SyntaxError - Clarified the text forbidding assignment to comprehension loop control variables - Clarified that the prohibition on := with annotation applies to inline annotation (i.e. they cannot be syntactically combined in the same expression) - Added conditional expressions to the things := binds less tightly than - Dropped section "This could be used to create ugly code" - Clarified the example in Appendix C Now on to the implementation work! (Maybe I'll sprint on this at the core-dev sprint in September.) -- --Guido van Rossum (python.org/~guido)_______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/zhuoql%40yahoo.com -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Thu Jul 12 11:48:52 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Thu, 12 Jul 2018 16:48:52 +0100 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On 12 July 2018 at 16:41, Victor Stinner wrote: > 2018-07-12 17:14 GMT+02:00 Abdur-Rahmaan Janhangeer >: > > sorry for reviving the dead but community acceptance, a fundamental pep > > principle has not been respected for 572 > > > > also 29 core devs dislike vs 3 like > > [...] *as the PEP evolved* in the meanwhile. Yes, the PEP has improved significantly since that time. My guess is the same poll taken now could give an opposite result. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Thu Jul 12 12:23:55 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 13 Jul 2018 01:23:55 +0900 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Fri, Jul 13, 2018 at 12:48 AM Ivan Levkivskyi wrote: > > On 12 July 2018 at 16:41, Victor Stinner wrote: >> >> 2018-07-12 17:14 GMT+02:00 Abdur-Rahmaan Janhangeer : >> > sorry for reviving the dead but community acceptance, a fundamental pep >> > principle has not been respected for 572 >> > >> > also 29 core devs dislike vs 3 like >> >> [...] *as the PEP evolved* in the meanwhile. > > > Yes, the PEP has improved significantly since that time. My guess is the same poll taken now could give an opposite result. > I still -0 on PEP 572. But strong -1 on restart discussion about changing it. We should polish and implement it for now, not change. -- INADA Naoki From barry at python.org Thu Jul 12 13:21:09 2018 From: barry at python.org (Barry Warsaw) Date: Thu, 12 Jul 2018 10:21:09 -0700 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Jul 12, 2018, at 09:23, INADA Naoki wrote: >> >> Yes, the PEP has improved significantly since that time. My guess is the same poll taken now could give an opposite result. >> > > I still -0 on PEP 572. But strong -1 on restart discussion about changing it. > We should polish and implement it for now, not change. I think that?s likely true. While extremely painful for so many of us, I think the end result is a much better PEP, and a much better feature. I was -1 as well, but I?d say I?m a firm +0 now[*]. I like how many of the problematic syntactic and semantic issues have been narrowed and prohibited, and I can see myself using this sparingly. It?s not the first time I?ve found myself in this position with a new Python feature, and it?s one of the reasons I deeply trust Guido?s intuition and sensibilities. Cheers, -Barry [*] Not that it matters; the PEP is accepted - time to move on! The world won?t end. :) -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From mertz at gnosis.cx Thu Jul 12 13:34:04 2018 From: mertz at gnosis.cx (David Mertz) Date: Thu, 12 Jul 2018 13:34:04 -0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Thu, Jul 12, 2018, 1:24 PM Barry Warsaw wrote: > It?s not the first time I?ve found myself in this position with a new > Python feature, and it?s one of the reasons I deeply trust Guido?s > intuition and sensibilities. > Sure... Except for the terrible choice to drop the '<>' inequality operator, and why all my scripts start with barry_as_FLUFL. :-) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Thu Jul 12 13:56:48 2018 From: steve at holdenweb.com (Steve Holden) Date: Thu, 12 Jul 2018 18:56:48 +0100 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: On Thu, Jul 12, 2018 at 6:21 PM, Barry Warsaw wrote: > ?[...] > > I was -1 as well, but I?d say I?m a firm +0 now[*]. I like how many of > the problematic syntactic and semantic issues have been narrowed and > prohibited, and I can see myself using this sparingly. > ?[...] I think experience will show that's how it's best used - only for measurable wins. But then Python is the kind of language where "if it's simpler, use that" is a part of the philosophy, thanks to the Zen's popularity.? -------------- next part -------------- An HTML attachment was scrubbed... URL: From nd at perlig.de Thu Jul 12 16:03:30 2018 From: nd at perlig.de (=?iso-8859-1?q?Andr=E9_Malo?=) Date: Thu, 12 Jul 2018 22:03:30 +0200 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: References: Message-ID: <201807122203.30359@news.perlig.de> * INADA Naoki wrote: > Is there any real application which marshal.dumps() performance is > critical? I'm using it for spooling big chunks of data on disk, exactly for the reason that it's faster than pickle. Cheers, -- "Das Verhalten von Gates hatte mir bewiesen, dass ich auf ihn und seine beiden Gef?hrten nicht zu z?hlen brauchte" -- Karl May, "Winnetou III" From solipsis at pitrou.net Thu Jul 12 16:09:41 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 12 Jul 2018 22:09:41 +0200 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? References: <201807122203.30359@news.perlig.de> Message-ID: <20180712220941.30dd68b4@fsol> On Thu, 12 Jul 2018 22:03:30 +0200 Andr? Malo wrote: > * INADA Naoki wrote: > > > Is there any real application which marshal.dumps() performance is > > critical? > > I'm using it for spooling big chunks of data on disk, exactly for the reason > that it's faster than pickle. Which kind of data is that? Regards Antoine. From arj.python at gmail.com Thu Jul 12 18:01:31 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Fri, 13 Jul 2018 02:01:31 +0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: *D in BDFL stands for Dictator. * The B diminishes that ex sugar not same as salty sugar Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From arj.python at gmail.com Thu Jul 12 18:03:32 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Fri, 13 Jul 2018 02:03:32 +0400 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: ok then was a point i wanted to clear btw smileys in this thread should have been :=) Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Thu Jul 12 18:33:18 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 12 Jul 2018 15:33:18 -0700 Subject: [Python-Dev] PEP 572: Do we really need a ":" in ":="? In-Reply-To: References: Message-ID: <5B47D72E.5060909@stoneleaf.us> On 07/12/2018 03:03 PM, Abdur-Rahmaan Janhangeer wrote: > btw smileys in this thread should have been :=) lol! -- ~Ethan~ From songofacandy at gmail.com Thu Jul 12 18:33:01 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Fri, 13 Jul 2018 07:33:01 +0900 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: <201807122203.30359@news.perlig.de> References: <201807122203.30359@news.perlig.de> Message-ID: On Fri, Jul 13, 2018 at 5:03 AM Andr? Malo wrote: > > * INADA Naoki wrote: > > > Is there any real application which marshal.dumps() performance is > > critical? > > I'm using it for spooling big chunks of data on disk, exactly for the reason > that it's faster than pickle. > > Cheers, Does your data contains repetition of same object (not same value)? If yes, this change will affects you. If no, you can use older version which doesn't have overhead of checking object identity. >>> x = [0]*100 >>> y = [0]*100 >>> data = [x,y,x] >>> import marshal >>> len(marshal.dumps(data)) # x is marshaled once 1020 >>> d[0] is d[2] True >>> d[0] is d[1] False >>> import json >>> len(json.dumps(data)) # x is marshaled twice 906 >>> d = marshal.loads(marshal.dumps(data, 2)) # x is marshaled twice >>> len(d) 1520 >>> d[0] is d[2] False -- INADA Naoki From chris.barker at noaa.gov Fri Jul 13 01:30:59 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 13 Jul 2018 00:30:59 -0500 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: On Mon, Jul 9, 2018 at 3:18 PM, Guido van Rossum wrote: > Definitely docs first. And we should keep references to generator > expressions too, if only to explain that they've been renamed. > > Perhaps someone should try it out in a 3rd party tutorial to see how it > goes? > I'm not sure what "trying it out in a tutorial" would look like. I try to be pretty clear about terminology when I teach newbies -- so I don't want to tell anyone this new thing is called a "generator comprehension" if they aren't going to see that term anywhere else. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From nd at perlig.de Fri Jul 13 05:41:40 2018 From: nd at perlig.de (=?ISO-8859-1?Q?Andr=E9?= Malo) Date: Fri, 13 Jul 2018 11:41:40 +0200 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: <20180712220941.30dd68b4@fsol> References: <201807122203.30359@news.perlig.de> <20180712220941.30dd68b4@fsol> Message-ID: <2315564.0cb8OQ2DeK@finnegan> On Donnerstag, 12. Juli 2018 22:09:41 CEST Antoine Pitrou wrote: > On Thu, 12 Jul 2018 22:03:30 +0200 > > Andr? Malo wrote: > > * INADA Naoki wrote: > > > Is there any real application which marshal.dumps() performance is > > > critical? > > > > I'm using it for spooling big chunks of data on disk, exactly for the > > reason that it's faster than pickle. > > Which kind of data is that? Basically iterators of builtin objects (dicts or tuples of strings or numbers). Typically one unit or "row" per dumps() call (they are written one after the next and marshal load can easily load them in the same manner). They're certainly never the same objects (except maybe for dict keys, which might be interned) Cheers, nd From random832 at fastmail.com Fri Jul 13 09:36:21 2018 From: random832 at fastmail.com (Random832) Date: Fri, 13 Jul 2018 09:36:21 -0400 Subject: [Python-Dev] Accepting PEP 572, Assignment Expressions In-Reply-To: References: Message-ID: <1531488981.4154225.1439701272.4F6F36F8@webmail.messagingengine.com> On Wed, Jul 11, 2018, at 20:10, Guido van Rossum wrote: > As anticippated, after a final round of feedback I am hereby accepting PEP > 572, Assignment Expressions: https://www.python.org/dev/peps/pep-0572/ I know everyone else is probably sick of discussing this (I somehow completely missed the discussion until it was almost done) but I have a question... Why does this not allow assignment to attributes and subscripts? I think this is, at least, surprising enough that there should be a rationale section explaining it in the PEP. As it is, it's barely even explicitly stated, other than the use of 'NAME' in a few places and an offhand mention "and [the assignment statement] can assign to attributes and subscripts." From rosuav at gmail.com Fri Jul 13 09:55:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 13 Jul 2018 23:55:18 +1000 Subject: [Python-Dev] Accepting PEP 572, Assignment Expressions In-Reply-To: <1531488981.4154225.1439701272.4F6F36F8@webmail.messagingengine.com> References: <1531488981.4154225.1439701272.4F6F36F8@webmail.messagingengine.com> Message-ID: On Fri, Jul 13, 2018 at 11:36 PM, Random832 wrote: > On Wed, Jul 11, 2018, at 20:10, Guido van Rossum wrote: >> As anticippated, after a final round of feedback I am hereby accepting PEP >> 572, Assignment Expressions: https://www.python.org/dev/peps/pep-0572/ > > I know everyone else is probably sick of discussing this (I somehow completely missed the discussion until it was almost done) but I have a question... > > Why does this not allow assignment to attributes and subscripts? I think this is, at least, surprising enough that there should be a rationale section explaining it in the PEP. As it is, it's barely even explicitly stated, other than the use of 'NAME' in a few places and an offhand mention "and [the assignment statement] can assign to attributes and subscripts." > It was discussed at some length, yes. Assignment to arbitrary targets would also mean permitting iterable unpacking, which is not desired ("x, y := 3, 4"??), and there weren't enough use-cases for attribute/item assignment to justify creating a rule of "you can assign to any single target, but can't unpack". In the future, if such use-cases are found, the grammar can be expanded. ChrisA From tismer at stackless.com Fri Jul 13 09:57:30 2018 From: tismer at stackless.com (Christian Tismer) Date: Fri, 13 Jul 2018 15:57:30 +0200 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: References: Message-ID: <4496ca29-3ff6-bb98-c2d3-9de37bbbe15d@stackless.com> Well, to my knowledge they did not modify the marshal code. They are in fact heavily dependent from marshal speed since that is used frequently to save and restore state of many actors. But haven't looked further since 2010 ;-) Btw., why are they considering to make the algorithm slower, just because someone wants the algorithm stable? An optional keyword argument would give the stability, and the default behavior would not be changed at all. Cheers - Chris On 12.07.18 12:07, Steve Holden wrote: > Eve is indeed based on stackless 2, and are well capable of ignoring > changes they don't think they need (or were when I was working with > them). At one point I seem to remember they optimised their interpreter > to use singleton floating-point values, saving large quantities of > memory by having only one floating-point zero. > > Steve Holden > > On Thu, Jul 12, 2018 at 9:55 AM, Alex Walters > wrote: > > > > > -----Original Message----- > > From: Python-Dev > list=sdamon.com at python.org > On Behalf Of > Victor Stinner > > Sent: Thursday, July 12, 2018 4:01 AM > > To: Serhiy Storchaka > > > Cc: python-dev > > > Subject: Re: [Python-Dev] Can I make marshal.dumps() slower but stabler? > > > > 2018-07-12 8:21 GMT+02:00 Serhiy Storchaka >: > > >> Is there any real application which marshal.dumps() performance is > > >> critical? > > > > > > EVE Online is a well known example. > > > > EVE Online has been created in 2003. I guess that it still uses Python > 2.7. > > > > I'm not sure that a video game would pick marshal in 2018. > > > > EVE doesn't use stock CPython, IIRC.? They use a version of stackless 2, > with their own patches.? If a company is willing to patch python > itself, I > don't think their practices should be cited without more context > about what > they actually modified. > > > Victor > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/tritium- > > > list%40sdamon.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com > > > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com > -- Christian Tismer-Sperling :^) tismer at stackless.com Software Consulting : http://www.stackless.com/ Karl-Liebknecht-Str. 121 : http://pyside.org 14482 Potsdam : GPG key -> 0xE7301150FB7BEE0E phone +49 173 24 18 776 fax +49 (30) 700143-0023 -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 496 bytes Desc: OpenPGP digital signature URL: From vano at mail.mipt.ru Fri Jul 13 10:43:43 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Fri, 13 Jul 2018 17:43:43 +0300 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: <4496ca29-3ff6-bb98-c2d3-9de37bbbe15d@stackless.com> References: <4496ca29-3ff6-bb98-c2d3-9de37bbbe15d@stackless.com> Message-ID: <28f28d64-19d0-2c0f-1611-ba73614ca88f@mail.mipt.ru> If the use case for stability is only .pyc compilation, I doubt it's even relevant 'cuz .pyc's are supposed to be compiled in isolation from other current objects (otherwise, they wouldn't be reusable or would be invalidated when dependent modules change, neither of which is the case), so relevant reference counts should always be the same. I may be mistaking though. On 13.07.2018 16:57, Christian Tismer wrote: > Well, to my knowledge they did not modify the marshal code. > They are in fact heavily dependent from marshal speed since that > is used frequently to save and restore state of many actors. > > But haven't looked further since 2010 ;-) > > Btw., why are they considering to make the algorithm slower, > just because someone wants the algorithm stable? > > An optional keyword argument would give the stability, and the > default behavior would not be changed at all. > > Cheers - Chris > > > On 12.07.18 12:07, Steve Holden wrote: >> Eve is indeed based on stackless 2, and are well capable of ignoring >> changes they don't think they need (or were when I was working with >> them). At one point I seem to remember they optimised their interpreter >> to use singleton floating-point values, saving large quantities of >> memory by having only one floating-point zero. >> >> Steve Holden >> >> On Thu, Jul 12, 2018 at 9:55 AM, Alex Walters > > wrote: >> >> >> >> > -----Original Message----- >> > From: Python-Dev > > list=sdamon.com at python.org > On Behalf Of >> Victor Stinner >> > Sent: Thursday, July 12, 2018 4:01 AM >> > To: Serhiy Storchaka > >> > Cc: python-dev > >> > Subject: Re: [Python-Dev] Can I make marshal.dumps() slower but stabler? >> > >> > 2018-07-12 8:21 GMT+02:00 Serhiy Storchaka >: >> > >> Is there any real application which marshal.dumps() performance is >> > >> critical? >> > > >> > > EVE Online is a well known example. >> > >> > EVE Online has been created in 2003. I guess that it still uses Python >> 2.7. >> > >> > I'm not sure that a video game would pick marshal in 2018. >> > >> >> EVE doesn't use stock CPython, IIRC.? They use a version of stackless 2, >> with their own patches.? If a company is willing to patch python >> itself, I >> don't think their practices should be cited without more context >> about what >> they actually modified. >> >> > Victor >> > _______________________________________________ >> > Python-Dev mailing list >> > Python-Dev at python.org >> > https://mail.python.org/mailman/listinfo/python-dev >> >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/tritium- >> >> > list%40sdamon.com >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/steve%40holdenweb.com >> >> >> >> >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/tismer%40stackless.com >> > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Jul 13 10:50:51 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 13 Jul 2018 07:50:51 -0700 Subject: [Python-Dev] Accepting PEP 572, Assignment Expressions In-Reply-To: References: <1531488981.4154225.1439701272.4F6F36F8@webmail.messagingengine.com> Message-ID: Also nobody had a use case. On Fri, Jul 13, 2018 at 6:57 AM Chris Angelico wrote: > On Fri, Jul 13, 2018 at 11:36 PM, Random832 > wrote: > > On Wed, Jul 11, 2018, at 20:10, Guido van Rossum wrote: > >> As anticippated, after a final round of feedback I am hereby accepting > PEP > >> 572, Assignment Expressions: https://www.python.org/dev/peps/pep-0572/ > > > > I know everyone else is probably sick of discussing this (I somehow > completely missed the discussion until it was almost done) but I have a > question... > > > > Why does this not allow assignment to attributes and subscripts? I think > this is, at least, surprising enough that there should be a rationale > section explaining it in the PEP. As it is, it's barely even explicitly > stated, other than the use of 'NAME' in a few places and an offhand mention > "and [the assignment statement] can assign to attributes and subscripts." > > > > It was discussed at some length, yes. Assignment to arbitrary targets > would also mean permitting iterable unpacking, which is not desired > ("x, y := 3, 4"??), and there weren't enough use-cases for > attribute/item assignment to justify creating a rule of "you can > assign to any single target, but can't unpack". In the future, if such > use-cases are found, the grammar can be expanded. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Fri Jul 13 11:19:49 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Sat, 14 Jul 2018 00:19:49 +0900 Subject: [Python-Dev] Can I make marshal.dumps() slower but stabler? In-Reply-To: <28f28d64-19d0-2c0f-1611-ba73614ca88f@mail.mipt.ru> References: <4496ca29-3ff6-bb98-c2d3-9de37bbbe15d@stackless.com> <28f28d64-19d0-2c0f-1611-ba73614ca88f@mail.mipt.ru> Message-ID: On Fri, Jul 13, 2018 at 11:46 PM Ivan Pozdeev via Python-Dev wrote: > > If the use case for stability is only .pyc compilation, I doubt it's even relevant 'cuz .pyc's are supposed to be compiled in isolation from other current objects (otherwise, they wouldn't be reusable or would be invalidated when dependent modules change, neither of which is the case), so relevant reference counts should always be the same. > I may be mistaking though. > Good point! You're right. Currently, there is one unstable pyc issue (except frozenset order which can be fixed by PYTHONHASHSEED). https://bugzilla.opensuse.org/show_bug.cgi?id=1049186 This is caused by interned string. Because of interning, reference count can be unstable. Like that, long objects, tuples and some others are cached and reused automatically. But they has refcnt>1 always: reference from object and cache. So we can use FLAG_REF always for interned string, even if refcnt==1. Let's try it and wait another issue are found. Thanks! -- INADA Naoki From status at bugs.python.org Fri Jul 13 12:09:16 2018 From: status at bugs.python.org (Python tracker) Date: Fri, 13 Jul 2018 18:09:16 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20180713160916.D65FD561AE@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2018-07-06 - 2018-07-13) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6706 (-15) closed 39153 (+63) total 45859 (+48) Open issues with patches: 2666 Issues opened (33) ================== #33922: [Windows] Document the launcher's -64 suffix https://bugs.python.org/issue33922 reopened by eryksun #34062: Python launcher on Windows does not work with --list or --list https://bugs.python.org/issue34062 opened by brett.cannon #34063: binhex REASONABLY_LARGE = 32768 so what is 128000 https://bugs.python.org/issue34063 opened by Jim.Jewett #34064: subprocess functions with shell=1 pass wrong command to win32 https://bugs.python.org/issue34064 opened by mandel59 #34067: Problem with contextlib.nullcontext https://bugs.python.org/issue34067 opened by serhiy.storchaka #34068: traceback.clear_frames(): Objects/typeobject.c:3086: _PyType_L https://bugs.python.org/issue34068 opened by serhiy.storchaka #34070: Superfluous call to isatty in open() when buffering >= 0 https://bugs.python.org/issue34070 opened by Dav1d #34071: asyncio: repr(task) raises AssertionError for coros which loop https://bugs.python.org/issue34071 opened by JDLH #34075: asyncio: We should prohibit setting a ProcessPoolExecutor in w https://bugs.python.org/issue34075 opened by yselivanov #34078: Broken CRL functionality in ssl.py https://bugs.python.org/issue34078 opened by Joe N #34079: Multiprocessing module fails to build on Solaris 11.3 https://bugs.python.org/issue34079 opened by clallen #34081: Sphinx duplicate label warning in docs https://bugs.python.org/issue34081 opened by xtreak #34082: EnumMeta.__new__ should use enum_class.__new__ https://bugs.python.org/issue34082 opened by rs2 #34084: possible free statically allocated string in compiler when eas https://bugs.python.org/issue34084 opened by xiang.zhang #34085: doc Improve wording on classmethod/staticmethod https://bugs.python.org/issue34085 opened by adelfino #34086: logging.Handler.handleError regressed in python3 https://bugs.python.org/issue34086 opened by oren #34087: int(s), float(s) and others may cause segmentation fault https://bugs.python.org/issue34087 opened by fenrrir #34088: [EASY] sndhdr.what() throws exceptions on unknown files https://bugs.python.org/issue34088 opened by Barro #34089: Remove required (non-optional) modules from Modules/Setup.dist https://bugs.python.org/issue34089 opened by nascheme #34090: Python function call optimization: avoid temporary tuple to pa https://bugs.python.org/issue34090 opened by vstinner #34091: REPL does not work in msys2, ConEmu terminals on windows https://bugs.python.org/issue34091 opened by maxnoe #34093: Reproducible pyc: FLAG_REF is not stable. https://bugs.python.org/issue34093 opened by inada.naoki #34095: [2.7] test_idle fails with: /usr/bin/xvfb-run: line 181: 3617 https://bugs.python.org/issue34095 opened by deep42thought #34096: [2.7] test_audioop.test_max() failed: AssertionError: -2147483 https://bugs.python.org/issue34096 opened by deep42thought #34097: ZIP does not support timestamps before 1980 https://bugs.python.org/issue34097 opened by petr.viktorin #34098: multiprocessing.Server swallows original exception traceback https://bugs.python.org/issue34098 opened by salgado #34099: Provide debuggers with a way to know that a function is exitin https://bugs.python.org/issue34099 opened by fabioz #34100: Same constants in tuples are not merged while compile() https://bugs.python.org/issue34100 opened by Dan Rose #34101: PyBuffer_GetPointer() not documented https://bugs.python.org/issue34101 opened by pitrou #34102: local variable 'parts' referenced before assignment in feedpar https://bugs.python.org/issue34102 opened by sshnaidm #34105: test_socket.test_host_resolution_bad_address fails on Mac OS X https://bugs.python.org/issue34105 opened by lys.nikolaou #34106: Add --with-module-config= to 'configure' script https://bugs.python.org/issue34106 opened by nascheme #34108: 2to3 munges new lines on Windows https://bugs.python.org/issue34108 opened by jason.coombs Most recent 15 issues with no replies (15) ========================================== #34106: Add --with-module-config= to 'configure' script https://bugs.python.org/issue34106 #34105: test_socket.test_host_resolution_bad_address fails on Mac OS X https://bugs.python.org/issue34105 #34101: PyBuffer_GetPointer() not documented https://bugs.python.org/issue34101 #34099: Provide debuggers with a way to know that a function is exitin https://bugs.python.org/issue34099 #34098: multiprocessing.Server swallows original exception traceback https://bugs.python.org/issue34098 #34089: Remove required (non-optional) modules from Modules/Setup.dist https://bugs.python.org/issue34089 #34085: doc Improve wording on classmethod/staticmethod https://bugs.python.org/issue34085 #34082: EnumMeta.__new__ should use enum_class.__new__ https://bugs.python.org/issue34082 #34081: Sphinx duplicate label warning in docs https://bugs.python.org/issue34081 #34079: Multiprocessing module fails to build on Solaris 11.3 https://bugs.python.org/issue34079 #34070: Superfluous call to isatty in open() when buffering >= 0 https://bugs.python.org/issue34070 #34063: binhex REASONABLY_LARGE = 32768 so what is 128000 https://bugs.python.org/issue34063 #34060: regrtest: log "CPU usage" on Windows https://bugs.python.org/issue34060 #34052: sqlite's create_function() raises exception on unhashable call https://bugs.python.org/issue34052 #34046: subparsers -> add_parser doesn't support hyphen char '-' https://bugs.python.org/issue34046 Most recent 15 issues waiting for review (15) ============================================= #34108: 2to3 munges new lines on Windows https://bugs.python.org/issue34108 #34106: Add --with-module-config= to 'configure' script https://bugs.python.org/issue34106 #34102: local variable 'parts' referenced before assignment in feedpar https://bugs.python.org/issue34102 #34098: multiprocessing.Server swallows original exception traceback https://bugs.python.org/issue34098 #34097: ZIP does not support timestamps before 1980 https://bugs.python.org/issue34097 #34093: Reproducible pyc: FLAG_REF is not stable. https://bugs.python.org/issue34093 #34089: Remove required (non-optional) modules from Modules/Setup.dist https://bugs.python.org/issue34089 #34087: int(s), float(s) and others may cause segmentation fault https://bugs.python.org/issue34087 #34085: doc Improve wording on classmethod/staticmethod https://bugs.python.org/issue34085 #34084: possible free statically allocated string in compiler when eas https://bugs.python.org/issue34084 #34081: Sphinx duplicate label warning in docs https://bugs.python.org/issue34081 #34079: Multiprocessing module fails to build on Solaris 11.3 https://bugs.python.org/issue34079 #34070: Superfluous call to isatty in open() when buffering >= 0 https://bugs.python.org/issue34070 #34067: Problem with contextlib.nullcontext https://bugs.python.org/issue34067 #34064: subprocess functions with shell=1 pass wrong command to win32 https://bugs.python.org/issue34064 Top 10 most discussed issues (10) ================================= #34087: int(s), float(s) and others may cause segmentation fault https://bugs.python.org/issue34087 30 msgs #34100: Same constants in tuples are not merged while compile() https://bugs.python.org/issue34100 12 msgs #34093: Reproducible pyc: FLAG_REF is not stable. https://bugs.python.org/issue34093 10 msgs #34058: Default Python 3.7 install broken on openSUSE Leap 42.3: $PYTH https://bugs.python.org/issue34058 9 msgs #34075: asyncio: We should prohibit setting a ProcessPoolExecutor in w https://bugs.python.org/issue34075 9 msgs #34084: possible free statically allocated string in compiler when eas https://bugs.python.org/issue34084 9 msgs #34086: logging.Handler.handleError regressed in python3 https://bugs.python.org/issue34086 7 msgs #34067: Problem with contextlib.nullcontext https://bugs.python.org/issue34067 6 msgs #34097: ZIP does not support timestamps before 1980 https://bugs.python.org/issue34097 6 msgs #8036: Interpreter crashes on invalid arg to spawnl on Windows https://bugs.python.org/issue8036 5 msgs Issues closed (61) ================== #4260: Document that ctypes.xFUNCTYPE are decorators. https://bugs.python.org/issue4260 closed by berker.peksag #12087: install_egg_info fails with UnicodeEncodeError depending on lo https://bugs.python.org/issue12087 closed by serhiy.storchaka #22689: Posix getenv makes no guarantee of lifetime of returned string https://bugs.python.org/issue22689 closed by vstinner #23770: Rework how exceptions are handled in the parser module (in val https://bugs.python.org/issue23770 closed by vstinner #23859: asyncio: document behaviour of wait() cancellation https://bugs.python.org/issue23859 closed by vstinner #24459: Mention PYTHONFAULTHANDLER in the man page https://bugs.python.org/issue24459 closed by berker.peksag #24665: CJK support for textwrap https://bugs.python.org/issue24665 closed by inada.naoki #25828: PyCode_Optimize() (peephole optimizer) doesn't handle Keyboard https://bugs.python.org/issue25828 closed by vstinner #27838: test_os.test_chown() failure on koobs-freebsd-{current,9} https://bugs.python.org/issue27838 closed by vstinner #28571: llist and scipy.stats conflicts, python segfault https://bugs.python.org/issue28571 closed by serhiy.storchaka #29342: os.posix_fadvise misreports errors https://bugs.python.org/issue29342 closed by benjamin.peterson #30316: test_default_timeout() of test_threading.BarrierTests: random https://bugs.python.org/issue30316 closed by vstinner #30331: TestPOP3_TLSClass: socket.timeout: timed out on AMD64 FreeBSD https://bugs.python.org/issue30331 closed by vstinner #30416: constant folding opens compiler to quadratic time hashing https://bugs.python.org/issue30416 closed by serhiy.storchaka #31014: webbrowser._synthesize uses outdated calling signature for web https://bugs.python.org/issue31014 closed by serhiy.storchaka #31623: Build MSI installer for 3.4 security releases on Windows https://bugs.python.org/issue31623 closed by terry.reedy #31839: datetime: add method to parse isoformat() output https://bugs.python.org/issue31839 closed by martin.panter #32729: socket.readinto() doesn't catch TimeoutError https://bugs.python.org/issue32729 closed by vstinner #33155: Use super().method instead in Logging https://bugs.python.org/issue33155 closed by rhettinger #33305: Improve syntax error for numbers with leading zero https://bugs.python.org/issue33305 closed by serhiy.storchaka #33597: Compact PyGC_Head https://bugs.python.org/issue33597 closed by inada.naoki #33648: unused with_c_locale_warning option in configure should be rem https://bugs.python.org/issue33648 closed by inada.naoki #33697: test_zipfile.test_write_filtered_python_package() failed on Ap https://bugs.python.org/issue33697 closed by vstinner #33702: Add some missings links in production lists and a little polis https://bugs.python.org/issue33702 closed by adelfino #33715: test_multiprocessing_spawn.test_wait_result() failed on x86 Wi https://bugs.python.org/issue33715 closed by vstinner #33716: test_concurrent_futures.test_crash() failed on x86 Windows7 3. https://bugs.python.org/issue33716 closed by vstinner #33723: test_time.test_thread_time() failed on AMD64 Debian root 3.x https://bugs.python.org/issue33723 closed by vstinner #33804: Document the default value of the size parameter of io.TextIOB https://bugs.python.org/issue33804 closed by serhiy.storchaka #33859: Spelling mistakes found using aspell https://bugs.python.org/issue33859 closed by xtreak #33865: [EASY] Missing code page aliases: "unknown encoding: 874" https://bugs.python.org/issue33865 closed by vstinner #33888: Use CPython instead of Python when talking about implementatio https://bugs.python.org/issue33888 closed by terry.reedy #33965: [Windows WSL] Fatal Python error: _Py_InitializeMainInterprete https://bugs.python.org/issue33965 closed by vstinner #33967: functools.singledispatch: Misleading exception when calling wi https://bugs.python.org/issue33967 closed by inada.naoki #34009: Document Debian 8 / Ubuntu 14.04 OpenSSL compatibility issues https://bugs.python.org/issue34009 closed by ncoghlan #34017: Tkinter CheckButton not working in EXE https://bugs.python.org/issue34017 closed by terry.reedy #34019: webbrowser: wrong arguments for Opera browser. https://bugs.python.org/issue34019 closed by taleinat #34031: [EASY] Incorrect usage of unittest.TestCase in test_urllib2_lo https://bugs.python.org/issue34031 closed by taleinat #34034: Python 3.7.0 multiprocessing forkserver ForkingPickler behavio https://bugs.python.org/issue34034 closed by pitrou #34041: add *deterministic* parameter to sqlite3.Connection.create_fun https://bugs.python.org/issue34041 closed by berker.peksag #34042: Reference loss for local classes https://bugs.python.org/issue34042 closed by yselivanov #34050: Broken links to "OpenSSL cipher list format" in documentation https://bugs.python.org/issue34050 closed by benjamin.peterson #34051: Update multiprocessing example https://bugs.python.org/issue34051 closed by Windson Yang #34056: checked hash-based pyc files not working with imp module https://bugs.python.org/issue34056 closed by benjamin.peterson #34057: Py_Initialize aborts when using static Python version. Windows https://bugs.python.org/issue34057 closed by ncoghlan #34059: multiprocessing deadlock https://bugs.python.org/issue34059 closed by gobbedy #34061: Document sqlite3.NotSupportedError exception https://bugs.python.org/issue34061 closed by berker.peksag #34065: 'force' should be printed as italic font not highlight font. https://bugs.python.org/issue34065 closed by corona10 #34066: Possible resource warning in "with open()" https://bugs.python.org/issue34066 closed by serhiy.storchaka #34069: shutil.move fails with AttributeError https://bugs.python.org/issue34069 closed by berker.peksag #34072: 3.7.0 Windows embeddable zip vcruntime140.dll https://bugs.python.org/issue34072 closed by steve.dower #34073: asyncio and ProcessPoolExecutor: OSError on loop.close() https://bugs.python.org/issue34073 closed by yselivanov #34074: Asyncio breaks coroutine finalization process https://bugs.python.org/issue34074 closed by yselivanov #34076: Nested loop in dictionary comprehension gives `global name not https://bugs.python.org/issue34076 closed by serhiy.storchaka #34077: doc Be explicit about mock_open created mocks not supporting _ https://bugs.python.org/issue34077 closed by berker.peksag #34080: Memory leak in the compiler in case of errors https://bugs.python.org/issue34080 closed by vstinner #34083: Functional Programming HOWTO: Dictionary ordering isn't "essen https://bugs.python.org/issue34083 closed by inada.naoki #34092: test_logging: SMTPHandlerTest.test_basic() fails randomly on x https://bugs.python.org/issue34092 closed by vstinner #34094: Porting Python 2 to Python 3 example contradicts its own advic https://bugs.python.org/issue34094 closed by brett.cannon #34103: Python3.7 places cwd instead of a scripts path in sys.path. https://bugs.python.org/issue34103 closed by AndreasPK #34104: email.message.get_payload should enforce correct encoding https://bugs.python.org/issue34104 closed by r.david.murray #34107: root.warning('msg') output format modified by logging.warning https://bugs.python.org/issue34107 closed by vinay.sajip From ncoghlan at gmail.com Sun Jul 15 01:21:07 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Jul 2018 15:21:07 +1000 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: On 13 July 2018 at 15:30, Chris Barker via Python-Dev wrote: > On Mon, Jul 9, 2018 at 3:18 PM, Guido van Rossum wrote: >> >> Definitely docs first. And we should keep references to generator >> expressions too, if only to explain that they've been renamed. >> >> Perhaps someone should try it out in a 3rd party tutorial to see how it >> goes? > > > I'm not sure what "trying it out in a tutorial" would look like. > > I try to be pretty clear about terminology when I teach newbies -- so I > don't want to tell anyone this new thing is called a "generator > comprehension" if they aren't going to see that term anywhere else. Nina Zakharenko made the "they're officially called generator expressions, but I find it more helpful to think of them as generator comprehensions" observation in her PyCon 2018 presentation on "Elegant Solutions for Everyday Python Problems": https://www.slideshare.net/nnja/elegant-solutions-for-everyday-python-problems-pycon-2018/27 The article from Ned Batchelder mentioned in that slide is this one, which goes through the history of Raymond originally proposing the notion as generator comprehensions, them getting changed to generator expressions during the PEP 289 discussions, and then asking if it might be worth going back to the originally suggested name: https://nedbatchelder.com/blog/201605/generator_comprehensions.html And then in PEP 572, it was found that being able to group all 4 constructs (list/set/dict comps + generator expressions) under a single term was a genuinely useful shorthand: https://www.python.org/dev/peps/pep-0572/#scope-of-the-target So trying out the terminology in a tutorial context would be to do something similar to what Nina did in her talk: introduce the notion of list/set/dict/generator comprehensions, and then make a side note that the latter are officially referred to as "generator expressions". This wouldn't be the first time that terminology has differed between Python-as-commonly-taught and Python-as-formally-defined, as I've yet to hear anyone refer to container displays outside a language design discussion - everyone else calls them container literals (or, more typically, a literal for the specific builtin container type being discussed). In this case, though, we'd be considering eventually changing the language reference as well, and perhaps even some day the AST node name (from GeneratorExp to GeneratorComp). We wouldn't need to change anything in the grammar definition (since that already shares the comp_for and comp_if syntax definitions between container comprehensions and generator expressions), or the AST node structure (since GeneratorExp already uses a "comprehensions" attribute, the same as the ListComp/SetComp/DictComp nodes). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Sun Jul 15 01:32:05 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 15 Jul 2018 15:32:05 +1000 Subject: [Python-Dev] Accepting PEP 572, Assignment Expressions In-Reply-To: <1531488981.4154225.1439701272.4F6F36F8@webmail.messagingengine.com> References: <1531488981.4154225.1439701272.4F6F36F8@webmail.messagingengine.com> Message-ID: On 13 July 2018 at 23:36, Random832 wrote: > On Wed, Jul 11, 2018, at 20:10, Guido van Rossum wrote: >> As anticippated, after a final round of feedback I am hereby accepting PEP >> 572, Assignment Expressions: https://www.python.org/dev/peps/pep-0572/ > > I know everyone else is probably sick of discussing this (I somehow completely missed the discussion until it was almost done) but I have a question... > > Why does this not allow assignment to attributes and subscripts? I think this is, at least, surprising enough that there should be a rationale section explaining it in the PEP. As it is, it's barely even explicitly stated, other than the use of 'NAME' in a few places and an offhand mention "and [the assignment statement] can assign to attributes and subscripts." It's in the same category as disallowing assignment to attributes and subscripts in other name binding operations like "def" and "class" statements: there isn't a technical limitation preventing it, there's a design decision that we currently believe allowing it would reduce code clarity rather than increase it. It's also the case that unlike local name bindings, assigning to attributes and subscripts as an expression can already be formulated as a helper function with side effects. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From chris.barker at noaa.gov Sun Jul 15 12:58:27 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Sun, 15 Jul 2018 11:58:27 -0500 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: Thanks Nick, I'll adopt this approach when I update my teaching materials. If I think of it, I"ll post here when I do that -CHB On Sun, Jul 15, 2018 at 12:21 AM, Nick Coghlan wrote: > On 13 July 2018 at 15:30, Chris Barker via Python-Dev > wrote: > > On Mon, Jul 9, 2018 at 3:18 PM, Guido van Rossum > wrote: > >> > >> Definitely docs first. And we should keep references to generator > >> expressions too, if only to explain that they've been renamed. > >> > >> Perhaps someone should try it out in a 3rd party tutorial to see how it > >> goes? > > > > > > I'm not sure what "trying it out in a tutorial" would look like. > > > > I try to be pretty clear about terminology when I teach newbies -- so I > > don't want to tell anyone this new thing is called a "generator > > comprehension" if they aren't going to see that term anywhere else. > > Nina Zakharenko made the "they're officially called generator > expressions, but I find it more helpful to think of them as generator > comprehensions" observation in her PyCon 2018 presentation on "Elegant > Solutions for Everyday Python Problems": > https://www.slideshare.net/nnja/elegant-solutions-for- > everyday-python-problems-pycon-2018/27 > > The article from Ned Batchelder mentioned in that slide is this one, > which goes through the history of Raymond originally proposing the > notion as generator comprehensions, them getting changed to generator > expressions during the PEP 289 discussions, and then asking if it > might be worth going back to the originally suggested name: > https://nedbatchelder.com/blog/201605/generator_comprehensions.html > > And then in PEP 572, it was found that being able to group all 4 > constructs (list/set/dict comps + generator expressions) under a > single term was a genuinely useful shorthand: > https://www.python.org/dev/peps/pep-0572/#scope-of-the-target > > So trying out the terminology in a tutorial context would be to do > something similar to what Nina did in her talk: introduce the notion > of list/set/dict/generator comprehensions, and then make a side note > that the latter are officially referred to as "generator expressions". > > This wouldn't be the first time that terminology has differed between > Python-as-commonly-taught and Python-as-formally-defined, as I've yet > to hear anyone refer to container displays outside a language design > discussion - everyone else calls them container literals (or, more > typically, a literal for the specific builtin container type being > discussed). > > In this case, though, we'd be considering eventually changing the > language reference as well, and perhaps even some day the AST node > name (from GeneratorExp to GeneratorComp). > > We wouldn't need to change anything in the grammar definition (since > that already shares the comp_for and comp_if syntax definitions > between container comprehensions and generator expressions), or the > AST node structure (since GeneratorExp already uses a "comprehensions" > attribute, the same as the ListComp/SetComp/DictComp nodes). > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From mertz at gnosis.cx Sun Jul 15 13:16:26 2018 From: mertz at gnosis.cx (David Mertz) Date: Sun, 15 Jul 2018 13:16:26 -0400 Subject: [Python-Dev] Naming comprehension syntax [was Re: Informal educator feedback on PEP 572 ...] In-Reply-To: References: <5B33936E.8080309@canterbury.ac.nz> <5B3423E8.3080208@canterbury.ac.nz> <20180702153411.GB14437@ando.pearwood.info> <5B3C0310.2010102@canterbury.ac.nz> <20180704001027.GI14437@ando.pearwood.info> <5B3C6F3F.8050400@canterbury.ac.nz> Message-ID: I've pretty much always taught using the "comprehension" term. It makes sense to introduce comprehensions on concrete collections first. Then I typically say something like "This special kind of comprehension is called a 'generator expression'". Usually that's accompanied by a motivation like creating a listcomp of a hundred million items, then showing that creating the generator expression is instantaneous. However, after the initial introduction, I consistently call it a generator expression. Albeit, for the students I teach?data scientists and quants, mostly?there's not a lot of time spent on that being the introduction (by then I'm talking about NumPy and Pandas, and scikit-learn, and Seaborn, abd Statsmodels, and so on. If things are "lazy" it's because they are Dask deferred or Dask DataFrame. On Sun, Jul 15, 2018, 1:02 PM Chris Barker via Python-Dev < python-dev at python.org> wrote: > Thanks Nick, > > I'll adopt this approach when I update my teaching materials. > > If I think of it, I"ll post here when I do that > > -CHB > > > On Sun, Jul 15, 2018 at 12:21 AM, Nick Coghlan wrote: > >> On 13 July 2018 at 15:30, Chris Barker via Python-Dev >> wrote: >> > On Mon, Jul 9, 2018 at 3:18 PM, Guido van Rossum >> wrote: >> >> >> >> Definitely docs first. And we should keep references to generator >> >> expressions too, if only to explain that they've been renamed. >> >> >> >> Perhaps someone should try it out in a 3rd party tutorial to see how it >> >> goes? >> > >> > >> > I'm not sure what "trying it out in a tutorial" would look like. >> > >> > I try to be pretty clear about terminology when I teach newbies -- so I >> > don't want to tell anyone this new thing is called a "generator >> > comprehension" if they aren't going to see that term anywhere else. >> >> Nina Zakharenko made the "they're officially called generator >> expressions, but I find it more helpful to think of them as generator >> comprehensions" observation in her PyCon 2018 presentation on "Elegant >> Solutions for Everyday Python Problems": >> >> https://www.slideshare.net/nnja/elegant-solutions-for-everyday-python-problems-pycon-2018/27 >> >> The article from Ned Batchelder mentioned in that slide is this one, >> which goes through the history of Raymond originally proposing the >> notion as generator comprehensions, them getting changed to generator >> expressions during the PEP 289 discussions, and then asking if it >> might be worth going back to the originally suggested name: >> https://nedbatchelder.com/blog/201605/generator_comprehensions.html >> >> And then in PEP 572, it was found that being able to group all 4 >> constructs (list/set/dict comps + generator expressions) under a >> single term was a genuinely useful shorthand: >> https://www.python.org/dev/peps/pep-0572/#scope-of-the-target >> >> So trying out the terminology in a tutorial context would be to do >> something similar to what Nina did in her talk: introduce the notion >> of list/set/dict/generator comprehensions, and then make a side note >> that the latter are officially referred to as "generator expressions". >> >> This wouldn't be the first time that terminology has differed between >> Python-as-commonly-taught and Python-as-formally-defined, as I've yet >> to hear anyone refer to container displays outside a language design >> discussion - everyone else calls them container literals (or, more >> typically, a literal for the specific builtin container type being >> discussed). >> >> In this case, though, we'd be considering eventually changing the >> language reference as well, and perhaps even some day the AST node >> name (from GeneratorExp to GeneratorComp). >> >> We wouldn't need to change anything in the grammar definition (since >> that already shares the comp_for and comp_if syntax definitions >> between container comprehensions and generator expressions), or the >> AST node structure (since GeneratorExp already uses a "comprehensions" >> attribute, the same as the ListComp/SetComp/DictComp nodes). >> >> Cheers, >> Nick. >> >> -- >> Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia >> > > > > -- > > Christopher Barker, Ph.D. > Oceanographer > > Emergency Response Division > NOAA/NOS/OR&R (206) 526-6959 voice > 7600 Sand Point Way NE (206) 526-6329 fax > Seattle, WA 98115 (206) 526-6317 main reception > > Chris.Barker at noaa.gov > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mertz%40gnosis.cx > -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Mon Jul 16 12:33:08 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 16 Jul 2018 18:33:08 +0200 Subject: [Python-Dev] PEP 580 (C call protocol) minor update Message-ID: <5B4CC8C4.2060300@UGent.be> I made some minor updates to PEP 580 (PEP editors: please merge https://github.com/python/peps/pull/741) and its reference implementation: - Added a new introductory section explaining the basic idea. - The C protocol no longer deals with __name__; a __name__ attribute is required but the protocol does not deal with its implementation. - The PEP no longer deals with profiling. This means that profiling only works for actual instances of builtin_function_or_method and method_descriptor. Profiling arbitrary callables would be nice, but that is deferred to a future PEP. The last two items are meant to simplify the PEP (although this is debatable since "simple" is very subjective). Enjoy! Jeroen. From acataldo at google.com Mon Jul 16 13:21:13 2018 From: acataldo at google.com (Adam Cataldo) Date: Mon, 16 Jul 2018 10:21:13 -0700 Subject: [Python-Dev] Question about PEP 484 Message-ID: *Hi Folks,Cc: Rebecca, pytypeThis is Adam Cataldo; I?m the engineering manager for the Python team at Google. Rebecca Chen, our lead pytype contributor, and I are interested in helping finalize PEP 484 if possible. To that end, we wanted to find out what technical issues the PEP 484 authors feel they still need to finalize. We also wanted to know how we can help.We have a large Python code base at Google, and may be able to use this to help resolve current incomplete definitions, by collecting data on how types are used. We also have a couple ambiguities that we?d love to get closure on: - One thing we care about in particular, given the implementation of pytype, is the detailed definition of what goes in a .pyi file. Do folks think this makes sense to include as part of PEP 484, or would this be better in a separate PEP? We?d love to get your thoughts.- The relationship between unicode and typing.Text in Python 2 has been a recurring source of confusion for our users. Especially since we contributed to the current state of affairs, we?d like to figure out how to establish clarity here.Thoughts?Thanks,Adam* -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Mon Jul 16 14:04:46 2018 From: brett at python.org (Brett Cannon) Date: Mon, 16 Jul 2018 11:04:46 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: On Mon, 16 Jul 2018 at 10:32 Adam Cataldo via Python-Dev < python-dev at python.org> wrote: > > > > > > *Hi Folks,Cc: Rebecca, pytypeThis is Adam Cataldo; I?m the engineering > manager for the Python team at Google. Rebecca Chen, our lead pytype > contributor, and I are interested in > helping finalize PEP 484 if possible. To that end, we wanted to find out > what technical issues the PEP 484 authors feel they still need to finalize. > We also wanted to know how we can help.We have a large Python code base at > Google, and may be able to use this to help resolve current incomplete > definitions, by collecting data on how types are used. We also have a > couple ambiguities that we?d love to get closure on: - One thing we care > about in particular, given the implementation of pytype, is the detailed > definition of what goes in a .pyi file. Do folks think this makes sense to > include as part of PEP 484, or would this be better in a separate PEP? We?d > love to get your thoughts.* > What specifically do you want beyond https://www.python.org/dev/peps/pep-0484/#stub-files? > > > > * - The relationship between unicode and typing.Text in Python 2 has been > a recurring source of confusion for our users. Especially since we > contributed to the current state > of affairs, we?d like to figure out how to establish clarity here.Thoughts?* > Do be aware, Adam, that due to Guido's retirement last week people might be a bit preoccupied and so a little slow in responding. But then again Guido just got a bit more free time so he might chime in on this one. ;) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tsudol at google.com Mon Jul 16 15:48:33 2018 From: tsudol at google.com (Teddy Sudol) Date: Mon, 16 Jul 2018 12:48:33 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: Hi, my name is Teddy Sudol. I work with Adam and Rebecca on pytype. The explanation of stub files is unclear. The section you linked starts with, "Stub files are files containing type hints that are only for use by the type checker, not at runtime." According to https://www.python.org/dev/peps/pep-0484/#acceptable-type-hints, type hints may be classes, abstract base classes, types defined in the `types` and `typing` modules, type variables, type aliases and None. Further in the section you linked, PEP 484 also states, "Stub files have the same syntax as regular Python modules," and, "no runtime behavior should be expected of stub files." "Have the same syntax as regular Python modules" and "are files containing type hints" are at odds with each other. This has led to compatibility issues between Mypy and pytype. For example, `b''` is not a valid type annotation, but until a month ago, `codecs.pyi` in typeshed used exactly that: https://github.com/python/typeshed/commit/6bbf3d89eb9b6c3fd5b0c0f632b2ad9258cecf15#diff-5f6f48c425bc0c283784cf5277880c0cL95. If statements can be useful for things like version checks, but on the other hand, pyi files aren't supposed to have any runtime behavior. Additionally, codifying the syntax for pyi files would settle questions like whether constants should be typed using "x: " or "x = ... # type: ". We would like to see a clear statement about the syntax of stub files. Personally, I think they should be a subset of Python, but I'd also be happy with an EBNF grammar for them. -- Teddy On Mon, Jul 16, 2018 at 11:05 AM Brett Cannon wrote: > > > On Mon, 16 Jul 2018 at 10:32 Adam Cataldo via Python-Dev < > python-dev at python.org> wrote: > >> >> >> >> >> >> *Hi Folks,Cc: Rebecca, pytypeThis is Adam Cataldo; I?m the engineering >> manager for the Python team at Google. Rebecca Chen, our lead pytype >> contributor, and I are interested in >> helping finalize PEP 484 if possible. To that end, we wanted to find out >> what technical issues the PEP 484 authors feel they still need to finalize. >> We also wanted to know how we can help.We have a large Python code base at >> Google, and may be able to use this to help resolve current incomplete >> definitions, by collecting data on how types are used. We also have a >> couple ambiguities that we?d love to get closure on: - One thing we care >> about in particular, given the implementation of pytype, is the detailed >> definition of what goes in a .pyi file. Do folks think this makes sense to >> include as part of PEP 484, or would this be better in a separate PEP? We?d >> love to get your thoughts.* >> > > What specifically do you want beyond > https://www.python.org/dev/peps/pep-0484/#stub-files? > > >> >> >> >> * - The relationship between unicode and typing.Text in Python 2 has been >> a recurring source of confusion for our users. Especially since we >> contributed to the current state >> of affairs, we?d like to figure out how to establish clarity here.Thoughts?* >> > > Do be aware, Adam, that due to Guido's retirement last week people might > be a bit preoccupied and so a little slow in responding. But then again > Guido just got a bit more free time so he might chime in on this one. ;) > > -- > You received this message because you are subscribed to the Google Groups > "pytype" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pytype+unsubscribe at googlegroups.com. > To post to this group, send email to pytype at googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pytype/CAP1%3D2W4NxcsSdsiMrh55KhjkwgD0PGRcZJF_Azq3g6QFQ2oiAw%40mail.gmail.com > > . > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Mon Jul 16 15:58:09 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 16 Jul 2018 21:58:09 +0200 Subject: [Python-Dev] Specification of C function profiling? Message-ID: <5B4CF8D1.8070308@UGent.be> Hello, it seems to me that there is no clear specification for the sys.setprofile() event c_call: the docs say "A C function is about to be called" but it's not clear what that means exactly, in particular when that C function is an unbound method like list.append. I also noticed that Lib/test/test_sys_setprofile.py doesn't test any of the c_* events. I'm asking in the context of https://bugs.python.org/issue34125 I found out that list.append([], None) *does* generate a c_call event but list.append([], None, **{}) does not. Jeroen. From acataldo at google.com Mon Jul 16 16:02:46 2018 From: acataldo at google.com (Adam Cataldo) Date: Mon, 16 Jul 2018 13:02:46 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: Thanks Brett and Teddy, Just so it doesn't get lost in the shuffle as folks dive into details, I'll re-ask my earlier question about stub files. Assuming there is consensus that there is ambiguity to resolve in the current definition, is updating the section on stub files the preferred option? The only alternative I can think of is to pull this out into a separate PEP. I frankly have no opinion on what the best way to capture this is. We're happy to help out either way. On Mon, Jul 16, 2018 at 12:48 PM Teddy Sudol wrote: > Hi, my name is Teddy Sudol. I work with Adam and Rebecca on pytype. > > The explanation of stub files is unclear. The section you linked starts > with, "Stub files are files containing type hints that are only for use by > the type checker, not at runtime." According to > https://www.python.org/dev/peps/pep-0484/#acceptable-type-hints, type > hints may be classes, abstract base classes, types defined in the `types` > and `typing` modules, type variables, type aliases and None. Further in the > section you linked, PEP 484 also states, "Stub files have the same syntax > as regular Python modules," and, "no runtime behavior should be expected of > stub files." > > "Have the same syntax as regular Python modules" and "are files containing > type hints" are at odds with each other. This has led to compatibility > issues between Mypy and pytype. For example, `b''` is not a valid type > annotation, but until a month ago, `codecs.pyi` in typeshed used exactly > that: > https://github.com/python/typeshed/commit/6bbf3d89eb9b6c3fd5b0c0f632b2ad9258cecf15#diff-5f6f48c425bc0c283784cf5277880c0cL95. > If statements can be useful for things like version checks, but on the > other hand, pyi files aren't supposed to have any runtime behavior. > Additionally, codifying the syntax for pyi files would settle questions > like whether constants should be typed using "x: " or "x = ... > # type: ". > > We would like to see a clear statement about the syntax of stub files. > Personally, I think they should be a subset of Python, but I'd also be > happy with an EBNF grammar for them. > > -- Teddy > > > On Mon, Jul 16, 2018 at 11:05 AM Brett Cannon wrote: > >> >> >> On Mon, 16 Jul 2018 at 10:32 Adam Cataldo via Python-Dev < >> python-dev at python.org> wrote: >> >>> >>> >>> >>> >>> >>> *Hi Folks,Cc: Rebecca, pytypeThis is Adam Cataldo; I?m the engineering >>> manager for the Python team at Google. Rebecca Chen, our lead pytype >>> contributor, and I are interested in >>> helping finalize PEP 484 if possible. To that end, we wanted to find out >>> what technical issues the PEP 484 authors feel they still need to finalize. >>> We also wanted to know how we can help.We have a large Python code base at >>> Google, and may be able to use this to help resolve current incomplete >>> definitions, by collecting data on how types are used. We also have a >>> couple ambiguities that we?d love to get closure on: - One thing we care >>> about in particular, given the implementation of pytype, is the detailed >>> definition of what goes in a .pyi file. Do folks think this makes sense to >>> include as part of PEP 484, or would this be better in a separate PEP? We?d >>> love to get your thoughts.* >>> >> >> What specifically do you want beyond >> https://www.python.org/dev/peps/pep-0484/#stub-files? >> >> >>> >>> >>> >>> * - The relationship between unicode and typing.Text in Python 2 has >>> been a recurring source of confusion for our users. Especially since we >>> contributed to the current state >>> of affairs, we?d like to figure out how to establish clarity here.Thoughts?* >>> >> >> Do be aware, Adam, that due to Guido's retirement last week people might >> be a bit preoccupied and so a little slow in responding. But then again >> Guido just got a bit more free time so he might chime in on this one. ;) >> >> -- >> You received this message because you are subscribed to the Google Groups >> "pytype" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to pytype+unsubscribe at googlegroups.com. >> To post to this group, send email to pytype at googlegroups.com. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/pytype/CAP1%3D2W4NxcsSdsiMrh55KhjkwgD0PGRcZJF_Azq3g6QFQ2oiAw%40mail.gmail.com >> >> . >> For more options, visit https://groups.google.com/d/optout. >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Jul 16 16:40:51 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Jul 2018 13:40:51 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: As one of the authors of PEP 484, *I* never thought there was an ambiguity here. The intention was for stub files to conform to the same grammar as regular .py files, but with a different interpretation. > "Have the same syntax as regular Python modules" and "are files containing type hints" are at odds with each other. That depends. *same syntax as regular Python* is normative while *containing type hints* is an informal description of intent. I happen to be somewhat familiar with the situation that lead to this question -- pytype has its own parser for stub files that cannot parse all Python constructs. But claiming that PEP 484 is ambiguous feels wrong, and if we really need to clarify it the only way to go is to make "same syntax as regular Python" more clearly normative. Type checkers should of course feel free ignore everything they don't care about. Regarding the unicode issue, that is indeed unfortunate, and there's a long but inconclusive discussion at https://github.com/python/typing/issues/208. (If you want a longer discussion here please start a new subject.) On Mon, Jul 16, 2018 at 1:02 PM, Adam Cataldo via Python-Dev < python-dev at python.org> wrote: > Thanks Brett and Teddy, > > Just so it doesn't get lost in the shuffle as folks dive into details, > I'll re-ask my earlier question about stub files. Assuming there is > consensus that there is ambiguity to resolve in the current definition, is > updating the section on stub files the preferred option? The only > alternative I can think of is to pull this out into a separate PEP. I > frankly have no opinion on what the best way to capture this is. We're > happy to help out either way. > > On Mon, Jul 16, 2018 at 12:48 PM Teddy Sudol wrote: > >> Hi, my name is Teddy Sudol. I work with Adam and Rebecca on pytype. >> >> The explanation of stub files is unclear. The section you linked starts >> with, "Stub files are files containing type hints that are only for use by >> the type checker, not at runtime." According to >> https://www.python.org/dev/peps/pep-0484/#acceptable-type-hints, type >> hints may be classes, abstract base classes, types defined in the `types` >> and `typing` modules, type variables, type aliases and None. Further in the >> section you linked, PEP 484 also states, "Stub files have the same syntax >> as regular Python modules," and, "no runtime behavior should be expected of >> stub files." >> >> "Have the same syntax as regular Python modules" and "are files >> containing type hints" are at odds with each other. This has led to >> compatibility issues between Mypy and pytype. For example, `b''` is not a >> valid type annotation, but until a month ago, `codecs.pyi` in typeshed used >> exactly that: https://github.com/python/typeshed/commit/ >> 6bbf3d89eb9b6c3fd5b0c0f632b2ad9258cecf15#diff- >> 5f6f48c425bc0c283784cf5277880c0cL95. If statements can be useful for >> things like version checks, but on the other hand, pyi files aren't >> supposed to have any runtime behavior. Additionally, codifying the syntax >> for pyi files would settle questions like whether constants should be typed >> using "x: " or "x = ... # type: ". >> >> We would like to see a clear statement about the syntax of stub files. >> Personally, I think they should be a subset of Python, but I'd also be >> happy with an EBNF grammar for them. >> >> -- Teddy >> >> >> On Mon, Jul 16, 2018 at 11:05 AM Brett Cannon wrote: >> >>> >>> >>> On Mon, 16 Jul 2018 at 10:32 Adam Cataldo via Python-Dev < >>> python-dev at python.org> wrote: >>> >>>> >>>> >>>> >>>> >>>> >>>> *Hi Folks,Cc: Rebecca, pytypeThis is Adam Cataldo; I?m the engineering >>>> manager for the Python team at Google. Rebecca Chen, our lead pytype >>>> contributor, and I are interested in >>>> helping finalize PEP 484 if possible. To that end, we wanted to find out >>>> what technical issues the PEP 484 authors feel they still need to finalize. >>>> We also wanted to know how we can help.We have a large Python code base at >>>> Google, and may be able to use this to help resolve current incomplete >>>> definitions, by collecting data on how types are used. We also have a >>>> couple ambiguities that we?d love to get closure on: - One thing we care >>>> about in particular, given the implementation of pytype, is the detailed >>>> definition of what goes in a .pyi file. Do folks think this makes sense to >>>> include as part of PEP 484, or would this be better in a separate PEP? We?d >>>> love to get your thoughts.* >>>> >>> >>> What specifically do you want beyond https://www.python.org/dev/ >>> peps/pep-0484/#stub-files? >>> >>> >>>> >>>> >>>> >>>> * - The relationship between unicode and typing.Text in Python 2 has >>>> been a recurring source of confusion for our users. Especially since we >>>> contributed to the current state >>>> of affairs, we?d like to figure out how to establish clarity here.Thoughts?* >>>> >>> >>> Do be aware, Adam, that due to Guido's retirement last week people might >>> be a bit preoccupied and so a little slow in responding. But then again >>> Guido just got a bit more free time so he might chime in on this one. ;) >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "pytype" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to pytype+unsubscribe at googlegroups.com. >>> To post to this group, send email to pytype at googlegroups.com. >>> To view this discussion on the web visit https://groups.google.com/d/ >>> msgid/pytype/CAP1%3D2W4NxcsSdsiMrh55KhjkwgD0PGRc >>> ZJF_Azq3g6QFQ2oiAw%40mail.gmail.com >>> >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From jelle.zijlstra at gmail.com Mon Jul 16 17:06:58 2018 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Mon, 16 Jul 2018 14:06:58 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: 2018-07-16 10:21 GMT-07:00 Adam Cataldo via Python-Dev < python-dev at python.org>: > > > > > > *Hi Folks,Cc: Rebecca, pytypeThis is Adam Cataldo; I?m the engineering > manager for the Python team at Google. Rebecca Chen, our lead pytype > contributor, and I are interested in > helping finalize PEP 484 if possible. To that end, we wanted to find out > what technical issues the PEP 484 authors feel they still need to finalize. > We also wanted to know how we can help.We have a large Python code base at > Google, and may be able to use this to help resolve current incomplete > definitions, by collecting data on how types are used. We also have a > couple ambiguities that we?d love to get closure on: - One thing we care > about in particular, given the implementation of pytype, is the detailed > definition of what goes in a .pyi file. Do folks think this makes sense to > include as part of PEP 484, or would this be better in a separate PEP? We?d > love to get your thoughts.* > I would be happy to contribute to this if that would be useful for type checkers like pytype, although like Guido I don't think the current text of the PEP is ambiguous. Typeshed already has a set of lint rules that limit what can be put in stub files, but it could be useful to communicate the exact set of allowed constructs. > > * - The relationship between unicode and typing.Text in Python 2 has been > a recurring source of confusion for our users. Especially since we > contributed to the current state > of affairs, we?d like to figure out how to establish clarity here.* > There has already been a long discussion on the typing issue tracker that resulted in no consensus for any change to the current recommendation. Perhaps there is something we can do, but it's not clear to me what that would be. > > > *Thoughts?Thanks,Adam* > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle. > zijlstra%40gmail.com > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Jul 16 17:56:38 2018 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 16 Jul 2018 14:56:38 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: On Mon, Jul 16, 2018 at 1:44 PM Guido van Rossum wrote: > As one of the authors of PEP 484, *I* never thought there was an ambiguity > here. The intention was for stub files to conform to the same grammar as > regular .py files, but with a different interpretation. > > > "Have the same syntax as regular Python modules" and "are files > containing type hints" are at odds with each other. > > That depends. *same syntax as regular Python* is normative while > *containing type hints* is an informal description of intent. > > I happen to be somewhat familiar with the situation that lead to this > question -- pytype has its own parser for stub files that cannot parse all > Python constructs. But claiming that PEP 484 is ambiguous feels wrong, and > if we really need to clarify it the only way to go is to make "same syntax > as regular Python" more clearly normative. Type checkers should of course > feel free ignore everything they don't care about. > I feel like the "same syntax as regular Python" is too broad a statement. That effectively requires a version specific Python interpreter to execute the files. With at least four different Python static analyzers in existence today, keeping the behavior of all of them consistent is important. Otherwise pyi files will be (are being) created that are analyzer specific and break other type checkers when distributed. ex: We're encountering pyi files with conditional logic in them. I believe we've encountered pyi files with del statements in them? Both of these are a slippery slope towards being turing complete in something that isn't supposed to be code. I don't like this. Interface declarations should not contain logic. If we allow conditions, we need to explicitly define what we do allow in the PEP. (if+else and del? what inputs are allowed for the expression in if statements?). Otherwise at some point someone is going to create a pyi file containing loops, function calls, and generator expressions and expect it to _do_ something. The horror! Lets avoid that. PEP-484 does contain the text, "This also reinforces the notion that no runtime behavior should be expected of stub files." But reinforcing a notion is not what I read as a concrete statement. I'd rather see that say something like, "There must not be any runtime behavior from a stub file. They will be parsed for information, not executed." Wordsmith that all you want, I'm not pedantic enough. :) I expect someone pedantic to happily point out that a def or class or assignment to ... with an annotation is runtime behavior... technically correct, but that isn't how people like me think of them in this context. We use a Pythonic syntax for stubs to be consistent with the language, that doesn't mean they are code. I wrote more than I thought I would here, I'll stop now. :) -gps > > Regarding the unicode issue, that is indeed unfortunate, and there's a > long but inconclusive discussion at > https://github.com/python/typing/issues/208. (If you want a longer > discussion here please start a new subject.) > > On Mon, Jul 16, 2018 at 1:02 PM, Adam Cataldo via Python-Dev < > python-dev at python.org> wrote: > >> Thanks Brett and Teddy, >> >> Just so it doesn't get lost in the shuffle as folks dive into details, >> I'll re-ask my earlier question about stub files. Assuming there is >> consensus that there is ambiguity to resolve in the current definition, is >> updating the section on stub files the preferred option? The only >> alternative I can think of is to pull this out into a separate PEP. I >> frankly have no opinion on what the best way to capture this is. We're >> happy to help out either way. >> >> On Mon, Jul 16, 2018 at 12:48 PM Teddy Sudol wrote: >> >>> Hi, my name is Teddy Sudol. I work with Adam and Rebecca on pytype. >>> >>> The explanation of stub files is unclear. The section you linked starts >>> with, "Stub files are files containing type hints that are only for use by >>> the type checker, not at runtime." According to >>> https://www.python.org/dev/peps/pep-0484/#acceptable-type-hints, type >>> hints may be classes, abstract base classes, types defined in the `types` >>> and `typing` modules, type variables, type aliases and None. Further in the >>> section you linked, PEP 484 also states, "Stub files have the same syntax >>> as regular Python modules," and, "no runtime behavior should be expected of >>> stub files." >>> >>> "Have the same syntax as regular Python modules" and "are files >>> containing type hints" are at odds with each other. This has led to >>> compatibility issues between Mypy and pytype. For example, `b''` is not a >>> valid type annotation, but until a month ago, `codecs.pyi` in typeshed used >>> exactly that: >>> https://github.com/python/typeshed/commit/6bbf3d89eb9b6c3fd5b0c0f632b2ad9258cecf15#diff-5f6f48c425bc0c283784cf5277880c0cL95. >>> If statements can be useful for things like version checks, but on the >>> other hand, pyi files aren't supposed to have any runtime behavior. >>> Additionally, codifying the syntax for pyi files would settle questions >>> like whether constants should be typed using "x: " or "x = ... >>> # type: ". >>> >>> We would like to see a clear statement about the syntax of stub files. >>> Personally, I think they should be a subset of Python, but I'd also be >>> happy with an EBNF grammar for them. >>> >>> -- Teddy >>> >>> >>> On Mon, Jul 16, 2018 at 11:05 AM Brett Cannon wrote: >>> >>>> >>>> >>>> On Mon, 16 Jul 2018 at 10:32 Adam Cataldo via Python-Dev < >>>> python-dev at python.org> wrote: >>>> >>>>> >>>>> >>>>> >>>>> >>>>> >>>>> *Hi Folks,Cc: Rebecca, pytypeThis is Adam Cataldo; I?m the engineering >>>>> manager for the Python team at Google. Rebecca Chen, our lead pytype >>>>> contributor, and I are interested in >>>>> helping finalize PEP 484 if possible. To that end, we wanted to find out >>>>> what technical issues the PEP 484 authors feel they still need to finalize. >>>>> We also wanted to know how we can help.We have a large Python code base at >>>>> Google, and may be able to use this to help resolve current incomplete >>>>> definitions, by collecting data on how types are used. We also have a >>>>> couple ambiguities that we?d love to get closure on: - One thing we care >>>>> about in particular, given the implementation of pytype, is the detailed >>>>> definition of what goes in a .pyi file. Do folks think this makes sense to >>>>> include as part of PEP 484, or would this be better in a separate PEP? We?d >>>>> love to get your thoughts.* >>>>> >>>> >>>> What specifically do you want beyond >>>> https://www.python.org/dev/peps/pep-0484/#stub-files? >>>> >>>> >>>>> >>>>> >>>>> >>>>> * - The relationship between unicode and typing.Text in Python 2 has >>>>> been a recurring source of confusion for our users. Especially since we >>>>> contributed to the current state >>>>> of affairs, we?d like to figure out how to establish clarity here.Thoughts?* >>>>> >>>> >>>> Do be aware, Adam, that due to Guido's retirement last week people >>>> might be a bit preoccupied and so a little slow in responding. But then >>>> again Guido just got a bit more free time so he might chime in on this one. >>>> ;) >>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "pytype" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to pytype+unsubscribe at googlegroups.com. >>>> To post to this group, send email to pytype at googlegroups.com. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/pytype/CAP1%3D2W4NxcsSdsiMrh55KhjkwgD0PGRcZJF_Azq3g6QFQ2oiAw%40mail.gmail.com >>>> >>>> . >>>> For more options, visit https://groups.google.com/d/optout. >>>> >>> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/guido%40python.org >> >> > > > -- > --Guido van Rossum (python.org/~guido) > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Jul 16 18:15:10 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Jul 2018 15:15:10 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: On Mon, Jul 16, 2018 at 2:56 PM, Gregory P. Smith wrote: > > On Mon, Jul 16, 2018 at 1:44 PM Guido van Rossum wrote: > >> As one of the authors of PEP 484, *I* never thought there was an >> ambiguity here. The intention was for stub files to conform to the same >> grammar as regular .py files, but with a different interpretation. >> >> > "Have the same syntax as regular Python modules" and "are files >> containing type hints" are at odds with each other. >> >> That depends. *same syntax as regular Python* is normative while >> *containing type hints* is an informal description of intent. >> >> I happen to be somewhat familiar with the situation that lead to this >> question -- pytype has its own parser for stub files that cannot parse all >> Python constructs. But claiming that PEP 484 is ambiguous feels wrong, and >> if we really need to clarify it the only way to go is to make "same syntax >> as regular Python" more clearly normative. Type checkers should of course >> feel free ignore everything they don't care about. >> > > I feel like the "same syntax as regular Python" is too broad a statement. > That effectively requires a version specific Python interpreter to execute > the files. With at least four different Python static analyzers in > existence today, keeping the behavior of all of them consistent is > important. Otherwise pyi files will be (are being) created that are > analyzer specific and break other type checkers when distributed. > it doesn't require an interpreter, just a parser. Is saying it should be syntactically valid Python 3.6 (though emphatically not executable!) still too much? > ex: We're encountering pyi files with conditional logic in them. I > believe we've encountered pyi files with del statements in them? Both of > these are a slippery slope towards being turing complete in something that > isn't supposed to be code. I don't like this. Interface declarations > should not contain logic. If we allow conditions, we need to explicitly > define what we do allow in the PEP. (if+else and del? what inputs are > allowed for the expression in if statements?). Otherwise at some point > someone is going to create a pyi file containing loops, function calls, and > generator expressions and expect it to _do_ something. The horror! Lets > avoid that. > Syntactically, conditional logic *is* part of the spec because of `sys.version_info` and `sys.platform` checks. These are mentioned in PEP 484 (though IIRC they were added at some point). Feel free to ignore `del` statements, but they are valid syntax. > PEP-484 does contain the text, "This also reinforces the notion that no > runtime behavior should be expected of stub files." But reinforcing a > notion is not what I read as a concrete statement. > > I'd rather see that say something like, "There must not be any runtime > behavior from a stub file. They will be parsed for information, not > executed." Wordsmith that all you want, I'm not pedantic enough. :) > I don't see that as any more precise as what the PEP says. :-) I'd be happy to claim that if the ast module can't parse a .pyi file, it's invalid, otherwise it's valid, except that `# type:` comments are significant and the ast module doesn't preserve those. > I expect someone pedantic to happily point out that a def or class or > assignment to ... with an annotation is runtime behavior... technically > correct, but that isn't how people like me think of them in this context. > And mypy agrees. :-) > We use a Pythonic syntax for stubs to be consistent with the language, > that doesn't mean they are code. > Define "code". To some people HTML is code. > I wrote more than I thought I would here, I'll stop now. :) > It would be nice if the pytype team could articulate the problems they're trying to solve, rather than offering to "help finalize PEP 484". My guess is that their parser for .pyi files only accepts a subset of Python and they're (you're? do you report to Adam?) reluctant to sink more time in that parser -- but I don't think that's my problem. :-) My intention for stubs was exactly what I've been saying in this thread: they must be syntactically valid Python, they are not meant to be executed, and type checkers are free to ignore things they don't need. If that's not sufficiently clear in the way the PEP is currently written, I welcome a PR to the peps repo. If the pytype team wants a different interpretation, the bar is much higher. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rechen at google.com Mon Jul 16 19:28:22 2018 From: rechen at google.com (Rebecca Chen) Date: Mon, 16 Jul 2018 16:28:22 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: Hi Guido and all, Let me try to clarify our (pytype team's) intentions a bit. Our overall goal is to see PEP 484 finalized, in the interest of having a definitive type-checking reference to point to and work off of. We're willing and eager to help with this, if we get some guidance on what the authors are still trying to finalize and how we can be useful. As a conversation starter, we offered two issues in which pytype has some personal stake. Best, Rebecca P.S. Yes, Adam manages Python at Google. On Mon, Jul 16, 2018 at 3:15 PM Guido van Rossum wrote: > > On Mon, Jul 16, 2018 at 2:56 PM, Gregory P. Smith wrote: >> >> >> On Mon, Jul 16, 2018 at 1:44 PM Guido van Rossum wrote: >>> >>> As one of the authors of PEP 484, *I* never thought there was an ambiguity here. The intention was for stub files to conform to the same grammar as regular .py files, but with a different interpretation. >>> >>> > "Have the same syntax as regular Python modules" and "are files containing type hints" are at odds with each other. >>> >>> That depends. *same syntax as regular Python* is normative while *containing type hints* is an informal description of intent. >>> >>> I happen to be somewhat familiar with the situation that lead to this question -- pytype has its own parser for stub files that cannot parse all Python constructs. But claiming that PEP 484 is ambiguous feels wrong, and if we really need to clarify it the only way to go is to make "same syntax as regular Python" more clearly normative. Type checkers should of course feel free ignore everything they don't care about. >> >> >> I feel like the "same syntax as regular Python" is too broad a statement.? That effectively requires a version specific Python interpreter to execute the files.? With at least four different Python static analyzers in existence today, keeping the behavior of all of them consistent is important.? Otherwise pyi files will be (are being) created that are analyzer specific and break other type checkers when distributed. > > > it doesn't require an interpreter, just a parser. Is saying it should be syntactically valid Python 3.6 (though emphatically not executable!) still too much? > >> >> ex: We're encountering pyi files with conditional logic in them.? I believe we've encountered pyi files with del statements in them?? Both of these are a slippery slope towards being turing complete in something that isn't supposed to be code.? I don't like this.? Interface declarations should not contain logic.? If we allow conditions, we need to explicitly define what we do allow in the PEP.? (if+else and del?? what inputs are allowed for the expression in if statements?).? Otherwise at some point someone is going to create a pyi file containing loops, function calls, and generator expressions and expect it to _do_ something.? The horror!? Lets avoid that. > > > Syntactically, conditional logic *is* part of the spec because of `sys.version_info` and `sys.platform` checks. These are mentioned in PEP 484 (though IIRC they were added at some point). > > Feel free to ignore `del` statements, but they are valid syntax. > >> >> PEP-484 does contain the text, "This also reinforces the notion that no runtime behavior should be expected of stub files." But reinforcing a notion is not what I read as a concrete statement. >> >> I'd rather see that say something like, "There must not be any runtime behavior from a stub file. They will be parsed for information, not executed."? Wordsmith that all you want, I'm not pedantic enough. :) > > > I don't see that as any more precise as what the PEP says. :-) > > I'd be happy to claim that if the ast module can't parse a .pyi file, it's invalid, otherwise it's valid, except that `# type:` comments are significant and the ast module doesn't preserve those. > >> >> I expect someone pedantic to happily point out that a def or class or assignment to ... with an annotation is runtime behavior... technically correct, but that isn't how people like me think of them in this context. > > > And mypy agrees. :-) > >> >> We use a Pythonic syntax for stubs to be consistent with the language, that doesn't mean they are code. > > > Define "code". To some people HTML is code. > >> >> I wrote more than I thought I would here, I'll stop now. :) > > > It would be nice if the pytype team could articulate the problems they're trying to solve, rather than offering to "help finalize PEP 484". My guess is that their parser for .pyi files only accepts a subset of Python and they're (you're? do you report to Adam?) reluctant to sink more time in that parser -- but I don't think that's my problem. :-) > > My intention for stubs was exactly what I've been saying in this thread: they must be syntactically valid Python, they are not meant to be executed, and type checkers are free to ignore things they don't need. If that's not sufficiently clear in the way the PEP is currently written, I welcome a PR to the peps repo. If the pytype team wants a different interpretation, the bar is much higher. > > -- > --Guido van Rossum (python.org/~guido) From guido at python.org Mon Jul 16 20:06:12 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 16 Jul 2018 17:06:12 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: Thanks, Rebecca! So the status of PEP 484 is that we don't want to add new features to it -- those require their own PEP. But we do accept PRs for clarifications, assuming the clarifications are about issues where the intention is clear but the text is not (usually because we thought there was only one possible interpretation). If PEP 484 had intended to only allow a specific subset of Python *syntax* we surely would have specified that subset -- so I think the intention is clear (enough). Unfortunately a lot of PEP 484 is written in the form of examples, and in many cases it would be possible to quibble about edge cases not covered by the examples. (An example that just came up: is `#type: foo` a valid type comment? What about `# type:foo`? Or `# < type: foo`? The intention was `#\s*type:\s*` but all the PEP examples use exactly one space.) On Mon, Jul 16, 2018 at 4:28 PM, Rebecca Chen wrote: > Hi Guido and all, > > Let me try to clarify our (pytype team's) intentions a bit. Our > overall goal is to see PEP 484 finalized, in the interest of having a > definitive type-checking reference to point to and work off of. We're > willing and eager to help with this, if we get some guidance on what > the authors are still trying to finalize and how we can be useful. As > a conversation starter, we offered two issues in which pytype has some > personal stake. > > Best, > Rebecca > > P.S. Yes, Adam manages Python at Google. > > On Mon, Jul 16, 2018 at 3:15 PM Guido van Rossum wrote: > > > > On Mon, Jul 16, 2018 at 2:56 PM, Gregory P. Smith > wrote: > >> > >> > >> On Mon, Jul 16, 2018 at 1:44 PM Guido van Rossum > wrote: > >>> > >>> As one of the authors of PEP 484, *I* never thought there was an > ambiguity here. The intention was for stub files to conform to the same > grammar as regular .py files, but with a different interpretation. > >>> > >>> > "Have the same syntax as regular Python modules" and "are files > containing type hints" are at odds with each other. > >>> > >>> That depends. *same syntax as regular Python* is normative while > *containing type hints* is an informal description of intent. > >>> > >>> I happen to be somewhat familiar with the situation that lead to this > question -- pytype has its own parser for stub files that cannot parse all > Python constructs. But claiming that PEP 484 is ambiguous feels wrong, and > if we really need to clarify it the only way to go is to make "same syntax > as regular Python" more clearly normative. Type checkers should of course > feel free ignore everything they don't care about. > >> > >> > >> I feel like the "same syntax as regular Python" is too broad a > statement. That effectively requires a version specific Python interpreter > to execute the files. With at least four different Python static analyzers > in existence today, keeping the behavior of all of them consistent is > important. Otherwise pyi files will be (are being) created that are > analyzer specific and break other type checkers when distributed. > > > > > > it doesn't require an interpreter, just a parser. Is saying it should be > syntactically valid Python 3.6 (though emphatically not executable!) still > too much? > > > >> > >> ex: We're encountering pyi files with conditional logic in them. I > believe we've encountered pyi files with del statements in them? Both of > these are a slippery slope towards being turing complete in something that > isn't supposed to be code. I don't like this. Interface declarations > should not contain logic. If we allow conditions, we need to explicitly > define what we do allow in the PEP. (if+else and del? what inputs are > allowed for the expression in if statements?). Otherwise at some point > someone is going to create a pyi file containing loops, function calls, and > generator expressions and expect it to _do_ something. The horror! Lets > avoid that. > > > > > > Syntactically, conditional logic *is* part of the spec because of > `sys.version_info` and `sys.platform` checks. These are mentioned in PEP > 484 (though IIRC they were added at some point). > > > > Feel free to ignore `del` statements, but they are valid syntax. > > > >> > >> PEP-484 does contain the text, "This also reinforces the notion that no > runtime behavior should be expected of stub files." But reinforcing a > notion is not what I read as a concrete statement. > >> > >> I'd rather see that say something like, "There must not be any runtime > behavior from a stub file. They will be parsed for information, not > executed." Wordsmith that all you want, I'm not pedantic enough. :) > > > > > > I don't see that as any more precise as what the PEP says. :-) > > > > I'd be happy to claim that if the ast module can't parse a .pyi file, > it's invalid, otherwise it's valid, except that `# type:` comments are > significant and the ast module doesn't preserve those. > > > >> > >> I expect someone pedantic to happily point out that a def or class or > assignment to ... with an annotation is runtime behavior... technically > correct, but that isn't how people like me think of them in this context. > > > > > > And mypy agrees. :-) > > > >> > >> We use a Pythonic syntax for stubs to be consistent with the language, > that doesn't mean they are code. > > > > > > Define "code". To some people HTML is code. > > > >> > >> I wrote more than I thought I would here, I'll stop now. :) > > > > > > It would be nice if the pytype team could articulate the problems > they're trying to solve, rather than offering to "help finalize PEP 484". > My guess is that their parser for .pyi files only accepts a subset of > Python and they're (you're? do you report to Adam?) reluctant to sink more > time in that parser -- but I don't think that's my problem. :-) > > > > My intention for stubs was exactly what I've been saying in this thread: > they must be syntactically valid Python, they are not meant to be executed, > and type checkers are free to ignore things they don't need. If that's not > sufficiently clear in the way the PEP is currently written, I welcome a PR > to the peps repo. If the pytype team wants a different interpretation, the > bar is much higher. > > > > -- > > --Guido van Rossum (python.org/~guido) > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From radim at rare-technologies.com Tue Jul 17 00:18:34 2018 From: radim at rare-technologies.com (=?UTF-8?B?UmFkaW0gxZhlaMWvxZllaw==?=) Date: Tue, 17 Jul 2018 06:18:34 +0200 Subject: [Python-Dev] Const access to CPython objects outside of GIL? Message-ID: Hi all, one of our Python projects calls for pretty heavy, low-level optimizations. We went down the rabbit hole and determined that having access to PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode) on Python objects **outside of GIL** might be a good-enough solution. The Python objects in question are guaranteed to live and not be mutated externally in any way. They're "frozen" and read-only. Under what conditions is it OK to call these 3 functions on such objects? More generally, what is the CPython 2.7/3.5 contract regarding (lack of) object mutation, and the need for reference counting and synchronization via GIL? Which C API functions are safe to call on "const" objects? Obviously releasing GIL and then calling C API is hacky, but from initial experiments, it seems to work (see https://stackoverflow.com/questions/51351609/can-i-const-access-cpython-objects-without-gil). But I'm wondering if there's a more formal contract around this behaviour. Cheers, Radim -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue Jul 17 01:15:04 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 17 Jul 2018 00:15:04 -0500 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: [Radim ?eh??ek ] > one of our Python projects calls for pretty heavy, low-level optimizations. > We went down the rabbit hole and determined that having access to > PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode) > on Python objects **outside of GIL** might be a good-enough solution. The > Python objects in question are guaranteed to live and not be mutated > externally in any way. They're "frozen" and read-only. > > Under what conditions is it OK to call these 3 functions on such objects? > >From the "initialization, Finalization, and Threads" section of the Python/C API Reference Manual: Therefore, the rule exists that only the thread that has acquired the GIL > may operate on Python objects or call Python/C API functions. Under protection of the GIL is the only documented - or intended - way to do anything with Python objects, or to call virtually any Python/C API function. > More generally, what is the CPython 2.7/3.5 contract regarding (lack of) > object mutation, and the need for reference counting and synchronization > via GIL? > There is no intention to support GIL-free access to any Python objects. So that's the contract: "All warranties are null & void if you do just about anything while not holding the GIL". > Which C API functions are safe to call on "const" objects? > None. If you find some combinations of CPython versions, functions, and objects, that happen to work, that's fine, but there's no guarantee they'll continue to work, not even across maintenance releases. Although they'll _probably_ continue to work. Obviously releasing GIL and then calling C API is hacky, but from initial > experiments, it seems to work (see > https://stackoverflow.com/questions/51351609/can-i-const-access-cpython-objects-without-gil). > But I'm wondering if there's a more formal contract around this behaviour. > Nope! You're on your own here. Which doesn't mean you can't do it. Just that if things blow up, you're still on your own - since you're far outside what the documentation says is required (which I suspect you knew before you asked ;-) ), the project won't even accept a bug report. If you want more help trying to guess which functions _may_ work outside the law, that's fine too, but "python-dev" (this mailing list) is for development _of_ Python itself, not for questions about _using_ Python. The more general python-list would be more appropriate. But, since you're trying to use the C API in unsupported ways it wasn't intended to be used, you'll likely have a hard time finding people with significant experience doing the same thing. Since it's not at all an intended use, there are no "general principles" at work to ease the pain of staring at the CPython source code to try to guess what might go wrong across all code paths. -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Jul 17 04:27:27 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 17 Jul 2018 10:27:27 +0200 Subject: [Python-Dev] Const access to CPython objects outside of GIL? References: Message-ID: <20180717102727.19956040@fsol> On Tue, 17 Jul 2018 00:15:04 -0500 Tim Peters wrote: > > More generally, what is the CPython 2.7/3.5 contract regarding (lack of) > > object mutation, and the need for reference counting and synchronization > > via GIL? > > There is no intention to support GIL-free access to any Python objects. So > that's the contract: "All warranties are null & void if you do just about > anything while not holding the GIL". Actually, accessing a Py_buffer that's been obtained in the regular way (e.g. with PyObject_GetBuffer) is safe even without the GIL. Regards Antoine. From storchaka at gmail.com Tue Jul 17 04:50:55 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Jul 2018 11:50:55 +0300 Subject: [Python-Dev] PEP 572 and assert Message-ID: Recently Barry shown an example: assert len(subdirs := list(path.iterdir())) == 0, subdirs It looks awful to me. It looks even worse than using asserts for validating the user input. The assert has a side effect, and it depends on the interpreter option (-O). Even if subdirs is not used outside of the assert *now*, it is easy to introduce an error later, and it is hard to notice it if tests are not ran with the -O option regularly. Does PEP 572 encourages writing such code, discourages this, or completely forbids? From rosuav at gmail.com Tue Jul 17 04:56:36 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 17 Jul 2018 18:56:36 +1000 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: On Tue, Jul 17, 2018 at 6:50 PM, Serhiy Storchaka wrote: > Recently Barry shown an example: > > assert len(subdirs := list(path.iterdir())) == 0, subdirs > > It looks awful to me. It looks even worse than using asserts for validating > the user input. The assert has a side effect, and it depends on the > interpreter option (-O). Even if subdirs is not used outside of the assert > *now*, it is easy to introduce an error later, and it is hard to notice it > if tests are not ran with the -O option regularly. > > Does PEP 572 encourages writing such code, discourages this, or completely > forbids? > Asserts with side effects are already a bad idea. PEP 572 makes no change to this. If you're putting any sort of side effects inside assertions, you're playing with fire. ChrisA From solipsis at pitrou.net Tue Jul 17 05:05:07 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 17 Jul 2018 11:05:07 +0200 Subject: [Python-Dev] PEP 572 and assert References: Message-ID: <20180717110507.0caa6478@fsol> On Tue, 17 Jul 2018 18:56:36 +1000 Chris Angelico wrote: > On Tue, Jul 17, 2018 at 6:50 PM, Serhiy Storchaka wrote: > > Recently Barry shown an example: > > > > assert len(subdirs := list(path.iterdir())) == 0, subdirs > > > > It looks awful to me. It looks even worse than using asserts for validating > > the user input. The assert has a side effect, and it depends on the > > interpreter option (-O). Even if subdirs is not used outside of the assert > > *now*, it is easy to introduce an error later, and it is hard to notice it > > if tests are not ran with the -O option regularly. > > > > Does PEP 572 encourages writing such code, discourages this, or completely > > forbids? > > > > Asserts with side effects are already a bad idea. PEP 572 makes no > change to this. If you're putting any sort of side effects inside > assertions, you're playing with fire. Well, PEP 572 makes it easier to put side effects in side assertions since you can now stuff an assignment inside an assert. Which is what Serhiy's / Barry's example shows. Regards Antoine. From srittau at rittau.biz Tue Jul 17 04:51:16 2018 From: srittau at rittau.biz (Sebastian Rittau) Date: Tue, 17 Jul 2018 10:51:16 +0200 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: On 16.07.2018 19:21, Adam Cataldo via Python-Dev wrote: > * > > * > > One thing we care about in particular, given the implementation of > pytype, is the detailed definition of what goes in a .pyi file. Do > folks think this makes sense to include as part of PEP 484, or > would this be better in a separate PEP? We?d love to get your > thoughts. > > * It would be useful to define - on a semantic, not syntactic level - what constructs a type checker is expected to understand. Not only for implementers, but more importantly for stub authors. Sometimes it's hard to judge, what constructs type checkers will understand. And while by now I think I personally have a solid understanding of what mypy understands, I have no idea whether that also applies to pytype, PyCharm, or other type checkers. For example, in one of my first pull requests for typeshed, I tried to use the following construct, expecting type checkers to understand it: ??? class Foo: ??????? def bar(self) -> None: ??????????? raise NotImplementedError() It seems they don't, but mypy understands: ??? class Foo: ??????? @abstractmethod ??????? def bar(self) -> None: ... But do I need to import abstractmethod? Would @abc.abstractmethod also work? Can I have an assignment "_am = abc.abstractmethod" and then @_am would work? Can I alias functions by using assignments in stubs or should I use a second function definition? How complex can Python version checks be? There are many more of those questions. If these expectations would be documents, implementers of type checkers can still decide not to support certain constructs (or not support them yet), or even to support more constructs. But at least such a deviation could be documented, so users know what to expect. On the other hand, stub authors will know what constructs will likely not work and should be avoided. ?- Sebastian -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Tue Jul 17 07:21:17 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 17 Jul 2018 13:21:17 +0200 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: 2018-07-17 10:50 GMT+02:00 Serhiy Storchaka : > assert len(subdirs := list(path.iterdir())) == 0, subdirs > > Does PEP 572 encourages writing such code, discourages this, or completely > forbids? If I understood correctly Guido, Python the language must not prevent developers to experiment various usage of assignement expressions. The f-string has a similar design: anything even f'{__import__("os").system("rm -rf /")}' is allowed. It's more the role of linters like flake8 to warn against "bad practices". I don't think that PEPs like f-string and assignment expressions must include *coding styles*. Sure, f-string and assignment expressions allow to write surprising and bad code. But you don't them them to write crappy code :-) I was strongly against f-string at the beginning because I immediately predicted that people will start to call functions (with side effects) in f-string... but I didn't see that happen. In fact, developers are reasonable and use f-string when it's appropriate and safe. Victor From vstinner at redhat.com Tue Jul 17 07:34:27 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 17 Jul 2018 13:34:27 +0200 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: 2018-07-17 6:18 GMT+02:00 Radim ?eh??ek : > one of our Python projects calls for pretty heavy, low-level optimizations. > > We went down the rabbit hole and determined that having access to > PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode) > on Python objects **outside of GIL** might be a good-enough solution. The > Python objects in question are guaranteed to live and not be mutated > externally in any way. They're "frozen" and read-only. IMHO it's a great path to introduce very tricky race conditions in multithreaded applications! :-) At the C level, even immutable Python objects are mutable: str, tuple, etc. IMHO you need a different approach to implement optimizations. For example, use your objects which don't rely on the GIL to be consistent. Sadly, I have no experience with that and so cannot provide any example. Python releases the GIL *often* and pass pointers to buffers to C functions. For example, os.stat() writes into a memory block allocated by Python. But it's a raw memory block, it's much simpler than a Python object like a tuple. Victor From guido at python.org Tue Jul 17 10:59:35 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Jul 2018 07:59:35 -0700 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: On Tue, Jul 17, 2018 at 1:50 AM, Serhiy Storchaka wrote: > Recently Barry shown an example: > > assert len(subdirs := list(path.iterdir())) == 0, subdirs > > It looks awful to me. It looks even worse than using asserts for > validating the user input. The assert has a side effect, and it depends on > the interpreter option (-O). Even if subdirs is not used outside of the > assert *now*, it is easy to introduce an error later, and it is hard to > notice it if tests are not ran with the -O option regularly. > > Does PEP 572 encourages writing such code, discourages this, or completely > forbids? > The PEP has no specific opinion except it is not forbidden. Personally I like Barry's example just fine -- assuming `subdirs` is not used later, this feels like a good use case. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jul 17 11:05:47 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Jul 2018 08:05:47 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: This is a good point. I presume specifying this unambiguously would be a huge amount of work, and it would mostly codify mypy's current behavior. I don't think that's within the scope of PEP 484, but it could well be done as a separate PEP (perhaps an informational one?). I hope you understand that I am not volunteering. On Tue, Jul 17, 2018 at 1:51 AM, Sebastian Rittau wrote: > On 16.07.2018 19:21, Adam Cataldo via Python-Dev wrote: > > > > > * - One thing we care about in particular, given the implementation of > pytype, is the detailed definition of what goes in a .pyi file. Do folks > think this makes sense to include as part of PEP 484, or would this be > better in a separate PEP? We?d love to get your thoughts. * > > It would be useful to define - on a semantic, not syntactic level - what > constructs a type checker is expected to understand. Not only for > implementers, but more importantly for stub authors. Sometimes it's hard to > judge, what constructs type checkers will understand. And while by now I > think I personally have a solid understanding of what mypy understands, I > have no idea whether that also applies to pytype, PyCharm, or other type > checkers. > > For example, in one of my first pull requests for typeshed, I tried to use > the following construct, expecting type checkers to understand it: > > class Foo: > def bar(self) -> None: > raise NotImplementedError() > > It seems they don't, but mypy understands: > > class Foo: > @abstractmethod > def bar(self) -> None: ... > > But do I need to import abstractmethod? Would @abc.abstractmethod also > work? Can I have an assignment "_am = abc.abstractmethod" and then @_am > would work? Can I alias functions by using assignments in stubs or should I > use a second function definition? How complex can Python version checks be? > There are many more of those questions. > > If these expectations would be documents, implementers of type checkers > can still decide not to support certain constructs (or not support them > yet), or even to support more constructs. But at least such a deviation > could be documented, so users know what to expect. On the other hand, stub > authors will know what constructs will likely not work and should be > avoided. > > - Sebastian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Tue Jul 17 11:22:00 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Tue, 17 Jul 2018 18:22:00 +0300 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: <082c7b0e-cc4a-c609-0444-df44dc914890@mail.mipt.ru> On 17.07.2018 7:18, Radim ?eh??ek wrote: > Hi all, > > one of our Python projects calls for pretty heavy, low-level > optimizations. > > We went down the rabbit hole and determined that having access to > PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, > unicode) on Python objects **outside of GIL** might be a good-enough > solution. The Python objects in question are guaranteed to live and > not be mutated externally in any way. They're "frozen" and read-only. > The standard practice if you need to access something outside of GIL is to get a private C object from it. For immutable types, you can get a pointer to the underlying data if the internal representation is compatible with some C type (e.g. char* |PyBytes_AsString|(PyObject/?*o/) and FILE* |PyFile_AsFile|(PyObject/?*p/) (Py2 only -- in Py3, PyFile no longer wraps stdio FILE) ); otherwise, the C API can produce a copy (e.g. wchar_t* |PyUnicode_AsWideCharString|(PyObject/?*unicode/, Py_ssize_t/?*size/) ). Though you can call whatever you want outside of GIL (it's not like we can prevent you), anything that's not officially guaranteed you're doing on your own risk. Even if it happens to work now, it can break in unpredictable ways at any point in the future. > Under what conditions is it OK to call these 3 functions on such objects? > > More generally, what is the CPython 2.7/3.5 contract regarding (lack > of) object mutation, and the need for reference counting and > synchronization via GIL? > > Which C API functions are safe to call on "const" objects? > > Obviously releasing GIL and then calling C API is hacky, but from > initial experiments, it seems to work (see > https://stackoverflow.com/questions/51351609/can-i-const-access-cpython-objects-without-gil). > But I'm wondering if there's a more formal contract around this behaviour. > > Cheers, > Radim > > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Jul 17 11:28:09 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Jul 2018 18:28:09 +0300 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: 17.07.18 17:59, Guido van Rossum ????: > The PEP has no specific opinion except it is not forbidden. > > Personally I like Barry's example just fine -- assuming `subdirs` is not > used later, this feels like a good use case. Shouldn't this problem be solved in the same way as for comprehensions? Should not the assert statement introduce an implicit lexical scope for preventing leaking variables? From guido at python.org Tue Jul 17 11:48:46 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Jul 2018 08:48:46 -0700 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka wrote: > 17.07.18 17:59, Guido van Rossum ????: > >> The PEP has no specific opinion except it is not forbidden. >> >> Personally I like Barry's example just fine -- assuming `subdirs` is not >> used later, this feels like a good use case. >> > > Shouldn't this problem be solved in the same way as for comprehensions? > No, it's nothing like those. > Should not the assert statement introduce an implicit lexical scope for > preventing leaking variables? > I don't see why. As Chris said, side effects in asserts are nothing new and this PEP is not the one to do something about it. Serhiy, have you even read the latest version of PEP 572? In its current form it does not introduce any implicit scopes and even goes out of its way to make assignments in comprehensions "leak" out of the comprehension scope into the surrounding (explicit, using 'def' or 'lambda') function's scope. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Tue Jul 17 11:59:02 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 17 Jul 2018 10:59:02 -0500 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: <20180717102727.19956040@fsol> References: <20180717102727.19956040@fsol> Message-ID: [Tim]> There is no intention to support GIL-free access to any Python objects. So > > that's the contract: "All warranties are null & void if you do just > about > > anything while not holding the GIL". > [Antoine] > Actually, accessing a Py_buffer that's been obtained in the regular way (e.g. with PyObject_GetBuffer) is safe even without the GIL. > Same as the docs, I use "Python object" to mean a pointer to PyObject. In that sense, a Py_buffer is no more a "Python object" than, e.g,, is a Py_ssize_t. If someone wants to muck with the `obj` member of a Py_buffer struct, _then_ they're back in "Python object" territory (`obj` is a PyObject*) and so are on their own if they don't obtain the GIL. Likewise all bets are off if they don't hold the GIL when calling PyObject_GetBuffer() to begin with, or PyBuffer_Release() when they're done. If they want to muck with the `buf` member without the GIL, then if they care about races among multiple threads mucking with `buf`, they'll have to supply their own mutual exclusion mechanism. So your "safe" comes with footnotes too, even though Py_buffer isn't itself a PyObject*. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Jul 17 12:09:20 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 17 Jul 2018 09:09:20 -0700 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: On Tue, Jul 17, 2018 at 4:34 AM, Victor Stinner wrote: > > IMHO you need a different approach to implement optimizations. For > example, use your objects which don't rely on the GIL to be > consistent. Sadly, I have no experience with that and so cannot > provide any example. > I DO NOT understand the technical details, but IIUC, numpy releases the GIL while doing pure numpy operations. (and I have experimented with it, and it does seem to be the case -- that is, you can get enhanced performance with computationally heavy numpy code an multiple threads) So maybe some lessons learned there. The key differences may be that numpy arrays are full-fledged python objects, but they are not (generally) containers of python objects -- that is, the actual values are i memory accessed via a C pointer. Whereas a PyList holds pointers to Python objects, so even if are confident in the stability of the list itself, the objects in the list may not be stable. For example, os.stat() writes into a memory block allocated > by Python. But it's a raw memory block, it's much simpler than a > Python object like a tuple. exactly. -- so is a numpy .data element -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Jul 17 12:12:10 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 17 Jul 2018 18:12:10 +0200 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: <20180717102727.19956040@fsol> Message-ID: <20180717181210.1e6e0c89@fsol> On Tue, 17 Jul 2018 10:59:02 -0500 Tim Peters wrote: > > Same as the docs, I use "Python object" to mean a pointer to PyObject. In > that sense, a Py_buffer is no more a "Python object" than, e.g,, is a > Py_ssize_t. Yes, and the payload of an int object is not a "Python object". The OP mentioned PyInt_AS_LONG as an example, so clearly the question was broader than you're painting it to be. > If they want to muck with the `buf` member without the GIL, then if they > care about races among multiple threads mucking with `buf`, they'll have to > supply their own mutual exclusion mechanism. Define "muck with". As long as you're reading the memory area(s) pointed to by the Py_buffer object, you're fine. If you plan to write to that memory, obviously you'll need to make sure the various threads don't overwrite each other, i.e. distribute the work carefully. But that's a normal provision for multi-threaded algorithms, not a Python-specific restriction. I'm assuming someone asking a question about multi-threaded access to CPython objects already knows all this :-) Regards Antoine. From chris.barker at noaa.gov Tue Jul 17 12:17:53 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 17 Jul 2018 09:17:53 -0700 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: > > I don't see why. As Chris said, side effects in asserts are nothing new > and > Indeed -- this new feature makes it easier to affect the local scope in all sorts of new places. It was decided that the additional complexity is worth it to make the language more expressive, and it was also decided not to try to "lock down" this new feature to only "appropriate" places. > this PEP is not the one to do something about it. Hmm --- not sure if it's PEP-worthy, but I'd sure love to see the community make a strong stance that: asserts are for tests, and only for tests Which doesn't necessarily mean that assert statements only exist in test code (though I personally follow that practice), but it does mean that assert statements, wherever they live, should not be counted on at run time, ever. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Jul 17 12:24:12 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Jul 2018 19:24:12 +0300 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: 17.07.18 18:48, Guido van Rossum ????: > On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka > wrote: > Should not the assert statement introduce an implicit lexical scope > for preventing leaking variables? > > I don't see why. As Chris said, side effects in asserts are nothing new > and this PEP is not the one to do something about it. This side effect is new. No other expressions that can be used in asserts leaked local variables before. The only exception is list comprehensions in Python 2, and this was fixed in Python 3. We can't make the assignment expression itself creating its own scope, because this will invalidate its purpose. But the problem with assert ccould be solved by making assert creating a new lexical scope. assert expr, msg could be translated to if __debug__ and not (lambda: expr)(): raise AssertionError(msg) instead of if __debug__ and not expr: raise AssertionError(msg) From tim.peters at gmail.com Tue Jul 17 12:25:05 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 17 Jul 2018 11:25:05 -0500 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: [Serhiy Storchaka] > Recently Barry shown an example: > > assert len(subdirs := list(path.iterdir())) == 0, subdirs > > It looks awful to me. It looks even worse than using asserts for > validating the user input. The assert has a side effect, and it depends > on the interpreter option (-O). Even if subdirs is not used outside of > the assert *now*, it is easy to introduce an error later, and it is hard > to notice it if tests are not ran with the -O option regularly. > Does PEP 572 encourages writing such code, discourages this, or > completely forbids? > The body of the PEP specifies semantics. My Appendix A gives some _opinions_ about "good" and "bad" uses, which boil down to "if it's not obviously at least a little win, don't use it". I can't really guess whether the above is an obvious win or not without context. It is a win (to my eyes) if the code it replaced was if __debug__: subdirs = list(path.iterdir()) assert len(subdirs) == 0, subdirs in which case the semantics are the same in either spelling, with or without -O, but the spelling at the top is less annoying ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Tue Jul 17 12:32:55 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Tue, 17 Jul 2018 19:32:55 +0300 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: 17.07.18 19:25, Tim Peters ????: > It is a win (to my eyes) if the code it replaced was > > ? ? if __debug__: > ? ? ? ? subdirs = list(path.iterdir()) > ? ? ? ? assert len(subdirs) == 0, subdirs > > in which case the semantics are the same in either spelling, with or > without -O, but the spelling at the top is less annoying ;-) I completely agree. From rosuav at gmail.com Tue Jul 17 12:37:42 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 18 Jul 2018 02:37:42 +1000 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: On Wed, Jul 18, 2018 at 2:24 AM, Serhiy Storchaka wrote: > 17.07.18 18:48, Guido van Rossum ????: >> >> On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka > > wrote: >> Should not the assert statement introduce an implicit lexical scope >> for preventing leaking variables? >> >> I don't see why. As Chris said, side effects in asserts are nothing new >> and this PEP is not the one to do something about it. > > > This side effect is new. No other expressions that can be used in asserts > leaked local variables before. The only exception is list comprehensions in > Python 2, and this was fixed in Python 3. There are plenty of expressions that have side effects, though. Creating local variables is just one type of side effect. If assertions are being misused, that is the fault of people writing bad assertions, not of assignment expressions, dict.setdefault(), subprocess.call(), or any other operation. Point of interest: You're basically asking for statement-local assignment, which was one of PEP 572's earliest forms. It received even more backlash than the current version did. The wheel turns and the same spoke comes up... ChrisA From tim.peters at gmail.com Tue Jul 17 12:45:40 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 17 Jul 2018 11:45:40 -0500 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: <20180717181210.1e6e0c89@fsol> References: <20180717102727.19956040@fsol> <20180717181210.1e6e0c89@fsol> Message-ID: [Tim] > > Same as the docs, I use "Python object" to mean a pointer to PyObject. > In > that sense, a Py_buffer is no more a "Python object" than, e.g,, is a > > Py_ssize_t. > [Antoine > Yes, and the payload of an int object is not a "Python object". > The OP mentioned PyInt_AS_LONG as an example, so clearly the question > was broader than you're painting it to be. > Ah, but I don't know that. The docs carve out no exception for the PyInt_AS_LONG API function, and since that doesn't appear to exist in Python 3 anymore (the only Python source I have on my box right now) I couldn't stare at the implementation to guess. Staring at implementations is the _only_ way to guess. > If they want to muck with the `buf` member without the GIL, then if they > > care about races among multiple threads mucking with `buf`, they'll have > to > > supply their own mutual exclusion mechanism. > > Define "muck with". Precisely my point: _your_ unqualified "safe" is missing all the qualifications needed to spell out when it's actually safe. "safe" is a subtler question than you're painting it to be ;-) As long as you're reading the memory area(s) pointed to by the Py_buffer object, you're fine. If you plan to write > to that memory, obviously you'll need to make sure the various threads > don't overwrite each other, i.e. distribute the work carefully. > > But that's a normal provision for multi-threaded algorithms, not a > Python-specific restriction. I'm assuming someone asking a question > about multi-threaded access to CPython objects already knows all > this :-) While I don't assume that. Looking it up, if they stared at #define PyInt_AS_LONG (op) (((PyIntObject *)(op))->ob_ival) in Python 2 and had to _ask_ whether it was safe to invoke without the GIL, then a more plausible assumption is they're going more by poke-&-hope than by reasoning. -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjudin+python at iki.fi Tue Jul 17 12:44:57 2018 From: jjudin+python at iki.fi (Jussi Judin) Date: Tue, 17 Jul 2018 19:44:57 +0300 Subject: [Python-Dev] Fuzzing the Python standard library Message-ID: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> Hi, I have been fuzzing[1] various parts of Python standard library for Python 3.7 with python-afl[2] to find out internal implementation issues that exist in the library. What I have been looking for are mainly following: * Exceptions that are something else than the documented ones. These usually indicate an internal implementation issue. For example one would not expect an UnicodeDecodeError from netrc.netrc() function when the documentation[3] promises netrc.NetrcParseError and there is no way to pass properly sanitized file object to the netrc.netrc(). * Differences between values returned by C and Python versions of some functions. quopri module may have these. * Unexpected performance and memory allocation issues. These can be somewhat controversial to fix, if at all, but at least in some cases from end-user perspective it can be really nasty if for example fractions.Fraction("1.64E6646466664") results in hundreds of megabytes of memory allocated and takes very long to calculate. I gave up waiting for that function call to finish after 5 minutes. As this is going to result in a decent amount of bug reports (currently I only filed one[4], although that audio processing area has much more issues to file), I would like to ask your opinion on filing these bug reports. Should I report all issues regarding some specific module in one bug report, or try to further split them into more fine grained reports that may be related? These different types of errors are specifically noticeable in zipfile module that includes a lot of different exception and behavioral types on invalid data . And in case of sndhdr module, there are multiple modules with issues (aifc, sunau, wave) that then show up also in sndhdr when they are used. Or are some of you willing to go through the crashes that pop up and help with the report filing? The code and more verbose description for this is available from . It works by default on some GNU/Linux systems only (I use Debian testing), as it relies on /dev/shm/ being available and uses shell scripts as wrappers that rely on various tools that may not be installed on all systems by default. As a bonus, as this uses coverage based fuzzing, it also opens up the possibility of automatically creating a regression test suite for each of the fuzzed modules to ensure that the existing functionality (input files under /corpus/ directory) does not suddenly result in additional exceptions and that it is more easy to test potential bug fixes (crash inducing files under /crashes/ directory). As a downside, this uses two quite specific tools (afl, python-afl) that have further dependencies (Cython) inside them, I doubt the viability of integrating this type of testing as part of normal Python verification process. As a difference to libFuzzer based fuzzing that is already integrated in Python[5], this instruments the actual (and only the) Python code and not the actions that the interpreter does in the background. So this should result in better fuzzer coverage for Python code that is used with the downside that when C functions are called, they are complete black boxes to the fuzzer. I have mainly run these fuzzer instances at most for several hours per module with 4 instances and stopped running no-issue modules after there have been no new coverage discovered after more than 10 minutes. Also I have not really created high quality initial input files, so I wouldn't be surprised if there are more issues lurking around that could be found with throwing more CPU and higher quality fuzzers at the problem. [1]: https://en.wikipedia.org/wiki/Fuzzing [2]: https://github.com/jwilk/python-afl [3]: https://docs.python.org/3/library/netrc.html [4]: https://bugs.python.org/issue34088 [5]: https://github.com/python/cpython/tree/3.7/Modules/_xxtestfuzz -- Jussi Judin https://jjudin.iki.fi/ From radim at rare-technologies.com Tue Jul 17 04:40:19 2018 From: radim at rare-technologies.com (=?UTF-8?B?UmFkaW0gxZhlaMWvxZllaw==?=) Date: Tue, 17 Jul 2018 10:40:19 +0200 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: Thanks Tim. That's one unambiguous answer. I did consider posting to python-list, but this seemed somewhat python-devish. Any appetite for documenting which foundational functions are const-safe in the sense I described? Perhaps we could help. What kind of analysis, demand or momentum is needed for the Python dev team to consider introducing such a contract? To be honest, I did do some CPython source code staring already. And at least for the functions we care about, it wasn't all that painful (PyDict_GetItem being the trickiest). Doing this wholesale might be a superhuman task, but I'm thinking a practical subset could be relatively straightforward while still useful, 80/20. On Tue, Jul 17, 2018 at 7:15 AM, Tim Peters wrote: > [Radim ?eh??ek ] > >> one of our Python projects calls for pretty heavy, low-level >> optimizations. > > >> We went down the rabbit hole and determined that having access to >> PyList_GET_ITEM(list), PyInt_AS_LONG(int) and PyDict_GetItem(dict, unicode) >> on Python objects **outside of GIL** might be a good-enough solution. The >> Python objects in question are guaranteed to live and not be mutated >> externally in any way. They're "frozen" and read-only. >> >> Under what conditions is it OK to call these 3 functions on such objects? >> > > From the "initialization, Finalization, and Threads" section of the > Python/C API Reference Manual: > > Therefore, the rule exists that only the thread that has acquired the GIL >> may operate on Python objects or call Python/C API functions. > > > Under protection of the GIL is the only documented - or intended - way to > do anything with Python objects, or to call virtually any Python/C API > function. > > > >> More generally, what is the CPython 2.7/3.5 contract regarding (lack of) >> object mutation, and the need for reference counting and synchronization >> via GIL? >> > > There is no intention to support GIL-free access to any Python objects. > So that's the contract: "All warranties are null & void if you do just > about anything while not holding the GIL". > > >> Which C API functions are safe to call on "const" objects? >> > > None. If you find some combinations of CPython versions, functions, and > objects, that happen to work, that's fine, but there's no guarantee they'll > continue to work, not even across maintenance releases. Although they'll > _probably_ continue to work. > > Obviously releasing GIL and then calling C API is hacky, but from initial >> experiments, it seems to work (see https://stackoverflow.com/ >> questions/51351609/can-i-const-access-cpython-objects-without-gil). But >> I'm wondering if there's a more formal contract around this behaviour. >> > > Nope! You're on your own here. > > Which doesn't mean you can't do it. Just that if things blow up, you're > still on your own - since you're far outside what the documentation says is > required (which I suspect you knew before you asked ;-) ), the project > won't even accept a bug report. > > If you want more help trying to guess which functions _may_ work outside > the law, that's fine too, but "python-dev" (this mailing list) is for > development _of_ Python itself, not for questions about _using_ Python. > The more general python-list would be more appropriate. But, since you're > trying to use the C API in unsupported ways it wasn't intended to be used, > you'll likely have a hard time finding people with significant experience > doing the same thing. Since it's not at all an intended use, there are no > "general principles" at work to ease the pain of staring at the CPython > source code to try to guess what might go wrong across all code paths. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jul 17 12:54:36 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Jul 2018 09:54:36 -0700 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: On Tue, Jul 17, 2018 at 9:24 AM, Serhiy Storchaka wrote: > 17.07.18 18:48, Guido van Rossum ????: > >> On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka > > wrote: >> Should not the assert statement introduce an implicit lexical scope >> for preventing leaking variables? >> >> I don't see why. As Chris said, side effects in asserts are nothing new >> and this PEP is not the one to do something about it. >> > > This side effect is new. No other expressions that can be used in asserts > leaked local variables before. The only exception is list comprehensions in > Python 2, and this was fixed in Python 3. > > We can't make the assignment expression itself creating its own scope, > because this will invalidate its purpose. But the problem with assert > ccould be solved by making assert creating a new lexical scope. > > assert expr, msg > > could be translated to > > if __debug__ and not (lambda: expr)(): > raise AssertionError(msg) > > instead of > > if __debug__ and not expr: > raise AssertionError(msg) > I don't believe this "problem" needs to be solved. It seems to be you are seeking an excuse to reopen the acrimonious PEP 572 discussion. Please stop. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From srittau at rittau.biz Tue Jul 17 12:55:41 2018 From: srittau at rittau.biz (Sebastian Rittau) Date: Tue, 17 Jul 2018 18:55:41 +0200 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: Message-ID: <0a098adf-bf9e-838b-ce3f-958acdc7ecae@rittau.biz> On 17.07.2018 17:05, Guido van Rossum wrote: > This is a good point. I presume specifying this unambiguously would be > a huge amount of work, and it would mostly codify mypy's current > behavior. I don't think that's within the scope of PEP 484, but it > could well be done as a separate PEP (perhaps an informational one?). > I hope you understand that I am not volunteering. An informational PEP sounds about right to me. Such a PEP could also include style recommendations like those from typeshed's CONTRIBUTING file (https://github.com/python/typeshed/blob/master/CONTRIBUTING.md). I guess I just volunteered to help with such a PEP, although I feel that someone from mypy's core team should take the lead on that. And if I understood this thread correctly, the pytype team is also willing to help out? ?- Sebastian From damian.peter.shaw at gmail.com Tue Jul 17 13:26:42 2018 From: damian.peter.shaw at gmail.com (Damian Shaw) Date: Tue, 17 Jul 2018 13:26:42 -0400 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> Message-ID: I'm not a core Python Dev, but quick question, why would you expect " fractions.Fraction("1.64E6646466664")" not to take 100s of megabytes and hours to run? Simply evaluating: 164 * 10**664646666 will take hundreds of megabytes by definition. Regards Damian On Tue, Jul 17, 2018, 12:54 Jussi Judin wrote: > Hi, > > I have been fuzzing[1] various parts of Python standard library for Python > 3.7 with python-afl[2] to find out internal implementation issues that > exist in the library. What I have been looking for are mainly following: > > * Exceptions that are something else than the documented ones. These > usually indicate an internal implementation issue. For example one would > not expect an UnicodeDecodeError from netrc.netrc() function when the > documentation[3] promises netrc.NetrcParseError and there is no way to pass > properly sanitized file object to the netrc.netrc(). > * Differences between values returned by C and Python versions of some > functions. quopri module may have these. > * Unexpected performance and memory allocation issues. These can be > somewhat controversial to fix, if at all, but at least in some cases from > end-user perspective it can be really nasty if for example > fractions.Fraction("1.64E6646466664") results in hundreds of megabytes of > memory allocated and takes very long to calculate. I gave up waiting for > that function call to finish after 5 minutes. > > As this is going to result in a decent amount of bug reports (currently I > only filed one[4], although that audio processing area has much more issues > to file), I would like to ask your opinion on filing these bug reports. > Should I report all issues regarding some specific module in one bug > report, or try to further split them into more fine grained reports that > may be related? These different types of errors are specifically noticeable > in zipfile module that includes a lot of different exception and behavioral > types on invalid data < > https://github.com/Barro/python-stdlib-fuzzers/tree/master/zipfile/crashes> > . And in case of sndhdr module, there are multiple modules with issues > (aifc, sunau, wave) that then show up also in sndhdr when they are used. Or > are some of you willing to go through the crashes that pop up and help with > the report filing? > > The code and more verbose description for this is available from < > https://github.com/Barro/python-stdlib-fuzzers>. It works by default on > some GNU/Linux systems only (I use Debian testing), as it relies on > /dev/shm/ being available and uses shell scripts as wrappers that rely on > various tools that may not be installed on all systems by default. > > As a bonus, as this uses coverage based fuzzing, it also opens up the > possibility of automatically creating a regression test suite for each of > the fuzzed modules to ensure that the existing functionality (input files > under /corpus/ directory) does not suddenly result in > additional exceptions and that it is more easy to test potential bug fixes > (crash inducing files under /crashes/ directory). > > As a downside, this uses two quite specific tools (afl, python-afl) that > have further dependencies (Cython) inside them, I doubt the viability of > integrating this type of testing as part of normal Python verification > process. As a difference to libFuzzer based fuzzing that is already > integrated in Python[5], this instruments the actual (and only the) Python > code and not the actions that the interpreter does in the background. So > this should result in better fuzzer coverage for Python code that is used > with the downside that when C functions are called, they are complete black > boxes to the fuzzer. > > I have mainly run these fuzzer instances at most for several hours per > module with 4 instances and stopped running no-issue modules after there > have been no new coverage discovered after more than 10 minutes. Also I > have not really created high quality initial input files, so I wouldn't be > surprised if there are more issues lurking around that could be found with > throwing more CPU and higher quality fuzzers at the problem. > > [1]: https://en.wikipedia.org/wiki/Fuzzing > [2]: https://github.com/jwilk/python-afl > [3]: https://docs.python.org/3/library/netrc.html > [4]: https://bugs.python.org/issue34088 > [5]: https://github.com/python/cpython/tree/3.7/Modules/_xxtestfuzz > > -- > Jussi Judin > https://jjudin.iki.fi/ > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/damian.peter.shaw%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jelle.zijlstra at gmail.com Tue Jul 17 13:40:04 2018 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Tue, 17 Jul 2018 10:40:04 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: <0a098adf-bf9e-838b-ce3f-958acdc7ecae@rittau.biz> References: <0a098adf-bf9e-838b-ce3f-958acdc7ecae@rittau.biz> Message-ID: 2018-07-17 9:55 GMT-07:00 Sebastian Rittau : > On 17.07.2018 17:05, Guido van Rossum wrote: > >> This is a good point. I presume specifying this unambiguously would be a >> huge amount of work, and it would mostly codify mypy's current behavior. I >> don't think that's within the scope of PEP 484, but it could well be done >> as a separate PEP (perhaps an informational one?). I hope you understand >> that I am not volunteering. >> > An informational PEP sounds about right to me. Such a PEP could also > include style recommendations like those from typeshed's CONTRIBUTING file ( > https://github.com/python/typeshed/blob/master/CONTRIBUTING.md). > > I guess I just volunteered to help with such a PEP, although I feel that > someone from mypy's core team should take the lead on that. And if I > understood this thread correctly, the pytype team is also willing to help > out? I can also help out. > > > - Sebastian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/jelle. > zijlstra%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Jul 17 13:41:08 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 18 Jul 2018 03:41:08 +1000 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: <20180717174107.GR7318@ando.pearwood.info> On Tue, Jul 17, 2018 at 07:24:12PM +0300, Serhiy Storchaka wrote: > 17.07.18 18:48, Guido van Rossum ????: > >On Tue, Jul 17, 2018 at 8:28 AM, Serhiy Storchaka >> wrote: > > Should not the assert statement introduce an implicit lexical scope > > for preventing leaking variables? > > > >I don't see why. As Chris said, side effects in asserts are nothing new > >and this PEP is not the one to do something about it. > > This side effect is new. No other expressions that can be used in > asserts leaked local variables before. assert vars().__setitem__('foo', 42) or foo Not that anyone would write such a thing (and it probably won't work inside a function in CPython), but it is possible. Besides, there is a legitimate use for assignment expressions in assertions: assert (spam := something) > 2, 'not enough spam' assert sequence[foo] == 999, 'sequence item isn't 999' Sometimes you need two assertions. > We can't make the assignment expression itself creating its own scope, > because this will invalidate its purpose. But the problem with assert > ccould be solved by making assert creating a new lexical scope. Assertions could be used to perform imports too: assert __import__('math') but people don't do that either. Let's not try fixing "problems" that won't exist. Most people have more sense than to do that. And for those who don't and misuse assertions, well, they've been misusing them for two decades and the sky hasn't fallen. > assert expr, msg > > could be translated to > > if __debug__ and not (lambda: expr)(): > raise AssertionError(msg) YAGNI. I do a lot of work in the REPL where I'm not afraid to bend the rules and use hacks I'd never use in long-lasting code. I'd never use assert for testing input in real code, but I do it in the REPL all the time. If I choose to abuse assertions in the REPL, you ought to trust me to do this responsibly. I'm a consenting adult. If I want to run with scissors, point a loaded gun at my foot, put all my eggs in one basket, or use assignment expressions in an assertion, I don't want the interpreter telling me I can't. -- Steve From acataldo at google.com Tue Jul 17 13:42:27 2018 From: acataldo at google.com (Adam Cataldo) Date: Tue, 17 Jul 2018 10:42:27 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: <0a098adf-bf9e-838b-ce3f-958acdc7ecae@rittau.biz> References: <0a098adf-bf9e-838b-ce3f-958acdc7ecae@rittau.biz> Message-ID: Hi Sebastian, Of course, we'd be happy to work with you on this! We just need to figure out which of us will drive this on our end (most likely Rebecca or Teddy). I'll huddle with the team and get back to you with an answer on who later today. On Tue, Jul 17, 2018 at 9:58 AM Sebastian Rittau wrote: > On 17.07.2018 17:05, Guido van Rossum wrote: > > This is a good point. I presume specifying this unambiguously would be > > a huge amount of work, and it would mostly codify mypy's current > > behavior. I don't think that's within the scope of PEP 484, but it > > could well be done as a separate PEP (perhaps an informational one?). > > I hope you understand that I am not volunteering. > An informational PEP sounds about right to me. Such a PEP could also > include style recommendations like those from typeshed's CONTRIBUTING > file (https://github.com/python/typeshed/blob/master/CONTRIBUTING.md). > > I guess I just volunteered to help with such a PEP, although I feel that > someone from mypy's core team should take the lead on that. And if I > understood this thread correctly, the pytype team is also willing to > help out? > > - Sebastian > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/acataldo%40google.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From acataldo at google.com Tue Jul 17 13:45:10 2018 From: acataldo at google.com (Adam Cataldo) Date: Tue, 17 Jul 2018 10:45:10 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: <0a098adf-bf9e-838b-ce3f-958acdc7ecae@rittau.biz> Message-ID: Great Jelle! We look forward to working with you to. On Tue, Jul 17, 2018 at 10:41 AM Jelle Zijlstra wrote: > > > 2018-07-17 9:55 GMT-07:00 Sebastian Rittau : > >> On 17.07.2018 17:05, Guido van Rossum wrote: >> >>> This is a good point. I presume specifying this unambiguously would be a >>> huge amount of work, and it would mostly codify mypy's current behavior. I >>> don't think that's within the scope of PEP 484, but it could well be done >>> as a separate PEP (perhaps an informational one?). I hope you understand >>> that I am not volunteering. >>> >> An informational PEP sounds about right to me. Such a PEP could also >> include style recommendations like those from typeshed's CONTRIBUTING file ( >> https://github.com/python/typeshed/blob/master/CONTRIBUTING.md). >> >> I guess I just volunteered to help with such a PEP, although I feel that >> someone from mypy's core team should take the lead on that. And if I >> understood this thread correctly, the pytype team is also willing to help >> out? > > I can also help out. > >> >> >> - Sebastian >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/jelle.zijlstra%40gmail.com >> > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/acataldo%40google.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Jul 17 14:24:43 2018 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Jul 2018 11:24:43 -0700 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: Message-ID: <951553AC-F713-49EE-AA4B-72FCC68CFC56@python.org> On Jul 17, 2018, at 07:59, Guido van Rossum wrote: > > Personally I like Barry's example just fine -- assuming `subdirs` is not used later, this feels like a good use case. Thanks! I thought it was cute. It was just something that occurred to me as I was reviewing some existing code. The intent wasn?t to use `subdirs` outside of the assert statement, but I?m warm to it because it means I don?t have to do wasted work outside of the assert statement, or repeat myself in the assert message part. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From tim.peters at gmail.com Tue Jul 17 14:34:55 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 17 Jul 2018 13:34:55 -0500 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: <951553AC-F713-49EE-AA4B-72FCC68CFC56@python.org> References: <951553AC-F713-49EE-AA4B-72FCC68CFC56@python.org> Message-ID: [Barry Warsaw] > Thanks! I thought it was cute. It was just something that occurred to me > as I was reviewing some existing code. The intent wasn?t to use `subdirs` > outside of the assert statement, but I?m warm to it because it means I > don?t have to do wasted work outside of the assert statement, or repeat > myself in the assert message part. > Because the latter ("repeat myself") is probably more tempting, I'll just note that the "laziness" of using an assignment expression instead may well have nudged you toward writing _better_ code too. assert len(subdirs := list(path.iterdir())) == 0, subdirs Assuming the result of list(path.iterdir()) can change over time (seems very likely), assert len(list(path.iterdir())) == 0, list(path.iterdir()) _could_ end up both triggering and displaying an empty list in the exception detail. The assignment-expression version cannot. -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at python.org Tue Jul 17 14:48:54 2018 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Jul 2018 11:48:54 -0700 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: <951553AC-F713-49EE-AA4B-72FCC68CFC56@python.org> Message-ID: <2EA7CB28-8967-4602-9E4E-0B4B255A1C8E@python.org> On Jul 17, 2018, at 11:34, Tim Peters wrote: > Assuming the result of list(path.iterdir()) can change over time (seems very likely), > > assert len(list(path.iterdir())) == 0, list(path.iterdir()) > > _could_ end up both triggering and displaying an empty list in the exception detail. The assignment-expression version cannot. Tim, you?re dangerously tempting me into +1 territory. :) -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From python at mrabarnett.plus.com Tue Jul 17 15:18:51 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 17 Jul 2018 20:18:51 +0100 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: References: <951553AC-F713-49EE-AA4B-72FCC68CFC56@python.org> Message-ID: <7b18d5bb-734b-0630-fa24-bf5887fee15f@mrabarnett.plus.com> On 2018-07-17 19:34, Tim Peters wrote: > > [Barry Warsaw] > > Thanks!? I thought it was cute.? It was just something that occurred > to me as I was reviewing some existing code.? The intent wasn?t to > use `subdirs` outside of the assert statement, but I?m warm to it > because it means I don?t have to do wasted work outside of the > assert statement, or repeat myself in the assert message part. > > > Because the latter ("repeat myself") is probably more tempting, I'll > just note that the "laziness" of using an assignment expression instead > may well have nudged you toward writing _better_ code too. > > ? ?assert len(subdirs := list(path.iterdir())) == 0, subdirs > > Assuming the result of list(path.iterdir()) can change over time (seems > very likely), > > ? ?assert len(list(path.iterdir())) == 0, list(path.iterdir()) > > _could_ end up both triggering and displaying an empty list in the > exception detail.? The assignment-expression version cannot. > Why use len(...) == 0 instead of not(...)? assert not(subdirs := list(path.iterdir())), subdirs From barry at python.org Tue Jul 17 15:58:42 2018 From: barry at python.org (Barry Warsaw) Date: Tue, 17 Jul 2018 12:58:42 -0700 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: <7b18d5bb-734b-0630-fa24-bf5887fee15f@mrabarnett.plus.com> References: <951553AC-F713-49EE-AA4B-72FCC68CFC56@python.org> <7b18d5bb-734b-0630-fa24-bf5887fee15f@mrabarnett.plus.com> Message-ID: <28F705E9-D428-4E06-8754-C39ABD01497F@python.org> On Jul 17, 2018, at 12:18, MRAB wrote: > Why use len(...) == 0 instead of not(...)? > > assert not(subdirs := list(path.iterdir())), subdirs Off-topic, but it?s a style thing. Some style guides (and my personal preference) require to be more explicit when checking for empty sequences. I?ll use a boolean test (e.g. `not` or false-iness) when the subject could be empty, None, or the empty string, and `len() == 0` when the subject is supposed to be a sequence. -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: Message signed with OpenPGP URL: From tim.peters at gmail.com Tue Jul 17 16:09:10 2018 From: tim.peters at gmail.com (Tim Peters) Date: Tue, 17 Jul 2018 15:09:10 -0500 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: [Radim ?eh??ek ] > Thanks Tim. That's one unambiguous answer. I'm glad that part came across ;-) > I did consider posting to python-list, but this seemed somewhat > python-devish. > I agree it's on the margin - I don't mind. Any appetite for documenting which foundational functions are const-safe in > the sense I described? Perhaps we could help. What kind of analysis, > demand or momentum is needed for the Python dev team to consider > introducing such a contract? > Most are volunteers scratching their own itches, and sometimes paid to scratch their employers' itches. This isn't my itch, so not me. Perhaps someone else - but from the messages so far it appears that most who replied suspect this is an "XY problem": https://en.wikipedia.org/wiki/XY_problem Python's been around for decades, and best I recall nobody has suggested anything really akin to what you're asking for here. There certainly are C extensions that want GIL-free access to (possibly massive amounts of) data visible from Python too. `numpy` is the prime example of that. As Antoine hinted, people working on those came up with the "buffer protocol" instead (documented in the Python/C API Reference Manual under the "Buffer Protocol" section). But nobody here knows anything about your app, so can't guess from here whether the buffer protocol is of any interest to you. The buffer protocol is aimed at uniform layouts of (possibly massive amounts of) native C-format data, not at GIL-free access to individual little Python objects. To be honest, I did do some CPython source code staring already. Which is the only way you can learn anything about whether breaking fundamental rules can get you in trouble. > And at least for the functions we care about, it wasn't all that painful > (PyDict_GetItem being the trickiest). Doing this wholesale might be a > superhuman task, but I'm thinking a practical subset could be relatively > straightforward while still useful, 80/20. > If you want to pursue this, it's probably necessary to first specify the set of API functions that need to change their requirements and promises, and to specify exactly what they're supposed to require and promise instead. Even then, you should expect resistance. It's been, historically, hard enough to avoid thread bugs in CPython's implementation even with the blanket "no access without the GIL, period" current requirement. Adding some number of "but these functions don't care about the GIL, in some circumstances they can't verify obtain, let alone enforce" exceptions would complicate an area that's already delicate. CPython's implementation not only requires that only one thread holds the GIL, it also works to verity that at runtime, and raises a fatal exception if it detects that it's even slightly confused about which thread currently holds the GIL. Adding weaker but unverifiable preconditions to some functions isn't attractive on the face of it. Note: the kind of people who find the GIL extremely intrusive tend instead to work on ways to eliminate the GIL entirely. They typically give up after a few years of intense pain ;-) -------------- next part -------------- An HTML attachment was scrubbed... URL: From phil.fremy at free.fr Tue Jul 17 15:37:35 2018 From: phil.fremy at free.fr (Philippe Fremy) Date: Tue, 17 Jul 2018 21:37:35 +0200 Subject: [Python-Dev] 2to3 for python annotations Message-ID: Hi, While contributing to pyannotate, I became familiar enough with 2to3 fixers to be able to convert Python 2 style annotations to Python 3. Is it something that would be interesting to put into python 2to3 ? If so I would propose a PR for this. Cheers, Philippe From jelle.zijlstra at gmail.com Tue Jul 17 16:17:17 2018 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Tue, 17 Jul 2018 13:17:17 -0700 Subject: [Python-Dev] 2to3 for python annotations In-Reply-To: References: Message-ID: 2018-07-17 12:37 GMT-07:00 Philippe Fremy : > Hi, > > While contributing to pyannotate, I became familiar enough with 2to3 > fixers to be able to convert Python 2 style annotations to Python 3. > > Is it something that would be interesting to put into python 2to3 ? If > so I would propose a PR for this. > > There already a tool that does this: https://github.com/ilevkivskyi/com2ann. I am not sure it would be appropriate for 2to3. Type comments still work in Python 3, and I feel like 2to3 should do the minimum necessary to get working code in Python 3. > Cheers, > > Philippe > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > jelle.zijlstra%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Jul 17 16:34:25 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 17 Jul 2018 13:34:25 -0700 Subject: [Python-Dev] 2to3 for python annotations In-Reply-To: References: Message-ID: On Tue, Jul 17, 2018 at 1:17 PM, Jelle Zijlstra wrote: > > > 2018-07-17 12:37 GMT-07:00 Philippe Fremy : > >> Hi, >> >> While contributing to pyannotate, I became familiar enough with 2to3 >> fixers to be able to convert Python 2 style annotations to Python 3. >> >> Is it something that would be interesting to put into python 2to3 ? If >> so I would propose a PR for this. >> >> There already a tool that does this: https://github.com/ > ilevkivskyi/com2ann. > IIRC that tool so far only converts variable declarations with `# type:` comments to PEP 526 style. Doing function signatures too seems on the TODO list: https://github.com/ilevkivskyi/com2ann/issues/3 > I am not sure it would be appropriate for 2to3. Type comments still work > in Python 3, and I feel like 2to3 should do the minimum necessary to get > working code in Python 3. > I think as an optional fixer it would be a fine contribution. Also I apologize for not yet reviewing https://github.com/dropbox/pyannotate/pull/74 (which I presume is yours?). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rechen at google.com Tue Jul 17 16:34:07 2018 From: rechen at google.com (Rebecca Chen) Date: Tue, 17 Jul 2018 13:34:07 -0700 Subject: [Python-Dev] Question about PEP 484 In-Reply-To: References: <0a098adf-bf9e-838b-ce3f-958acdc7ecae@rittau.biz> Message-ID: Hi Sebastian, Both Teddy (cc'd) and I would like to volunteer to help. We're excited about the prospect of an informational pyi PEP. Best, Rebecca On Tue, Jul 17, 2018 at 10:42 AM 'Adam Cataldo' via pytype < pytype at googlegroups.com> wrote: > Hi Sebastian, > > Of course, we'd be happy to work with you on this! We just need to figure > out which of us will drive this on our end (most likely Rebecca or Teddy). > I'll huddle with the team and get back to you with an answer on who later > today. > > > > On Tue, Jul 17, 2018 at 9:58 AM Sebastian Rittau > wrote: > >> On 17.07.2018 17:05, Guido van Rossum wrote: >> > This is a good point. I presume specifying this unambiguously would be >> > a huge amount of work, and it would mostly codify mypy's current >> > behavior. I don't think that's within the scope of PEP 484, but it >> > could well be done as a separate PEP (perhaps an informational one?). >> > I hope you understand that I am not volunteering. >> An informational PEP sounds about right to me. Such a PEP could also >> include style recommendations like those from typeshed's CONTRIBUTING >> file (https://github.com/python/typeshed/blob/master/CONTRIBUTING.md). >> >> I guess I just volunteered to help with such a PEP, although I feel that >> someone from mypy's core team should take the lead on that. And if I >> understood this thread correctly, the pytype team is also willing to >> help out? >> >> - Sebastian >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/acataldo%40google.com >> > -- > You received this message because you are subscribed to the Google Groups > "pytype" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to pytype+unsubscribe at googlegroups.com. > To post to this group, send email to pytype at googlegroups.com. > To view this discussion on the web visit > https://groups.google.com/d/msgid/pytype/CAKTwdc4Teidod9SnUy6Dp7BBBU21WTu%3DMQ7iPa%2BMvgLDshHcNg%40mail.gmail.com > > . > For more options, visit https://groups.google.com/d/optout. > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjudin+python at iki.fi Tue Jul 17 16:55:04 2018 From: jjudin+python at iki.fi (Jussi Judin) Date: Tue, 17 Jul 2018 23:55:04 +0300 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> Message-ID: <1531860904.733490.1444082664.1CAF20DA@webmail.messagingengine.com> Quick answer: undocumented billion laughs/exponential entity expansion type of an attack that is accessible through web through any library that uses fractions module to parse user input (that are actually available on Github). Could be mitigated by explicitly mentioning this in documentation, exposing a setting for engineering notation exponent limits, using non-naive way of storing numbers, or limiting the total size that a number representation can take by default to some limited, but large (for example 1 megabyte), value. More details should be discussed in a bug report that what is the preferred mitigation approach in this case. On Tue, Jul 17, 2018, at 20:26, Damian Shaw wrote: > I'm not a core Python Dev, but quick question, why would you expect " > fractions.Fraction("1.64E6646466664")" not to take 100s of megabytes and > hours to run? > > Simply evaluating: 164 * 10**664646666 will take hundreds of megabytes by > definition. > > Regards > Damian > > > On Tue, Jul 17, 2018, 12:54 Jussi Judin wrote: > > > Hi, > > > > I have been fuzzing[1] various parts of Python standard library for Python > > 3.7 with python-afl[2] to find out internal implementation issues that > > exist in the library. What I have been looking for are mainly following: > > > > * Exceptions that are something else than the documented ones. These > > usually indicate an internal implementation issue. For example one would > > not expect an UnicodeDecodeError from netrc.netrc() function when the > > documentation[3] promises netrc.NetrcParseError and there is no way to pass > > properly sanitized file object to the netrc.netrc(). > > * Differences between values returned by C and Python versions of some > > functions. quopri module may have these. > > * Unexpected performance and memory allocation issues. These can be > > somewhat controversial to fix, if at all, but at least in some cases from > > end-user perspective it can be really nasty if for example > > fractions.Fraction("1.64E6646466664") results in hundreds of megabytes of > > memory allocated and takes very long to calculate. I gave up waiting for > > that function call to finish after 5 minutes. > > > > As this is going to result in a decent amount of bug reports (currently I > > only filed one[4], although that audio processing area has much more issues > > to file), I would like to ask your opinion on filing these bug reports. > > Should I report all issues regarding some specific module in one bug > > report, or try to further split them into more fine grained reports that > > may be related? These different types of errors are specifically noticeable > > in zipfile module that includes a lot of different exception and behavioral > > types on invalid data < > > https://github.com/Barro/python-stdlib-fuzzers/tree/master/zipfile/crashes> > > . And in case of sndhdr module, there are multiple modules with issues > > (aifc, sunau, wave) that then show up also in sndhdr when they are used. Or > > are some of you willing to go through the crashes that pop up and help with > > the report filing? > > > > The code and more verbose description for this is available from < > > https://github.com/Barro/python-stdlib-fuzzers>. It works by default on > > some GNU/Linux systems only (I use Debian testing), as it relies on > > /dev/shm/ being available and uses shell scripts as wrappers that rely on > > various tools that may not be installed on all systems by default. > > > > As a bonus, as this uses coverage based fuzzing, it also opens up the > > possibility of automatically creating a regression test suite for each of > > the fuzzed modules to ensure that the existing functionality (input files > > under /corpus/ directory) does not suddenly result in > > additional exceptions and that it is more easy to test potential bug fixes > > (crash inducing files under /crashes/ directory). > > > > As a downside, this uses two quite specific tools (afl, python-afl) that > > have further dependencies (Cython) inside them, I doubt the viability of > > integrating this type of testing as part of normal Python verification > > process. As a difference to libFuzzer based fuzzing that is already > > integrated in Python[5], this instruments the actual (and only the) Python > > code and not the actions that the interpreter does in the background. So > > this should result in better fuzzer coverage for Python code that is used > > with the downside that when C functions are called, they are complete black > > boxes to the fuzzer. > > > > I have mainly run these fuzzer instances at most for several hours per > > module with 4 instances and stopped running no-issue modules after there > > have been no new coverage discovered after more than 10 minutes. Also I > > have not really created high quality initial input files, so I wouldn't be > > surprised if there are more issues lurking around that could be found with > > throwing more CPU and higher quality fuzzers at the problem. > > > > [1]: https://en.wikipedia.org/wiki/Fuzzing > > [2]: https://github.com/jwilk/python-afl > > [3]: https://docs.python.org/3/library/netrc.html > > [4]: https://bugs.python.org/issue34088 > > [5]: https://github.com/python/cpython/tree/3.7/Modules/_xxtestfuzz > > > > -- > > Jussi Judin > > https://jjudin.iki.fi/ > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: > > https://mail.python.org/mailman/options/python-dev/damian.peter.shaw%40gmail.com > > -- Jussi Judin https://jjudin.iki.fi/ From mike at selik.org Tue Jul 17 18:15:09 2018 From: mike at selik.org (Michael Selik) Date: Tue, 17 Jul 2018 18:15:09 -0400 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: <1531860904.733490.1444082664.1CAF20DA@webmail.messagingengine.com> References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> <1531860904.733490.1444082664.1CAF20DA@webmail.messagingengine.com> Message-ID: On Tue, Jul 17, 2018 at 4:57 PM Jussi Judin wrote: > Quick answer: undocumented billion laughs/exponential entity expansion > type of an attack that is accessible through web through any library that > uses fractions module to parse user input (that are actually available on > Github). > Are you suggesting a warning in the fractions documentation to mention that large numbers require large amounts of memory? -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Jul 17 18:39:56 2018 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 17 Jul 2018 15:39:56 -0700 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> Message-ID: On Tue, Jul 17, 2018 at 9:44 AM, Jussi Judin wrote: > * Exceptions that are something else than the documented ones. These usually indicate an internal implementation issue. For example one would not expect an UnicodeDecodeError from netrc.netrc() function when the documentation[3] promises netrc.NetrcParseError and there is no way to pass properly sanitized file object to the netrc.netrc(). My main advice would be, before mass-filing bugs make sure that you and the maintainers agree on what a bug is :-). For example, I can see the argument that invalid encodings in netrc should be reported as NetrcParseError, but there are also many APIs where it's normal to get something like a TypeError even if that's not a documented exception, and it's very annoying as a maintainer to suddenly get 20 bugs where you don't even agree that they're bugs and just have to wade through and close them all. Any assistance you can give with triaging and prioritizing the bugs is also super helpful, since volunteer maintainers aren't necessarily prepared for sudden influxes of issues. In general this sounds like cool work, and help improving Python's quality is always welcome. Just be careful that it's actually helpful :-). -n -- Nathaniel J. Smith -- https://vorpus.org From paul at ganssle.io Tue Jul 17 18:44:23 2018 From: paul at ganssle.io (Paul G) Date: Tue, 17 Jul 2018 18:44:23 -0400 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> <1531860904.733490.1444082664.1CAF20DA@webmail.messagingengine.com> Message-ID: In many languages numeric types can't hold arbitrarily large values, and I for one hadn't really previously recognized that if you read in a numeric value with an exponent that it would be represented *exactly* in memory (and thus one object with a very compact representation can take up huge amounts of memory). It's also not *inconceivable* that under the hood Python would represent fractions.Fraction("1.64E6646466664") "lazily" in some fashion so that it did not consume all the memory on disk. It seems to me that "Hey by the way the size of this thing is unbounded and because of exponents small strings can expand to huge objects" is a good tip. On 07/17/2018 06:15 PM, Michael Selik wrote: > On Tue, Jul 17, 2018 at 4:57 PM Jussi Judin > wrote: > > Quick answer: undocumented billion laughs/exponential entity expansion type of an attack that is accessible through web through any library that uses fractions module to parse user input (that are actually available on Github). > > > Are you suggesting a warning in the fractions documentation to mention that large numbers require large amounts of memory? > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/paul%40ganssle.io > -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 833 bytes Desc: OpenPGP digital signature URL: From steve at pearwood.info Tue Jul 17 19:33:24 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 18 Jul 2018 09:33:24 +1000 Subject: [Python-Dev] PEP 572 and assert In-Reply-To: <20180717174107.GR7318@ando.pearwood.info> References: <20180717174107.GR7318@ando.pearwood.info> Message-ID: <20180717233324.GS7318@ando.pearwood.info> On Wed, Jul 18, 2018 at 03:41:08AM +1000, Steven D'Aprano wrote: > Besides, there is a legitimate use for assignment expressions in > assertions: > > assert (spam := something) > 2, 'not enough spam' > assert sequence[foo] == 999, 'sequence item isn't 999' Ah, obviously the index in the second line ought to be spam, not foo. -- Steve From ronaldoussoren at mac.com Wed Jul 18 02:37:59 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Wed, 18 Jul 2018 07:37:59 +0100 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: Op 17 jul. 2018 om 09:40 heeft Radim ?eh??ek het volgende geschreven: > > > To be honest, I did do some CPython source code staring already. And at least for the functions we care about, it wasn't all that painful (PyDict_GetItem being the trickiest). Doing this wholesale might be a superhuman task, but I'm thinking a practical subset could be relatively straightforward while still useful, 80/20. This is would likely lead to a fairly vague document. Using PyDict_GetItem as an example, even if it might sometimes be possible to call without holding the GIL there are definitely use cases where it is not, for example when there are keys with a type implemented in python, or when the dict is modified in another thread. Ronald From steve at holdenweb.com Wed Jul 18 03:49:03 2018 From: steve at holdenweb.com (Steve Holden) Date: Wed, 18 Jul 2018 08:49:03 +0100 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> <1531860904.733490.1444082664.1CAF20DA@webmail.messagingengine.com> Message-ID: On Tue, Jul 17, 2018 at 11:44 PM, Paul G wrote: > In many languages numeric types can't hold arbitrarily large values, and I > for one hadn't really previously recognized that if you read in a numeric > value with an exponent that it would be represented *exactly* in memory > (and thus one object with a very compact representation can take up huge > amounts of memory). It's also not *inconceivable* that under the hood > Python would represent fractions.Fraction("1.64E6646466664") "lazily" in > some fashion so that it did not consume all the memory on disk. > > ?Sooner or later you are going to need the digits of the number to perform a computation. Exactly when would you propose the deferred evaluation should take place? There are already occasional inquiries about the effects of creation of such large numbers and their unexpected effects, so they aren't completely unknown. At the same time, this isn't exactly a mainstream "bug", as evidenced by the fact that such issues ? ?are relatively rare. ? > It seems to me that "Hey by the way the size of this thing is unbounded > and because of exponents small strings can expand to huge objects" is a > good tip. > > ?Not an unreasonable suggestion. Perhaps you could draft a documentation change - personally I'm not even sure where the best place for the warning would be. ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From radim at rare-technologies.com Wed Jul 18 06:18:06 2018 From: radim at rare-technologies.com (=?UTF-8?B?UmFkaW0gxZhlaMWvxZllaw==?=) Date: Wed, 18 Jul 2018 12:18:06 +0200 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: Thanks for your feedback everyone. Given the overwhelmingly negative response, we'll drop this line of investigation. If more people bring up the same request in the future (unlikely), feel free to reach out to us for some extra set of hands. Given the initial poking, I still think a "reasonable subset" might be "reasonably easy"; IMHO more a process/maintenance/ROI question than a strictly technical one. On Tue, Jul 17, 2018 at 10:09 PM, Tim Peters wrote: > Note: the kind of people who find the GIL extremely intrusive tend > instead to work on ways to eliminate the GIL entirely. They typically give > up after a few years of intense pain ;-) > If you mean "writing an alternative Python interpreter", that's not of any interest. If you mean "eliminating GIL from mission-critical parts of the code" -- we've done that many times, with good success and only moderate pain. The current "const" question was a probe about the cost of bringing the worlds a little closer. Radim -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Wed Jul 18 14:19:53 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 19 Jul 2018 04:19:53 +1000 Subject: [Python-Dev] Const access to CPython objects outside of GIL? In-Reply-To: References: Message-ID: On Wed, Jul 18, 2018 at 8:18 PM, Radim ?eh??ek wrote: > Thanks for your feedback everyone. Given the overwhelmingly negative > response, we'll drop this line of investigation. > > If more people bring up the same request in the future (unlikely), feel free > to reach out to us for some extra set of hands. Given the initial poking, I > still think a "reasonable subset" might be "reasonably easy"; IMHO more a > process/maintenance/ROI question than a strictly technical one. The trouble would be defining that "reasonable subset", which would end up having a very large number of words in it. For example, accessing a Python list after any sort of size change could crash the interpreter hard (as the buffer will have been reallocated). I'm fairly sure you can safely read from a tuple so long as you retain a ref to the tuple itself, though, so you may find that there are options there. Maybe, depending on your needs, the best solution might be to NOT access Python objects at all. Instead, have an API for changing info that is referenced outside of the GIL, and then the key info gets grabbed in a form that doesn't require Python. That would require some changes in the Python code (function calls rather than list manipulation), but would be 100% guaranteed safe. But you've probably already thought of that, so this is a case where that doesn't work :) ChrisA From brett at python.org Wed Jul 18 14:32:23 2018 From: brett at python.org (Brett Cannon) Date: Wed, 18 Jul 2018 11:32:23 -0700 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> Message-ID: On Tue, 17 Jul 2018 at 15:41 Nathaniel Smith wrote: > On Tue, Jul 17, 2018 at 9:44 AM, Jussi Judin wrote: > > * Exceptions that are something else than the documented ones. These > usually indicate an internal implementation issue. For example one would > not expect an UnicodeDecodeError from netrc.netrc() function when the > documentation[3] promises netrc.NetrcParseError and there is no way to pass > properly sanitized file object to the netrc.netrc(). > > My main advice would be, before mass-filing bugs make sure that you > and the maintainers agree on what a bug is :-). For example, I can see > the argument that invalid encodings in netrc should be reported as > NetrcParseError, but there are also many APIs where it's normal to get > something like a TypeError even if that's not a documented exception, > and it's very annoying as a maintainer to suddenly get 20 bugs where > you don't even agree that they're bugs and just have to wade through > and close them all. Any assistance you can give with triaging and > prioritizing the bugs is also super helpful, since volunteer > maintainers aren't necessarily prepared for sudden influxes of issues. > That was my initial reaction to that first bullet point as well. If the exception isn't at least explicitly raised then it shouldn't be considered a documentation problem, and even then I don't know if I would expect an explicit mentioning of ValueError if the docs say e.g. "only values within this range are expected" as that implicitly suggests ValueError will be used. > > In general this sounds like cool work, and help improving Python's > quality is always welcome. Just be careful that it's actually helpful > :-). > It's definitely a balancing act. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Wed Jul 18 15:11:40 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Wed, 18 Jul 2018 22:11:40 +0300 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> Message-ID: On 17.07.2018 19:44, Jussi Judin wrote: > Hi, > > I have been fuzzing[1] various parts of Python standard library for Python 3.7 with python-afl[2] to find out internal implementation issues that exist in the library. What I have been looking for are mainly following: > > * Exceptions that are something else than the documented ones. These usually indicate an internal implementation issue. For example one would not expect an UnicodeDecodeError from netrc.netrc() function when the documentation[3] promises netrc.NetrcParseError and there is no way to pass properly sanitized file object to the netrc.netrc(). > * Differences between values returned by C and Python versions of some functions. quopri module may have these. > * Unexpected performance and memory allocation issues. These can be somewhat controversial to fix, if at all, but at least in some cases from end-user perspective it can be really nasty if for example fractions.Fraction("1.64E6646466664") results in hundreds of megabytes of memory allocated and takes very long to calculate. I gave up waiting for that function call to finish after 5 minutes. > > As this is going to result in a decent amount of bug reports (currently I only filed one[4], although that audio processing area has much more issues to file), I would like to ask your opinion on filing these bug reports. Should I report all issues regarding some specific module in one bug report, or try to further split them into more fine grained reports that may be related? These different types of errors are specifically noticeable in zipfile module that includes a lot of different exception and behavioral types on invalid data . And in case of sndhdr module, there are multiple modules with issues (aifc, sunau, wave) that then show up also in sndhdr when they are used. Or are some of you willing to go through the crashes that pop up and help with the report filing? I'm not from the core team, so will recite best practices from my own experience. Bugs should be reported "one per root cause" aka 1bug report=1fix. It's permissible to report separately, especially if you're not sure if they are the same bug (then add a prominent link), but since this is a volunteer project, you really should be doing any diplicate checks _before_ reporting. Since you'll be checking existing tickets before reporting each new one anyway, that'll automatically include _your own_ previous tickets ;-) For ditto bugs in multiple places, it's better to err on the side of fewer tickets -- this will both be less work for everyone and give a more complete picture. If something proves to warrant a separate ticket, it can be split off later. > The code and more verbose description for this is available from . It works by default on some GNU/Linux systems only (I use Debian testing), as it relies on /dev/shm/ being available and uses shell scripts as wrappers that rely on various tools that may not be installed on all systems by default. > > As a bonus, as this uses coverage based fuzzing, it also opens up the possibility of automatically creating a regression test suite for each of the fuzzed modules to ensure that the existing functionality (input files under /corpus/ directory) does not suddenly result in additional exceptions and that it is more easy to test potential bug fixes (crash inducing files under /crashes/ directory). > > As a downside, this uses two quite specific tools (afl, python-afl) that have further dependencies (Cython) inside them, I doubt the viability of integrating this type of testing as part of normal Python verification process. As a difference to libFuzzer based fuzzing that is already integrated in Python[5], this instruments the actual (and only the) Python code and not the actions that the interpreter does in the background. So this should result in better fuzzer coverage for Python code that is used with the downside that when C functions are called, they are complete black boxes to the fuzzer. > > I have mainly run these fuzzer instances at most for several hours per module with 4 instances and stopped running no-issue modules after there have been no new coverage discovered after more than 10 minutes. Also I have not really created high quality initial input files, so I wouldn't be surprised if there are more issues lurking around that could be found with throwing more CPU and higher quality fuzzers at the problem. > > [1]: https://en.wikipedia.org/wiki/Fuzzing > [2]: https://github.com/jwilk/python-afl > [3]: https://docs.python.org/3/library/netrc.html > [4]: https://bugs.python.org/issue34088 > [5]: https://github.com/python/cpython/tree/3.7/Modules/_xxtestfuzz > -- Regards, Ivan From vstinner at redhat.com Wed Jul 18 16:54:46 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 18 Jul 2018 22:54:46 +0200 Subject: [Python-Dev] Status of Python CIs (buildbots, Travis CI, AppVeyor): july 2018 Message-ID: Hi, It seems like my latest status of Python CIs was already one year ago! https://mail.python.org/pipermail/python-dev/2017-June/148511.html Since last year, Zachary Ware (with the help of others, but I forgot names, sorry!) migrated our buildbot server from buildbot 0.8 (Python 2.7) to buildbot 0.9 (Python 3.4). The new buildbot version has a very different web UI: https://buildbot.python.org/all/#/ It took me time to get used to it, but now I prefer the new UI especially to see the result of a single build. The page loads faster and it's easier to access data. I also like the readable list of all builders: https://buildbot.python.org/all/#/builders The buildbot "warnings" step now contains test failures and test errors for a quick overview of bugs. Example: FAIL: test_threads (test.test_gdb.PyBtTests) Re-running failed tests in verbose mode Re-running test 'test_gdb' in verbose mode FAIL: test_threads (test.test_gdb.PyBtTests) I also modified libregrtest (our test runner: python3 -m test) to display a better tests summary at the end, especially when there is a least one failure. Truncated example: --- == Tests result: FAILURE then SUCCESS == 378 tests OK. 10 slowest tests: - test_multiprocessing_spawn: 1 min 57 sec - test_concurrent_futures: 1 min 36 sec - test_nntplib: 30 sec 275 ms - (...) 28 tests skipped: test_crypt (...) 1 re-run test: test_threading Total duration: 4 min 59 sec Tests result: FAILURE then SUCCESS --- "FAILURE then SUCCESS" means that at least one test failed, but then all re-run tests succeeded. "1 re-run test: test_threading" is the list of tests that failed previously. That's also a new feature. Last May, we worked hard to fix many random test failures on all CIs before Python 3.7 final release. Today, the number of tests which fail randomly is *very* low. Since the the beginning of the year, I fixed bugs in more than 35 test files. The most complex issues were in multiprocessing tests: the most common random failures should now be fixed. Many memory and reference leaks have been fixed. I also started to fix leaks of Windows handles: https://github.com/python/cpython/pull/7827 I added new keys to test.pythoninfo: Py_DEBUG, C compiler version, gdbm version, memory allocator, etc. The test.bisect tool has been optimized to be usable on test_asyncio, one of the test which has the most test cases and methods. I spent a lot of time to fix each test failure even when a test only failed once on one specific CI on a specific pull request. I increased many timeouts to make fragile tests more "reliable" (reduce the risk of failures on slow buildbots). Some timeouts are just too strict for no good reason. Python CIs are not perfect, but random failures should now be more rare. Mailing list for email notifications when a buildbot fails. That's my main source to detect regressions and tests which fail randomly: https://mail.python.org/mm3/mailman3/lists/buildbot-status.python.org/ Buildbot builders: http://buildbot.python.org/all/#/builders Travis CI build history: https://travis-ci.org/python/cpython/builds AppVeyor build history: https://ci.appveyor.com/project/python/cpython/history My notes on Python CIs: http://pythondev.readthedocs.io/ci.html Thanks Zachary Ware for maintaining our buildbot servers, thanks Pablo Galindo Salgado who helped me to triage buildbot failures (on the buildbot-status mailing list), thanks all other developers who helped me to fix random test failures and make our test suite more stable! Victor From barry at python.org Wed Jul 18 17:13:39 2018 From: barry at python.org (Barry Warsaw) Date: Wed, 18 Jul 2018 14:13:39 -0700 Subject: [Python-Dev] Status of Python CIs (buildbots, Travis CI, AppVeyor): july 2018 In-Reply-To: References: Message-ID: On Jul 18, 2018, at 13:54, Victor Stinner wrote: > Last May, we worked hard to fix many random test failures on all CIs > before Python 3.7 final release. Thank you thank you thank you Victor for work on keeping the buildbots happy! -Barry -------------- next part -------------- A non-text attachment was scrubbed... Name: signature.asc Type: application/pgp-signature Size: 195 bytes Desc: Message signed with OpenPGP URL: From ericsnowcurrently at gmail.com Wed Jul 18 17:20:13 2018 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Wed, 18 Jul 2018 15:20:13 -0600 Subject: [Python-Dev] Status of Python CIs (buildbots, Travis CI, AppVeyor): july 2018 In-Reply-To: References: Message-ID: On Wed, Jul 18, 2018 at 3:16 PM Barry Warsaw wrote: > On Jul 18, 2018, at 13:54, Victor Stinner wrote: > > Last May, we worked hard to fix many random test failures on all CIs > > before Python 3.7 final release. > > Thank you thank you thank you Victor for work on keeping the buildbots happy! Yes, thank you Victor (and friends). Your work on this makes a concrete difference. -eric From phil.fremy at free.fr Wed Jul 18 18:59:44 2018 From: phil.fremy at free.fr (Philippe Fremy) Date: Thu, 19 Jul 2018 00:59:44 +0200 Subject: [Python-Dev] 2to3 for python annotations In-Reply-To: References: Message-ID: Le 17/07/2018 ? 22:34, Guido van Rossum a ?crit?: > On Tue, Jul 17, 2018 at 1:17 PM, Jelle Zijlstra > > wrote: > > > > 2018-07-17 12:37 GMT-07:00 Philippe Fremy >: > > Hi, > > While contributing to pyannotate, I became familiar enough > with 2to3 > fixers to be able to convert Python 2 style annotations to > Python 3. > > Is it something that would be interesting to put into python > 2to3 ? If > so I would propose a PR for this. > > [...] > > > I think as an optional fixer it would be a fine contribution. I'll work on something then. Out of curiosity, is anybody else seeing an interest in this ? > > Also I apologize for not yet reviewing > https://github.com/dropbox/pyannotate/pull/74 (which I presume is yours?). This is mine indeed. You said that it would take time to review and I said that I would be patient, so ... I am patient, no worries. Cheers, Philippe -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Jul 19 08:50:46 2018 From: larry at hastings.org (Larry Hastings) Date: Thu, 19 Jul 2018 05:50:46 -0700 Subject: [Python-Dev] 3.4.9rc1 and 3.5.6rc1 slipping by one day to Thursday July 19 2018 Message-ID: I was working with Serhiy on fixing the documentation for some bytecodes in 3.5 (GH-8338) and time got away from me.? They'll both be out later today, Thursday July 19 2018. *yawn,* //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From larry at hastings.org Thu Jul 19 22:21:06 2018 From: larry at hastings.org (Larry Hastings) Date: Thu, 19 Jul 2018 19:21:06 -0700 Subject: [Python-Dev] [RELEASED] Python 3.4.9rc1 and Python 3.5.6rc1 are now available Message-ID: <3cd1fce0-6ce1-1812-d3b5-629d618b4315@hastings.org> On behalf of the Python development community, I'm pleased to announce the availability of Python 3.4.9rc1 and Python 3.5.6rc1. Both Python 3.4 and 3.5 are in "security fixes only" mode.? Both versions only accept security fixes, not conventional bug fixes, and both releases are source-only. You can find Python 3.4.9rc1 here: https://www.python.org/downloads/release/python-349rc1/ And you can find Python 3.5.6rc1 here: https://www.python.org/downloads/release/python-356rc1/ Python's entrenched bureaucracy soldiers on, //arry/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jul 20 12:09:57 2018 From: status at bugs.python.org (Python tracker) Date: Fri, 20 Jul 2018 18:09:57 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20180720160957.2BCD456231@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2018-07-13 - 2018-07-20) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6733 (+27) closed 39188 (+35) total 45921 (+62) Open issues with patches: 2681 Issues opened (46) ================== #25150: 3.5: Include/pyatomic.h is incompatible with OpenMP (compilati https://bugs.python.org/issue25150 reopened by vstinner #34068: traceback.clear_frames(): Objects/typeobject.c:3086: _PyType_L https://bugs.python.org/issue34068 reopened by vstinner #34110: cPickle may raise AttributeError when loading concurrently in https://bugs.python.org/issue34110 opened by sangongs #34111: python-config breaks when symlinked to another location https://bugs.python.org/issue34111 opened by tobik #34113: LLTRACE segv https://bugs.python.org/issue34113 opened by vandyswa #34115: code.InteractiveConsole.interact() closes stdin https://bugs.python.org/issue34115 opened by Yonatan Zunger #34117: Rename "generator expressions" to "generator comprehensions" https://bugs.python.org/issue34117 opened by brett.cannon #34118: Fix some class entries in 'Built-in Functions' https://bugs.python.org/issue34118 opened by terry.reedy #34120: IDLE Caret/Focus Lost https://bugs.python.org/issue34120 opened by vtudorache #34125: Profiling depends on whether **kwargs is given https://bugs.python.org/issue34125 opened by jdemeyer #34126: Profiling certain invalid calls crashes Python https://bugs.python.org/issue34126 opened by jdemeyer #34127: Gramatically incorrect error message for some descriptor calls https://bugs.python.org/issue34127 opened by ppperry #34128: Release GIL periodically in _pickle module https://bugs.python.org/issue34128 opened by Martin Bammer #34129: CGITB does not mangle variables names https://bugs.python.org/issue34129 opened by pjurkas #34131: test_threading: BarrierTests.test_default_timeout() failed on https://bugs.python.org/issue34131 opened by vstinner #34132: Obscure netrc parser "bug" https://bugs.python.org/issue34132 opened by skip.montanaro #34134: multiprocessing memory huge usage https://bugs.python.org/issue34134 opened by Windson Yang #34135: The results of time.tzname print broken. https://bugs.python.org/issue34135 opened by maxtortime #34136: Del on class __annotations__ regressed, failing test https://bugs.python.org/issue34136 opened by kayhayen #34137: Add Path.lexist() to pathlib https://bugs.python.org/issue34137 opened by lordmauve #34138: imaplib RFC 6855 issue https://bugs.python.org/issue34138 opened by Sam Varshavchik #34139: Remove stale unix datagram socket before binding https://bugs.python.org/issue34139 opened by qdawans #34142: Windows launcher version lookup flawed https://bugs.python.org/issue34142 opened by LarryZA #34144: venv activate.bat reset codepage fails on windows 10 https://bugs.python.org/issue34144 opened by LorenzMende #34145: uuid3 and uuid5 hard to use portably between Python 2 and 3 https://bugs.python.org/issue34145 opened by rubasov #34146: PyCData_Type.tp_hash doesn't use PyObject_HashNotImplemented https://bugs.python.org/issue34146 opened by bup #34148: Fatal error on SSL transport https://bugs.python.org/issue34148 opened by yjqiang #34149: Behavior of the min/max with key=None https://bugs.python.org/issue34149 opened by amper #34150: test_multiprocessing_spawn: Dangling processes leaked on AMD64 https://bugs.python.org/issue34150 opened by vstinner #34151: use malloc() for better performance of some list operations https://bugs.python.org/issue34151 opened by sir-sigurd #34152: performance of some list slice assignment margin cases can be https://bugs.python.org/issue34152 opened by sir-sigurd #34154: Tkinter __init__ documentations sometimes missing valid keywor https://bugs.python.org/issue34154 opened by Creation Elemental #34155: email.utils.parseaddr mistakenly parse an email https://bugs.python.org/issue34155 opened by Cyril Nicod??me #34156: Nail down and document the behavior of range expressions in RE https://bugs.python.org/issue34156 opened by zwol #34158: Documentation of datetime '%z' format code is odd https://bugs.python.org/issue34158 opened by Christophe Nanteuil #34159: asyncio basic event loop stuck with no tasks scheduled or read https://bugs.python.org/issue34159 opened by Sheng Zhong #34160: ElementTree not preserving attribute order https://bugs.python.org/issue34160 opened by rhettinger #34162: idlelib/NEWS.txt for 3.8.0 (and backports) https://bugs.python.org/issue34162 opened by terry.reedy #34163: Python latest release 2.7 shows SSL error https://bugs.python.org/issue34163 opened by Janibasha #34164: base64.b32decode() leads into UnboundLocalError and OverflowEr https://bugs.python.org/issue34164 opened by Barro #34165: uu.decode() raises binascii.Error instead of uu.Error on inval https://bugs.python.org/issue34165 opened by Barro #34166: Tools/msgfmt.py emits a DeprecationWarning under Python 3.7 https://bugs.python.org/issue34166 opened by lelit #34167: Standard library docs: prev/next skip right over 16.11 https://bugs.python.org/issue34167 opened by Eric Janson #34168: RAM consumption too high using concurrent.futures (Python 3.7 https://bugs.python.org/issue34168 opened by DemGiran #34169: itertools.repeat does not accept None as an explicit count https://bugs.python.org/issue34169 opened by chepner #34170: Py_Initialize(): computing path configuration must not have si https://bugs.python.org/issue34170 opened by vstinner Most recent 15 issues with no replies (15) ========================================== #34169: itertools.repeat does not accept None as an explicit count https://bugs.python.org/issue34169 #34167: Standard library docs: prev/next skip right over 16.11 https://bugs.python.org/issue34167 #34166: Tools/msgfmt.py emits a DeprecationWarning under Python 3.7 https://bugs.python.org/issue34166 #34165: uu.decode() raises binascii.Error instead of uu.Error on inval https://bugs.python.org/issue34165 #34158: Documentation of datetime '%z' format code is odd https://bugs.python.org/issue34158 #34146: PyCData_Type.tp_hash doesn't use PyObject_HashNotImplemented https://bugs.python.org/issue34146 #34145: uuid3 and uuid5 hard to use portably between Python 2 and 3 https://bugs.python.org/issue34145 #34144: venv activate.bat reset codepage fails on windows 10 https://bugs.python.org/issue34144 #34139: Remove stale unix datagram socket before binding https://bugs.python.org/issue34139 #34127: Gramatically incorrect error message for some descriptor calls https://bugs.python.org/issue34127 #34125: Profiling depends on whether **kwargs is given https://bugs.python.org/issue34125 #34120: IDLE Caret/Focus Lost https://bugs.python.org/issue34120 #34115: code.InteractiveConsole.interact() closes stdin https://bugs.python.org/issue34115 #34113: LLTRACE segv https://bugs.python.org/issue34113 #34111: python-config breaks when symlinked to another location https://bugs.python.org/issue34111 Most recent 15 issues waiting for review (15) ============================================= #34170: Py_Initialize(): computing path configuration must not have si https://bugs.python.org/issue34170 #34164: base64.b32decode() leads into UnboundLocalError and OverflowEr https://bugs.python.org/issue34164 #34162: idlelib/NEWS.txt for 3.8.0 (and backports) https://bugs.python.org/issue34162 #34158: Documentation of datetime '%z' format code is odd https://bugs.python.org/issue34158 #34152: performance of some list slice assignment margin cases can be https://bugs.python.org/issue34152 #34151: use malloc() for better performance of some list operations https://bugs.python.org/issue34151 #34149: Behavior of the min/max with key=None https://bugs.python.org/issue34149 #34144: venv activate.bat reset codepage fails on windows 10 https://bugs.python.org/issue34144 #34139: Remove stale unix datagram socket before binding https://bugs.python.org/issue34139 #34134: multiprocessing memory huge usage https://bugs.python.org/issue34134 #34132: Obscure netrc parser "bug" https://bugs.python.org/issue34132 #34129: CGITB does not mangle variables names https://bugs.python.org/issue34129 #34128: Release GIL periodically in _pickle module https://bugs.python.org/issue34128 #34126: Profiling certain invalid calls crashes Python https://bugs.python.org/issue34126 #34110: cPickle may raise AttributeError when loading concurrently in https://bugs.python.org/issue34110 Top 10 most discussed issues (10) ================================= #34060: regrtest: log "CPU usage" on Windows https://bugs.python.org/issue34060 25 msgs #34128: Release GIL periodically in _pickle module https://bugs.python.org/issue34128 11 msgs #34134: multiprocessing memory huge usage https://bugs.python.org/issue34134 10 msgs #29710: Incorrect representation caveat on bitwise operation docs https://bugs.python.org/issue29710 8 msgs #25150: 3.5: Include/pyatomic.h is incompatible with OpenMP (compilati https://bugs.python.org/issue25150 7 msgs #33111: Merely importing tkinter breaks parallel code (multiprocessing https://bugs.python.org/issue33111 7 msgs #34068: traceback.clear_frames(): Objects/typeobject.c:3086: _PyType_L https://bugs.python.org/issue34068 7 msgs #34118: Fix some class entries in 'Built-in Functions' https://bugs.python.org/issue34118 6 msgs #34149: Behavior of the min/max with key=None https://bugs.python.org/issue34149 6 msgs #34008: Do we support calling Py_Main() after Py_Initialize()? https://bugs.python.org/issue34008 5 msgs Issues closed (35) ================== #8799: Hang in lib/test/test_threading.py https://bugs.python.org/issue8799 closed by vstinner #18397: Python with MinGW https://bugs.python.org/issue18397 closed by martin.panter #18605: 2.7: test_threading hangs on Solaris 9 https://bugs.python.org/issue18605 closed by vstinner #24618: Invalid read in PyCode_New https://bugs.python.org/issue24618 closed by serhiy.storchaka #26439: ctypes.util.find_library fails when ldconfig/glibc not availab https://bugs.python.org/issue26439 closed by vstinner #28643: Broken makefile depends for profile-opt target https://bugs.python.org/issue28643 closed by nascheme #29199: test_regrtest fails if PCBuild directory doesn't exist https://bugs.python.org/issue29199 closed by ppperry #29253: Fix test_asyncore tests on Cygwin https://bugs.python.org/issue29253 closed by erik.bray #30928: Copy modified blurbs to idlelib/NEWS.txt for 3.7.0 https://bugs.python.org/issue30928 closed by terry.reedy #31342: test.bisect module causes tests to fail https://bugs.python.org/issue31342 closed by nascheme #32430: Simplify Modules/Setup{,.dist,.local} https://bugs.python.org/issue32430 closed by pitrou #32692: [3.6] test_threading.test_set_and_clear fails in AppVeyor CI https://bugs.python.org/issue32692 closed by vstinner #33869: doc Add link to list definition in Glossary list entry https://bugs.python.org/issue33869 closed by adelfino #33911: [EASY] test_docxmlrpc fails when run with -Werror https://bugs.python.org/issue33911 closed by vstinner #34087: int(s), float(s) and others may cause segmentation fault https://bugs.python.org/issue34087 closed by inada.naoki #34096: [2.7] test_audioop.test_max() failed: AssertionError: -2147483 https://bugs.python.org/issue34096 closed by gregory.p.smith #34102: None.splitlines raises AttributeError in email.feedparser https://bugs.python.org/issue34102 closed by terry.reedy #34108: 2to3 munges new lines on Windows https://bugs.python.org/issue34108 closed by jason.coombs #34109: Accumulator bug https://bugs.python.org/issue34109 closed by vicpires #34112: 3.7.0 build error with --enable-optimizations https://bugs.python.org/issue34112 closed by jayanthkoushik #34114: Build failing: use of undeclared identifier '_Py_END_SUPPRESS_ https://bugs.python.org/issue34114 closed by inada.naoki #34116: Move all bytes/bytearray/int/float/complex documentation to Bu https://bugs.python.org/issue34116 closed by rhettinger #34119: Able to name a variable as 'input'. This creates problem when https://bugs.python.org/issue34119 closed by rhettinger #34121: configure fails to detect C11 atomic support on clang https://bugs.python.org/issue34121 closed by benjamin.peterson #34122: inspect.getmembers does not retrive dataclass's __dataclass_fi https://bugs.python.org/issue34122 closed by corona10 #34123: ambiguous documentation for dict.popitem https://bugs.python.org/issue34123 closed by rhettinger #34124: email.message_from_binary_file documentation is broken https://bugs.python.org/issue34124 closed by berker.peksag #34130: test_signal: test_warn_on_full_buffer() failed on AppVeyor https://bugs.python.org/issue34130 closed by vstinner #34133: ValueError documented as being restricted to only "a built-in https://bugs.python.org/issue34133 closed by rhettinger #34140: Possible multiprocessing deadlock when placing too many object https://bugs.python.org/issue34140 closed by pitrou #34141: Speed up pickling simple non-recusive values https://bugs.python.org/issue34141 closed by serhiy.storchaka #34143: There is a constant definition error in errno.py https://bugs.python.org/issue34143 closed by brett.cannon #34147: doc Describe briefly sampling w/out replacement in random https://bugs.python.org/issue34147 closed by rhettinger #34153: 10/3 - last digit https://bugs.python.org/issue34153 closed by paul.moore #34157: NamedTemporaryFile can leave temporary files behind https://bugs.python.org/issue34157 closed by ncoghlan From J.Demeyer at UGent.be Sat Jul 21 12:21:12 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 21 Jul 2018 18:21:12 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 Message-ID: <5B535D78.5000307@UGent.be> Hello, I finally managed to get some real-life benchmarks for why we need a faster C calling protocol (see PEPs 576, 579, 580). I focused on the Cython compilation of SageMath. By default, a function in Cython is an instance of builtin_function_or_method (analogously, method_descriptor for a method), which has special optimizations in the CPython interpreter. But the option "binding=True" changes those to a custom class which is NOT optimized. I ran the full SageMath testsuite several times without and with binding=True to find out any significant differences. The most dramatic difference is multiplication for generic matrices. More precisely, with the following command: python -m timeit -s "from sage.all import MatrixSpace, GF; M = MatrixSpace(GF(9), 200).random_element()" "M * M" With binding=False, I got 10 loops, best of 3: 692 msec per loop With binding=True, I got 10 loops, best of 3: 1.16 sec per loop This is a big regression which should be gone completely with PEP 580. I should mention that this was done on Python 2.7.15 (SageMath is not yet ported to Python 3) but I see no reason why the conclusions shouldn't be valid for newer Python versions. I used SageMath 8.3.rc1 and Cython 0.28.4. I hope that this finally shows that the problems mentioned in PEP 579 are real. Jeroen. From songofacandy at gmail.com Sat Jul 21 13:07:32 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Sun, 22 Jul 2018 02:07:32 +0900 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5B535D78.5000307@UGent.be> References: <5B535D78.5000307@UGent.be> Message-ID: On Sun, Jul 22, 2018 at 1:28 AM Jeroen Demeyer wrote: > > Hello, > > I finally managed to get some real-life benchmarks for why we need a > faster C calling protocol (see PEPs 576, 579, 580). Good job. But I already +1 for adding support for extension callable type. Do you think this benchmark can be optimized more in future optimization which is possible by PEP 580, but not 576? > > I should mention that this was done on Python 2.7.15 (SageMath is not > yet ported to Python 3) but I see no reason why the conclusions > shouldn't be valid for newer Python versions. I used SageMath 8.3.rc1 > and Cython 0.28.4. Do you mean you backport LOAD_METHOD and fastcall to Python 2.7 for benchmarking? Reproducing it seems hard job to me... -- INADA Naoki From guido at python.org Sat Jul 21 13:55:03 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Jul 2018 10:55:03 -0700 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> Message-ID: I don?t think we can safely assume Python 3.7 has the same performance, actually. A lot has changed. On Sat, Jul 21, 2018 at 10:10 AM INADA Naoki wrote: > On Sun, Jul 22, 2018 at 1:28 AM Jeroen Demeyer wrote: > > > > Hello, > > > > I finally managed to get some real-life benchmarks for why we need a > > faster C calling protocol (see PEPs 576, 579, 580). > > Good job. But I already +1 for adding support for extension callable type. > Do you think this benchmark can be optimized more in future optimization > which is possible by PEP 580, but not 576? > > > > > I should mention that this was done on Python 2.7.15 (SageMath is not > > yet ported to Python 3) but I see no reason why the conclusions > > shouldn't be valid for newer Python versions. I used SageMath 8.3.rc1 > > and Cython 0.28.4. > > Do you mean you backport LOAD_METHOD and fastcall to Python 2.7 > for benchmarking? > Reproducing it seems hard job to me... > > -- > INADA Naoki > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Sat Jul 21 13:59:19 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 21 Jul 2018 19:59:19 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <3ce081fa87d04d728a8bb4c6c94b0f33@xmail103.UGent.be> References: <5B535D78.5000307@UGent.be> <3ce081fa87d04d728a8bb4c6c94b0f33@xmail103.UGent.be> Message-ID: <5B537477.1060102@UGent.be> On 2018-07-21 19:07, INADA Naoki wrote: > Good job. But I already +1 for adding support for extension callable type. > Do you think this benchmark can be optimized more in future optimization > which is possible by PEP 580, but not 576? I should clarify that the benchmark did not involve an implementation of PEP 576 or PEP 580. It simply shows what kind of regressions one gets when *not* implementing something like those PEPs. So this can't be used to compare PEP 576 versus PEP 580. I still think that PEP 576 would slow down bound method calls but without a reference implementation, I can only guess. (I know that PEP 576 claims a reference implementation but it doesn't correspond to the PEP. I'm basing myself on the text of PEP 576, not the "reference implementation".) > Do you mean you backport LOAD_METHOD and fastcall to Python 2.7 > for benchmarking? No, I did not. This is just benchmarking the difference between tp_call and more specialized call functions (In Python 2.7, that is call_function()). Jeroen. From J.Demeyer at UGent.be Sat Jul 21 14:16:06 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 21 Jul 2018 20:16:06 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> Message-ID: <5B537866.4020803@UGent.be> On 2018-07-21 19:55, Guido van Rossum wrote: > I don?t think we can safely assume Python 3.7 has the same performance, > actually. A lot has changed. I'm not denying that some things have changed. Rather, I claim that those changes wouldn't invalidate the benchmarks. I am comparing calls through tp_call (A) versus optimized call paths (B). I only need to assume that the speed improvements to (A) between 2.7 and 3.7 are not bigger than the speed improvements to (B). Most optimizations which have been done in Python 3.x target (B). In fact, I'm not aware of any optimization to (A) apart from maybe some minor code improvements. So I think it's a relatively safe assumption that the speed difference between (A) and (B) did not decrease from 2.7 to 3.7. Jeroen. From guido at python.org Sat Jul 21 16:46:21 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Jul 2018 13:46:21 -0700 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5B537866.4020803@UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: Given the cost of a mistake here I recommend a higher standard. But in the end it?s no longer my decision. On Sat, Jul 21, 2018 at 11:18 AM Jeroen Demeyer wrote: > On 2018-07-21 19:55, Guido van Rossum wrote: > > I don?t think we can safely assume Python 3.7 has the same performance, > > actually. A lot has changed. > > I'm not denying that some things have changed. Rather, I claim that > those changes wouldn't invalidate the benchmarks. > > I am comparing calls through tp_call (A) versus optimized call paths > (B). I only need to assume that the speed improvements to (A) between > 2.7 and 3.7 are not bigger than the speed improvements to (B). > > Most optimizations which have been done in Python 3.x target (B). In > fact, I'm not aware of any optimization to (A) apart from maybe some > minor code improvements. So I think it's a relatively safe assumption > that the speed difference between (A) and (B) did not decrease from 2.7 > to 3.7. > > > Jeroen. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Sat Jul 21 17:37:05 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sat, 21 Jul 2018 23:37:05 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: Guido van Rossum schrieb am 21.07.2018 um 22:46: > Given the cost of a mistake here I recommend a higher standard. May I ask what you think the "cost of a mistake" is here? Jeroen has already implemented most of this, and is willing to provide essentially a shrink-wrapped implementation. He has shown, using the current application benchmark suite, that his implementation does not degrade the application performance (that we know of). He has motivated in PEP form, and shown in his implementation, that the changes avoid much of the special casing that's currently littered in various spots of the interpreter and replace them by a much clearer protocol, thus reducing the overall maintenance cost. He has layed out a cleanup path to get rid of the current quirks in the split between function/method types, thus making the code easier to explain and lowering the entry barrier for newcomers to the code base. And, he has motivated that this protocol enables a future extension towards a specialised (faster) C level call protocol, which third party extensions would benefit from. Given all that, I'm having a hard time finding a "cost" in this. To me, it reads like a plain net win for all sides. Stefan From guido at python.org Sat Jul 21 19:14:38 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 21 Jul 2018 16:14:38 -0700 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: The cost would be if we were to end up maintaining all that code and it wouldn?t make much difference. Jeroen was asked to provide benchmarks but only provided them for Python 2. The reasoning that not much has changed that could affect the benchmarks feels a bit optimistic, that?s all. The new BDFL may be less demanding though. :=) On Sat, Jul 21, 2018 at 2:39 PM Stefan Behnel wrote: > Guido van Rossum schrieb am 21.07.2018 um 22:46: > > Given the cost of a mistake here I recommend a higher standard. > > May I ask what you think the "cost of a mistake" is here? > > Jeroen has already implemented most of this, and is willing to provide > essentially a shrink-wrapped implementation. He has shown, using the > current application benchmark suite, that his implementation does not > degrade the application performance (that we know of). He has motivated in > PEP form, and shown in his implementation, that the changes avoid much of > the special casing that's currently littered in various spots of the > interpreter and replace them by a much clearer protocol, thus reducing the > overall maintenance cost. He has layed out a cleanup path to get rid of the > current quirks in the split between function/method types, thus making the > code easier to explain and lowering the entry barrier for newcomers to the > code base. And, he has motivated that this protocol enables a future > extension towards a specialised (faster) C level call protocol, which third > party extensions would benefit from. > > Given all that, I'm having a hard time finding a "cost" in this. To me, it > reads like a plain net win for all sides. > > Stefan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/guido%40python.org > -- --Guido (mobile) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at holdenweb.com Sat Jul 21 19:57:08 2018 From: steve at holdenweb.com (Steve Holden) Date: Sun, 22 Jul 2018 00:57:08 +0100 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: On Sun, Jul 22, 2018 at 12:14 AM, Guido van Rossum wrote: > ?[...] > The new BDFL may be less demanding though. :=) > ?I sincerely hope not. regards Steve? -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Sun Jul 22 02:27:41 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Sun, 22 Jul 2018 15:27:41 +0900 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5B537866.4020803@UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: > > I am comparing calls through tp_call (A) versus optimized call paths > (B). I only need to assume that the speed improvements to (A) between > 2.7 and 3.7 are not bigger than the speed improvements to (B). > It's interesting... But I failed to build sage. It's build step is too different from normal Python package. It tooks very long time to build. And "install from source" document only describe step to `./sage` command work. It doesn't describe step to `improt sage` works. I feel it's not suitable for target application for discussing future optimization. Target application should be easy to test, benchmark and profile for all of core-devs interesting in these PEPs. From J.Demeyer at UGent.be Sun Jul 22 06:39:50 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sun, 22 Jul 2018 12:39:50 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5be87a63aa164f9c836b084e867b2b6f@xmail103.UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5be87a63aa164f9c836b084e867b2b6f@xmail103.UGent.be> Message-ID: <5B545EF6.3050607@UGent.be> On 2018-07-22 08:27, INADA Naoki wrote: > It's interesting... But I failed to build sage. What went wrong? > It's build step is > too different from > normal Python package. That's true because Sage considers itself a distribution rather than a package. But it's possible to pick the distribution apart and build just the Python package "sage". In fact, various Linux distros package Sage that way. The reason for it being a distribution is mainly that it has a huge number of dependencies (many of them not Python), so it wouldn't be possible to do "pip install sage" anyway. > It tooks very long time to build. That's just a matter of waiting a few hours. > And > "install from source" > document only describe step to `./sage` command work. It doesn't describe > step to `improt sage` works. Those two are pretty much equivalent. If you really want just the latter, you can run "make sageruntime" in the Sage root. > Target application should be easy to test, benchmark and profile for all of > core-devs interesting in these PEPs. I feel like the bar for this PEP is being raised all the time. First, you ask for an application benchmark and I provided an application benchmark. Now you complain that my application is not suitable. Why don't you just believe my timings? Jeroen. From J.Demeyer at UGent.be Sun Jul 22 07:02:04 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sun, 22 Jul 2018 13:02:04 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <374869393e964f7f852cde748d0509b1@xmail103.UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <374869393e964f7f852cde748d0509b1@xmail103.UGent.be> Message-ID: <5B54642C.7090100@UGent.be> On 2018-07-22 01:14, Guido van Rossum wrote: > Jeroen was asked to provide benchmarks > but only provided them for Python 2. The reasoning that not much has > changed that could affect the benchmarks feels a bit optimistic, that?s all. The micro-benchmarks showed a clear difference on Python 3.8 (git master) between tp_call and optimized call paths. The application benchmark on Python 2.7.15 shows that the difference between tp_call and optimized call paths matters in real applications. I agree that this isn't 100% bullet-proof evidence, but at least it's a strong indication that it might be worth it to start discussing the actual PEP. Jeroen. From stefan_ml at behnel.de Sun Jul 22 08:52:52 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 22 Jul 2018 14:52:52 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: Guido van Rossum schrieb am 22.07.2018 um 01:14: > The cost would be if we were to end up maintaining all that code and it > wouldn?t make much difference. Well, this is exactly my point. Someone has to maintain the *existing* code base and help newcomers to get into it and understand it. This is not easy. The proposed implementation *already* makes a difference. And it does not even degrade the performance while doing that, isn't that great? To make this clear ? right now, there is someone who stands up and volunteers to invest the work to clean up the current implementation. He has already designed, and even implemented, a protocol that applies to all types of callables in the same way *and* that is extensible for current and future needs and optimisations. I think this is way more than anyone could ask for, and it would be very sad if this chance was wasted, and we would have to remain with the current implementation. Stefan From guido at python.org Sun Jul 22 12:14:27 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Jul 2018 09:14:27 -0700 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: OK, I believe you, I was just referring to the relative irreproducibility of Jeroen's results (see INADA Naoki's problems). My main point is actually that until the Python core devs have elected a new BDFL or come up with some other process for accepting PEPs, no action will be taken on this PEP -- AFAIK there's no BDFL-Delegate for this PEP. You can argue until you're blue in the face, but the new process won't be ready until Jan 2019. (Read the python-committers archives, e.g. https://mail.python.org/pipermail/python-committers/2018-July/005935.html.) To make this clear -- right now, there is no-one who can approve this PEP, and you will have to wait until 2019 until there is. Of course you can form a subcommittee where you try to agree on the details, and I recommend that. But getting argumentative on python-dev is not going to be very productive. In 2019 people will remember that you were very forceful in taking your position, not what that position was or why it's reasonable. Sorry I don't have better news. On Sun, Jul 22, 2018 at 5:52 AM, Stefan Behnel wrote: > Guido van Rossum schrieb am 22.07.2018 um 01:14: > > The cost would be if we were to end up maintaining all that code and it > > wouldn?t make much difference. > > Well, this is exactly my point. Someone has to maintain the *existing* code > base and help newcomers to get into it and understand it. This is not easy. > The proposed implementation *already* makes a difference. And it does not > even degrade the performance while doing that, isn't that great? > > To make this clear ? right now, there is someone who stands up and > volunteers to invest the work to clean up the current implementation. He > has already designed, and even implemented, a protocol that applies to all > types of callables in the same way *and* that is extensible for current and > future needs and optimisations. I think this is way more than anyone could > ask for, and it would be very sad if this chance was wasted, and we would > have to remain with the current implementation. > > Stefan > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Sun Jul 22 12:30:55 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sun, 22 Jul 2018 18:30:55 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: <5B54B13F.6060004@UGent.be> On 2018-07-22 18:14, Guido van Rossum wrote: > Sorry I don't have better news. I don't consider that particularly bad news (but not good news either). I feel like the status on PEP 580 isn't anywhere near accepted anyway. I just hope that Python development won't stall completely. Even if no formal action can be taken on this PEP (or any other), I hope that there will still be informal discussion. Jeroen. From J.Demeyer at UGent.be Sun Jul 22 16:11:25 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sun, 22 Jul 2018 22:11:25 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> Message-ID: <5B54E4ED.7030200@UGent.be> On 2018-07-22 14:52, Stefan Behnel wrote: > Someone has to maintain the *existing* code > base and help newcomers to get into it and understand it. This is an important point that people seem to be overlooking. The complexity and maintenance burden of PEP 580 should be compared to the status-quo. The existing code is complicated, yet nobody seems to find that a problem. But when PEP 580 makes changes to that complicated code (and documents some of the existing complexity), it's suddenly the fault of PEP 580 that the code is complicated. I also wonder if there is a psychological difference simply because this is a PEP and not an issue on bugs.python.org. That might give the impression that it's a more serious thing somehow. Previous optimizations (https://bugs.python.org/issue26110 for example) were not done in a PEP and nobody ever mentioned that the extra complexity might be a problem. Finally, in some ways, my PEP would actually be a simplification because it replaces several special cases by one general protocol. Admittedly, the general protocol that I propose is more complicated than each existing special case individually but the overall complexity might actually decrease. Jeroen. From rosuav at gmail.com Sun Jul 22 16:12:17 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 Jul 2018 06:12:17 +1000 Subject: [Python-Dev] Finding Guido's replacement Message-ID: Guido's term as Benevolent Dictator For Life has been a long one, but in the wake of his resignation, we have an opportunity to correct some fundamental flaws in the system. Among them: * Guido lacks patience, as evidenced by the brevity of his acceptance posts. See https://mail.python.org/pipermail/python-dev/2017-December/151038.html and https://mail.python.org/pipermail/python-dev/2011-November/114545.html and particularly https://mail.python.org/pipermail/python-dev/2016-May/144646.html where Guido specifically cites his own lack of patience. * Lately, all Guido's actions have been to benefit his employer, not the Common Pythonista. We have proof of this from reliable reporting sources such as Twitter and social media. * Finally, "For Life" is far too long. We need to change our rulers periodically. I propose a new way to appoint a project head. All candidates shall be flown to an island owned by the Python Secret Underground (which emphatically does NOT exist, but an island that would be owned by it if it did), whereupon they parachute down, search for guns, and proceed to fight each other until only one is left alive. The survivor shall be treated to a chicken dinner and proclaimed Patient, Understanding, Benevolent Governor, a title which shall be retained for one fortnight, after which we repeat the exercise. If this plan meets with broad approval, I shall write up PEP 3401, in honour of the prior art in PEP 401. ChrisA From solipsis at pitrou.net Sun Jul 22 16:32:04 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Jul 2018 22:32:04 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5B54E4ED.7030200@UGent.be> Message-ID: <20180722223204.5122e81d@fsol> On Sun, 22 Jul 2018 22:11:25 +0200 Jeroen Demeyer wrote: > On 2018-07-22 14:52, Stefan Behnel wrote: > > Someone has to maintain the *existing* code > > base and help newcomers to get into it and understand it. > > This is an important point that people seem to be overlooking. The > complexity and maintenance burden of PEP 580 should be compared to the > status-quo. The existing code is complicated, yet nobody seems to find > that a problem. But when PEP 580 makes changes to that complicated code > (and documents some of the existing complexity), it's suddenly the fault > of PEP 580 that the code is complicated. > > I also wonder if there is a psychological difference simply because this > is a PEP and not an issue on bugs.python.org. That might give the > impression that it's a more serious thing somehow. Previous > optimizations (https://bugs.python.org/issue26110 for example) were not > done in a PEP and nobody ever mentioned that the extra complexity might > be a problem. Two things: - the issue26110 changes are not very large, it's just two additional opcodes and a bit of support code - more importantly, issue26110 is entirely internal changes, while you are proposing to add a new protocol (which is like a new API) Regards Antoine. From J.Demeyer at UGent.be Sun Jul 22 16:54:24 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sun, 22 Jul 2018 22:54:24 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5B54E4ED.7030200@UGent.be> Message-ID: <5B54EF00.3060507@UGent.be> On 2018-07-22 22:32, Antoine Pitrou wrote: > Two things: > - the issue26110 changes are not very large, it's just two additional > opcodes and a bit of support code I didn't mean to compare PEP 580 to that specific issue, it was just an example. Even if issue26110 is small, the total complexity added by many incremental changes can still be big. > - more importantly, issue26110 is entirely internal changes, while you > are proposing to add a new protocol (which is like a new API) Just to make sure I understand you: your point is that it's automatically more complicated because it's an API instead of an internal change? From stefan_ml at behnel.de Sun Jul 22 17:09:05 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 22 Jul 2018 23:09:05 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5B54EF00.3060507@UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5B54E4ED.7030200@UGent.be> <5B54EF00.3060507@UGent.be> Message-ID: Jeroen Demeyer schrieb am 22.07.2018 um 22:54: > On 2018-07-22 22:32, Antoine Pitrou wrote: >> - more importantly, issue26110 is entirely internal changes, while you >> ?? are proposing to add a new protocol (which is like a new API) > > Just to make sure I understand you: your point is that it's automatically > more complicated because it's an API instead of an internal change? I think it's more that it changes something substantial in a way that is not just internal but visible to users. I think it's more than just a tracker issue and worth going through the PEP process. Stefan From solipsis at pitrou.net Sun Jul 22 17:11:26 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Sun, 22 Jul 2018 23:11:26 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5B54E4ED.7030200@UGent.be> <5B54EF00.3060507@UGent.be> Message-ID: <20180722231126.247e6bfb@fsol> On Sun, 22 Jul 2018 22:54:24 +0200 Jeroen Demeyer wrote: > > > - more importantly, issue26110 is entirely internal changes, while you > > are proposing to add a new protocol (which is like a new API) > > Just to make sure I understand you: your point is that it's > automatically more complicated because it's an API instead of an > internal change? No, what I mean here is that adding a public API requires additional thought compared to adding an internal optimization (the internal optimization can easily be reverted if there's a problem, while the public API can not). Hence the preference for a PEP. Regards Antoine. From guido at python.org Sun Jul 22 18:28:39 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 22 Jul 2018 15:28:39 -0700 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5B54E4ED.7030200@UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5B54E4ED.7030200@UGent.be> Message-ID: On Sun, Jul 22, 2018 at 1:11 PM, Jeroen Demeyer wrote: > On 2018-07-22 14:52, Stefan Behnel wrote: > >> Someone has to maintain the *existing* code >> base and help newcomers to get into it and understand it. >> > > This is an important point that people seem to be overlooking. The > complexity and maintenance burden of PEP 580 should be compared to the > status-quo. The existing code is complicated, yet nobody seems to find that > a problem. But when PEP 580 makes changes to that complicated code (and > documents some of the existing complexity), it's suddenly the fault of PEP > 580 that the code is complicated. > > I also wonder if there is a psychological difference simply because this > is a PEP and not an issue on bugs.python.org. That might give the > impression that it's a more serious thing somehow. Previous optimizations ( > https://bugs.python.org/issue26110 for example) were not done in a PEP > and nobody ever mentioned that the extra complexity might be a problem. > > Finally, in some ways, my PEP would actually be a simplification because > it replaces several special cases by one general protocol. Admittedly, the > general protocol that I propose is more complicated than each existing > special case individually but the overall complexity might actually > decrease. So does your implementation of the PEP result in a net increase or decrease of the total lines of code? I know that that's a poor proxy for complexity (we can all imagine example bits of code that would become less complex by rewriting them in more lines), but if your diff actually deleted more lines than it adds, that would cut short a lot of discussion. I have a feeling though that that's not the case, and now you're in the defense. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfahlgren at gmail.com Sun Jul 22 19:04:59 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Sun, 22 Jul 2018 16:04:59 -0700 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: Message-ID: On Sun, Jul 22, 2018 at 1:19 PM Chris Angelico wrote: > * Finally, "For Life" is far too long. We need to change our rulers > periodically. > ?With the proposed bi-weekly death matches, "for life" may actually be too short.? -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Sun Jul 22 19:54:42 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Mon, 23 Jul 2018 02:54:42 +0300 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5B54E4ED.7030200@UGent.be> Message-ID: <1886fbbc-08e4-528c-588e-685e80a4f81d@mail.mipt.ru> I think it'll benefit all to keep the discussion objective, or at least "good subjective" (https://stackoverflow.blog/2010/09/29/good-subjective-bad-subjective/). Laments or mutual accusations are only wasting everyone's time, including the writers. It's strange that even Guido jumped on the bandwagon -- since he's supposed to have had lots of experience to tell right away when a discussion has become unproductive. (Or maybe he's testing us?) All the material to discuss that we have in this thread is a single test result that's impossible to reproduce and impossible to run in Py3. All that this shows is that the PEPs will _likely_ substantially improve performance in some scenarios. It's however impossible to say from this how frequent these scenarios are in practice, and how consistent the improvement is among them. Likewise, it's impossible to say anything about the complexity the changes will reduce/introduce without a proof-of-concept implementation. So while this is an argument in favor of the PEPs, it's too flimsy _on its own_ to accept anything. More and better tests and/or sample implementations are needed to say anything more conclusive. All that was already pointed out, and that's where the thread should have ended IMO 'cuz there's nothing else to say on the matter. On 23.07.2018 1:28, Guido van Rossum wrote: > On Sun, Jul 22, 2018 at 1:11 PM, Jeroen Demeyer > wrote: > > On 2018-07-22 14:52, Stefan Behnel wrote: > > Someone has to maintain the *existing* code > base and help newcomers to get into it and understand it. > > > This is an important point that people seem to be overlooking. The > complexity and maintenance burden of PEP 580 should be compared to > the status-quo. The existing code is complicated, yet nobody seems > to find that a problem. But when PEP 580 makes changes to that > complicated code (and documents some of the existing complexity), > it's suddenly the fault of PEP 580 that the code is complicated. > > I also wonder if there is a psychological difference simply > because this is a PEP and not an issue on bugs.python.org > . That might give the impression that it's > a more serious thing somehow. Previous optimizations > (https://bugs.python.org/issue26110 > for example) were not done in > a PEP and nobody ever mentioned that the extra complexity might be > a problem. > > Finally, in some ways, my PEP would actually be a simplification > because it replaces several special cases by one general protocol. > Admittedly, the general protocol that I propose is more > complicated than each existing special case individually but the > overall complexity might actually decrease. > > > So does your implementation of the PEP result in a net increase or > decrease of the total lines of code? I know that that's a poor proxy > for complexity (we can all imagine example bits of code that would > become less complex by rewriting them in more lines), but if your diff > actually deleted more lines than it adds, that would cut short a lot > of discussion. I have a feeling though that that's not the case, and > now you're in the defense. > > -- > --Guido van Rossum (python.org/~guido ) > > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From vano at mail.mipt.ru Sun Jul 22 20:41:58 2018 From: vano at mail.mipt.ru (Ivan Pozdeev) Date: Mon, 23 Jul 2018 03:41:58 +0300 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: Message-ID: <9e936880-1aaa-5b4e-b1d4-82c4a57c921b@mail.mipt.ru> Whatever you decide, please research existing practices and their results so as not to repeat the same mistakes as others made before you. In particular, http://meatballwiki.org/wiki/BenevolentDictator and https://en.wikipedia.org/wiki/Anti-pattern . It would be a waste if Python falls victim to the same trapping as thousands before it. On 22.07.2018 23:12, Chris Angelico wrote: > Guido's term as Benevolent Dictator For Life has been a long one, but > in the wake of his resignation, we have an opportunity to correct some > fundamental flaws in the system. Among them: > > * Guido lacks patience, as evidenced by the brevity of his acceptance > posts. See https://mail.python.org/pipermail/python-dev/2017-December/151038.html > and https://mail.python.org/pipermail/python-dev/2011-November/114545.html > and particularly > https://mail.python.org/pipermail/python-dev/2016-May/144646.html > where Guido specifically cites his own lack of patience. > > * Lately, all Guido's actions have been to benefit his employer, not > the Common Pythonista. We have proof of this from reliable reporting > sources such as Twitter and social media. > > * Finally, "For Life" is far too long. We need to change our rulers > periodically. > > I propose a new way to appoint a project head. All candidates shall be > flown to an island owned by the Python Secret Underground (which > emphatically does NOT exist, but an island that would be owned by it > if it did), whereupon they parachute down, search for guns, and > proceed to fight each other until only one is left alive. The survivor > shall be treated to a chicken dinner and proclaimed Patient, > Understanding, Benevolent Governor, a title which shall be retained > for one fortnight, after which we repeat the exercise. > > If this plan meets with broad approval, I shall write up PEP 3401, in > honour of the prior art in PEP 401. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vano%40mail.mipt.ru -- Regards, Ivan From arj.python at gmail.com Mon Jul 23 00:40:15 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Mon, 23 Jul 2018 08:40:15 +0400 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: <9e936880-1aaa-5b4e-b1d4-82c4a57c921b@mail.mipt.ru> References: <9e936880-1aaa-5b4e-b1d4-82c4a57c921b@mail.mipt.ru> Message-ID: sometimes back after the event the BDFL1 said that "the new BDFL might be less demanding" hint to an imminent new one? i won't tell much the core devs (not all) have already shown their preferences fun fact: weirdly enough after BDFL1 took a vac (for life?), google made it's appearance on the mailing list THE NEED FOR CENTRAL AUTHORITY there is an absolute need for a central coordinator. the main threat to open source projects is politics. when you have someone to settle stuffs, fine you move on PROCEDURE OF ACCEPTANCE since BDFL1 is still alive, that saves the community some trouble. he is i presume aware of py-related stuffs and his choosing of a successor is a most viable option PREMISE OF CHOICE the choice should be made in consultation with the core devs as they are recognised members of the community based on their contributions. the core devs should orient the choice, they in themselves have not the power to veto, topple or reverse the decision. in the unlikely case of complete unfavouritism, the BDFL1 can pursue further CORE DEVS AS COMMITTEE the above hints to the core devs as a body in itself, beyond programming VALIDITY OF CHOICE the choice of BDFL1 must be acknowledged by the community. psf members should vote of whether they like the choice or not, fir it is the users who make the language valuable. by psf members, reference is made to the basic type of membership, open to all py programmers who have agreed to the psf rules. in case of non agreement, the process is to be repeated CRITICISM OF THE BDFLs BY FORMER AND LATTER ONES no BFFL shall criticise others. in case of non-acceptance of actions, a PHA shall be submitted PHA a Python Halt Appeal is a document submitted to the current core devs and the current BDFL to halt a specific activity considered "nuisible" to the growth and development of python. in case of majority acceptance of the core devs, it shall be accepted THE NEED FOR ALTERNATIVE NAMING OF DICTATOR OR EMPHASISING THE MEANING the python leader if he has absolute power should not be questioned or should not be made to back out as his appointment is by definition CITING THIS DOCUMENT / LICENSE this mail can be freely used, modified or built upon provided that attribution is made to the author in clear ways with no obfuscation whatsoever Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ Mauritius -------------- next part -------------- An HTML attachment was scrubbed... URL: From yselivanov.ml at gmail.com Mon Jul 23 04:47:25 2018 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 23 Jul 2018 11:47:25 +0300 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: Message-ID: On Sun, Jul 22, 2018 at 11:18 PM Chris Angelico wrote: > > * Lately, all Guido's actions have been to benefit his employer, not > the Common Pythonista. We have proof of this from reliable reporting > sources such as Twitter and social media. > This accusation is ridiculous and not appreciated. Type hinting is one of the most praised Python features in pretty much any big company, where managing millions of lines of Python is challenging. Next time, I also suggest you to cite and name your 'reliable reporting sources' otherwise this is just bs. Yury -- Yury -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Jul 23 05:01:21 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Jul 2018 11:01:21 +0200 Subject: [Python-Dev] Finding Guido's replacement References: Message-ID: <20180723110121.7aee2ebe@fsol> On Mon, 23 Jul 2018 11:47:25 +0300 Yury Selivanov wrote: > On Sun, Jul 22, 2018 at 11:18 PM Chris Angelico wrote: > > > > > * Lately, all Guido's actions have been to benefit his employer, not > > the Common Pythonista. We have proof of this from reliable reporting > > sources such as Twitter and social media. > > > > This accusation is ridiculous and not appreciated. Type hinting is one of > the most praised Python features in pretty much any big company, where > managing millions of lines of Python is challenging. Next time, I also > suggest you to cite and name your 'reliable reporting sources' otherwise > this is just bs. I suspect Chris A. was merely joking, though I'm not sure what the joke ultimately is supposed to be about. Regards Antoine. From yselivanov.ml at gmail.com Mon Jul 23 05:21:13 2018 From: yselivanov.ml at gmail.com (Yury Selivanov) Date: Mon, 23 Jul 2018 12:21:13 +0300 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: <20180723110121.7aee2ebe@fsol> References: <20180723110121.7aee2ebe@fsol> Message-ID: On Mon, Jul 23, 2018 at 12:03 PM Antoine Pitrou wrote: > > I suspect Chris A. was merely joking, though I'm not sure what the joke > ultimately is supposed to be about. > Ah, right, I stopped reading his email after the quoted line. Well executed. Yury > > -- Yury -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Mon Jul 23 05:07:32 2018 From: phd at phdru.name (Oleg Broytman) Date: Mon, 23 Jul 2018 11:07:32 +0200 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: Message-ID: <20180723090732.mki55hinukwugmzs@phdru.name> Hi! On Mon, Jul 23, 2018 at 11:47:25AM +0300, Yury Selivanov wrote: > On Sun, Jul 22, 2018 at 11:18 PM Chris Angelico wrote: > > > > > * Lately, all Guido's actions have been to benefit his employer, not > > the Common Pythonista. We have proof of this from reliable reporting > > sources such as Twitter and social media. > > This accusation is ridiculous and not appreciated. Yury, the entire message from Chris was a joke. Of bad taste, in my opinion. And I also didn't get initially that it was a joke, it took me a few minutes to understand. > Yury > -- > Yury Oleg. -- Oleg Broytman https://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From J.Demeyer at UGent.be Mon Jul 23 05:51:41 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 23 Jul 2018 11:51:41 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> Message-ID: <5B55A52D.3060207@UGent.be> I did exactly the same benchmark again with Python 3.7 and the results are similar. I'm copying and editing the original post for completeness: I finally managed to get some real-life benchmarks for why we need a faster C calling protocol (see PEPs 576, 579, 580). I focused on the Cython compilation of SageMath. By default, a function in Cython is an instance of builtin_function_or_method (analogously, method_descriptor for a method), which has special optimizations in the CPython interpreter. But the option "binding=True" changes those to a custom class which is NOT optimized. I ran the full SageMath testsuite several times on Python 2.7 without and with binding=True to find out any significant differences. I then checked if those differences could be reproduced on Python 3.7 (SageMath has not been fully ported to Python 3 yet). The most dramatic difference is multiplication for generic matrices. More precisely, with the following command: python3 -m timeit -s "from sage.all import MatrixSpace, GF; M = MatrixSpace(GF(9), 200).random_element()" "M * M" With binding=False, I got 1 loop, best of 5: 1.19 sec per loop With binding=True, I got 1 loop, best of 5: 1.83 sec per loop This is a big regression which should be gone completely with PEP 580. I used Python 3.7, SageMath 8.3.rc1 (plus a few patches to make it work with binding=True and with Python 3.7) and Cython 0.28.4. I hope that this finally shows that the problems mentioned in PEP 579 are real. Jeroen. From solipsis at pitrou.net Mon Jul 23 06:13:55 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Jul 2018 12:13:55 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B55A52D.3060207@UGent.be> Message-ID: <20180723121355.4a7cf567@fsol> On Mon, 23 Jul 2018 11:51:41 +0200 Jeroen Demeyer wrote: > I did exactly the same benchmark again with Python 3.7 and the results > are similar. I'm copying and editing the original post for completeness: > > [...] > > I hope that this finally shows that the problems mentioned in PEP 579 > are real. Just for the record, personally I have no doubt that the problems are real. IMHO the main point of discussion should be judging the solution you are proposing (and thank you, really, for being persistent, since this isn't an easy discussion). Best regards Antoine. From rosuav at gmail.com Mon Jul 23 06:37:05 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 23 Jul 2018 20:37:05 +1000 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: <20180723090732.mki55hinukwugmzs@phdru.name> References: <20180723090732.mki55hinukwugmzs@phdru.name> Message-ID: On Mon, Jul 23, 2018 at 7:07 PM, Oleg Broytman wrote: > Hi! > > On Mon, Jul 23, 2018 at 11:47:25AM +0300, Yury Selivanov wrote: >> On Sun, Jul 22, 2018 at 11:18 PM Chris Angelico wrote: >> >> > >> > * Lately, all Guido's actions have been to benefit his employer, not >> > the Common Pythonista. We have proof of this from reliable reporting >> > sources such as Twitter and social media. >> >> This accusation is ridiculous and not appreciated. > > Yury, the entire message from Chris was a joke. Of bad taste, in my > opinion. And I also didn't get initially that it was a joke, it took me > a few minutes to understand. If the reference to PEP 401 didn't tip you off and you don't recognize the references to famous Battle Royale games, then perhaps the fact that I called social media "reliable" might have been a good hint that I wasn't being entirely serious? ChrisA From phd at phdru.name Mon Jul 23 06:41:19 2018 From: phd at phdru.name (Oleg Broytman) Date: Mon, 23 Jul 2018 12:41:19 +0200 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: <20180723090732.mki55hinukwugmzs@phdru.name> Message-ID: <20180723104119.izkiose7j52kg4m4@phdru.name> On Mon, Jul 23, 2018 at 08:37:05PM +1000, Chris Angelico wrote: > On Mon, Jul 23, 2018 at 7:07 PM, Oleg Broytman wrote: > > I also didn't get initially that it was a joke, it took me > > a few minutes to understand. > > If the reference to PEP 401 didn't tip you off and you don't recognize > the references to famous Battle Royale games, then perhaps the fact > that I called social media "reliable" might have been a good hint that > I wasn't being entirely serious? I was sure you've been serious until I got to the last paragraph. > ChrisA Oleg. -- Oleg Broytman https://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From solipsis at pitrou.net Mon Jul 23 06:46:46 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 23 Jul 2018 12:46:46 +0200 Subject: [Python-Dev] Finding Guido's replacement References: <20180723090732.mki55hinukwugmzs@phdru.name> Message-ID: <20180723124646.59e49d7a@fsol> Please, take this thread off-list. Nobody is interested in you explaining a joke. On Mon, 23 Jul 2018 20:37:05 +1000 Chris Angelico wrote: > On Mon, Jul 23, 2018 at 7:07 PM, Oleg Broytman wrote: > > Hi! > > > > On Mon, Jul 23, 2018 at 11:47:25AM +0300, Yury Selivanov wrote: > >> On Sun, Jul 22, 2018 at 11:18 PM Chris Angelico wrote: > >> > >> > > >> > * Lately, all Guido's actions have been to benefit his employer, not > >> > the Common Pythonista. We have proof of this from reliable reporting > >> > sources such as Twitter and social media. > >> > >> This accusation is ridiculous and not appreciated. > > > > Yury, the entire message from Chris was a joke. Of bad taste, in my > > opinion. And I also didn't get initially that it was a joke, it took me > > a few minutes to understand. > > If the reference to PEP 401 didn't tip you off and you don't recognize > the references to famous Battle Royale games, then perhaps the fact > that I called social media "reliable" might have been a good hint that > I wasn't being entirely serious? > > ChrisA From J.Demeyer at UGent.be Mon Jul 23 07:13:02 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 23 Jul 2018 13:13:02 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5B54E4ED.7030200@UGent.be> Message-ID: <5B55B83E.8070609@UGent.be> On 2018-07-23 01:54, Ivan Pozdeev via Python-Dev wrote: > All the material to discuss that we have in this thread is a single test > result that's impossible to reproduce and impossible to run in Py3. I just posted that it can be reproduced on Python 3.7: https://mail.python.org/pipermail/python-dev/2018-July/154740.html I admit that it's not entirely trivial to do that. The Python 3 port of SageMath is still work in progress and the Python 3.7 port even more so. So it requires a few patches. If somebody wants to reproduce those results right now, I could give more details. But really, I would recommend to wait a month or so and then hopefully those patches will be merged. > It's however impossible to say from this > how frequent these scenarios are in practice And how would you suggest that we measure that? All benchmarks are artificial in some way: for every benchmark, one can find reasons why it's not relevant. > and how consistent the improvement is among them. I only posted the most serious regression. As another data point, the total time to run the full SageMath testsuite increased by about 1.8% when compiling the Cython code with binding=True. So one could assume that there is an average improvement of 1.8% with a much larger improvement in a few specific cases. > Likewise, it's impossible to say anything > about the complexity the changes will reduce/introduce without a > proof-of-concept implementation. Why do you think that there is no implementation? As mentioned in PEP 580, there is an implementation at https://github.com/jdemeyer/cpython/tree/pep580 Jeroen. From songofacandy at gmail.com Mon Jul 23 07:48:54 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 23 Jul 2018 20:48:54 +0900 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <20180723121355.4a7cf567@fsol> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B55A52D.3060207@UGent.be> <20180723121355.4a7cf567@fsol> Message-ID: On Mon, Jul 23, 2018 at 7:20 PM Antoine Pitrou wrote: > > Just for the record, personally I have no doubt that the problems are > real. IMHO the main point of discussion should be judging the solution > you are proposing (and thank you, really, for being persistent, since > this isn't an easy discussion). > Me too. My interest is what is the best benefit / complexity ratio. -- INADA Naoki From J.Demeyer at UGent.be Mon Jul 23 08:16:00 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 23 Jul 2018 14:16:00 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <11461446fe1e4954af86545e3eea74a9@xmail103.UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5B54E4ED.7030200@UGent.be> <11461446fe1e4954af86545e3eea74a9@xmail103.UGent.be> Message-ID: <5B55C700.8020807@UGent.be> On 2018-07-23 00:28, Guido van Rossum wrote: > So does your implementation of the PEP result in a net increase or > decrease of the total lines of code? 12 files changed, 918 insertions(+), 704 deletions(-) That's a net increase, so there is no obvious win here. Still, I have various excuses for this increased number of lines of code: 1. I'm adding more comments and lines containing only characters from the set " {};". This accounts for about 60% in the increase in number of lines of code. Clearly, this shouldn't count against me. 2. I still need to support some old ways of doing things. For backwards compatibility, functions and methods are still defined using a PyMethodDef. So I need to convert this old structure to the new protocol. I also keep support for some functions that my PEP makes obsolete, such as PyCFunction_Call(). All this requires code, but that code is simple. 3. In a few cases, specialized code is replaced by general code. For example, code that needs the __name__ of a function changes from accessing a known field in a C structure (func->m_ml->ml_name) to an actual Python attribute lookup. And Python code like def __qualname__(func): name = func.__name__ try: parent_qualname = func.__parent__.__qualname__ except AttributeError: return name return str(parent_qualname) + "." + name is conceptually pretty simple, but it becomes 37 lines of C code in my implementation. 4. The PEP does actually add a completely new feature: the flag CCALL_FUNCARG. That is something that I could easily remove for now, but PEP 580 would be a lot less useful without it. So it's something that I would certainly want to add in a later PEP anyway. It's also required for PEP 573. Jeroen. From songofacandy at gmail.com Mon Jul 23 09:01:22 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 23 Jul 2018 22:01:22 +0900 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5B545EF6.3050607@UGent.be> References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B537866.4020803@UGent.be> <5be87a63aa164f9c836b084e867b2b6f@xmail103.UGent.be> <5B545EF6.3050607@UGent.be> Message-ID: On Sun, Jul 22, 2018 at 7:42 PM Jeroen Demeyer wrote: > > On 2018-07-22 08:27, INADA Naoki wrote: > > It's interesting... But I failed to build sage. > > What went wrong? > I can't install Sage into my virtual environment, so I can't run > python -m timeit -s "from sage.all import MatrixSpace, GF; M = > MatrixSpace(GF(9), 200).random_element()" "M * M" But I can run it finally, with `sage -python` command. And I profiled what is the bottleneck of this oneliner. https://gist.github.com/methane/91533f68e88f89b7ec7f71855c069792 Cython does optimizations like CPython does internally. When calling function with one arg, and if the function is PyCFunction, Cython call underlaying C function directly. cfunc = PyCFunction_GET_FUNCTION(func); self = PyCFunction_GET_SELF(func); result = cfunc(self, arg); This is why such regression happened on even Python 2.7. And I think I can write small benchmark emulate it for now. -- INADA Naoki From J.Demeyer at UGent.be Mon Jul 23 11:02:47 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 23 Jul 2018 17:02:47 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: References: <5B535D78.5000307@UGent.be> <5ff560bdd3a44a7ea03defe310947f95@xmail103.UGent.be> <5B55A52D.3060207@UGent.be> Message-ID: <5B55EE17.1070202@UGent.be> On 2018-07-23 12:13, Antoine Pitrou wrote: > IMHO the main point of discussion should be judging the solution > you are proposing Yes please. I would very much welcome a discussion about the actual content of the PEP instead of meta-discussions like how complicated it is. From contact at elelay.fr Mon Jul 23 10:31:12 2018 From: contact at elelay.fr (Eric Le Lay) Date: Mon, 23 Jul 2018 16:31:12 +0200 Subject: [Python-Dev] [Windows] how to prevent the wrong version of zlib1.dll to be used by lib-dynload modules Message-ID: <20180723163112.59a2677d@dulcinea.localdomain> Hello list, I encountered a problem with the Windows packaging of gPodder[1] using msys2: basic libraries (zlib, openssl) depended upon by python platform-specific modules are loaded preferably : 1. from lib-dynload (where they are not) 2. from the Windows directory (can be any version) 3. from the binary directory, next to gpodder.exe (where they are) So an old zlib1.dll installed by another application in c:\Windows was loaded, incompatible with libpng and gPodder couldn't start. I don't know what's the best approach to solve it: - copy those libraries to lib\pythonxx\lib-dynload (works) - preload them in my main script before they are loaded by the module (works) - patch something in python (dynload_win.c ?) to search first in the executable directory (not tried) Please can you provide me with insight on this? Details in the issue: [1] Thanks, [1] https://github.com/gpodder/gpodder/issues/478 -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: Signature digitale OpenPGP URL: From steve.dower at python.org Mon Jul 23 12:30:39 2018 From: steve.dower at python.org (Steve Dower) Date: Mon, 23 Jul 2018 17:30:39 +0100 Subject: [Python-Dev] [Windows] how to prevent the wrong version of zlib1.dll to be used by lib-dynload modules In-Reply-To: <20180723163112.59a2677d@dulcinea.localdomain> References: <20180723163112.59a2677d@dulcinea.localdomain> Message-ID: <6106ef02-39b1-71cc-17d2-bb18cdd9ff2b@python.org> In general, if the dependent DLL is in the same directory as the module loading it (the .pyd or .exe), then it should be loaded first. If it's alongside the .exe, it should be loaded before any of the other search paths. If it's being loaded directly from Python, your best option is to resolve the full path before trying to load it (via ctypes or whatever). I have argued in the past about trying to hack the importer in order to "fix" this (since any fix is highly likely to break currently working install layouts), and this sounds like it is best solved by putting dependencies in an expected location. However, I don't know how much impact the msys2 aspect of this has. You mention "lib\pythonxx\lib-dynload", which is not present in a normal Python install on Windows. So possibly you're already on a non-standard build, which means my advice is pretty useless to you. There are also significant parts of both zlib and openssl available in a normal Python install on Windows, so perhaps you don't need to include alternate copies of those with your package? Are there particular features or APIs missing that you need? Cheers, Steve On 23Jul2018 1531, Eric Le Lay wrote: > Hello list, > > I encountered a problem with the Windows packaging of gPodder[1] > using msys2: > > basic libraries (zlib, openssl) depended upon by python > platform-specific modules are loaded preferably : > 1. from lib-dynload (where they are not) > 2. from the Windows directory (can be any version) > 3. from the binary directory, next to gpodder.exe (where they are) > > So an old zlib1.dll installed by another application in c:\Windows was > loaded, incompatible with libpng and gPodder couldn't start. > > I don't know what's the best approach to solve it: > - copy those libraries to lib\pythonxx\lib-dynload (works) > - preload them in my main script before they are loaded by the module > (works) > - patch something in python (dynload_win.c ?) to search first in the > executable directory (not tried) > > Please can you provide me with insight on this? > > Details in the issue: [1] > > Thanks, > > [1] https://github.com/gpodder/gpodder/issues/478 From njs at pobox.com Mon Jul 23 13:47:33 2018 From: njs at pobox.com (Nathaniel Smith) Date: Mon, 23 Jul 2018 10:47:33 -0700 Subject: [Python-Dev] [Windows] how to prevent the wrong version of zlib1.dll to be used by lib-dynload modules In-Reply-To: <20180723163112.59a2677d@dulcinea.localdomain> References: <20180723163112.59a2677d@dulcinea.localdomain> Message-ID: On Mon, Jul 23, 2018, 08:22 Eric Le Lay wrote: > Hello list, > > I encountered a problem with the Windows packaging of gPodder[1] > using msys2: > > basic libraries (zlib, openssl) depended upon by python > platform-specific modules are loaded preferably : > 1. from lib-dynload (where they are not) > 2. from the Windows directory (can be any version) > 3. from the binary directory, next to gpodder.exe (where they are) > > So an old zlib1.dll installed by another application in c:\Windows was > loaded, incompatible with libpng and gPodder couldn't start. > > I don't know what's the best approach to solve it: > - copy those libraries to lib\pythonxx\lib-dynload (works) > - preload them in my main script before they are loaded by the module > (works) > - patch something in python (dynload_win.c ?) to search first in the > executable directory (not tried) > The way we avoid these kinds of issues on Linux is to rename included libraries to unique names (e.g. zlib1-aef3742bc3e.dll), and patch the extension modules to look for the new names. There's a script (auditwheel) that does the heavy lifting. On MacOS there's a similar script people use, called "delocate". Unfortunately no one has written a script like this for Windows yet. I think it'd be neat if eventually we could consolidate all this into auditwheel, but there aren't that many people interested in working on it. If you wanted to help, the hardest part is already done: https://github.com/njsmith/machomachomangler/blob/master/README.rst#pe-features Alternatively if you just want a hack that works quick, I'd do the preload thing. In general you need to do this anyway, if you don't have all your extension modules in the same directory. (Unfortunately Windows doesn't have the equivalent of RPATH.) -n > -------------- next part -------------- An HTML attachment was scrubbed... URL: From acataldo at google.com Mon Jul 23 19:14:48 2018 From: acataldo at google.com (Adam Cataldo) Date: Mon, 23 Jul 2018 16:14:48 -0700 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: <9e936880-1aaa-5b4e-b1d4-82c4a57c921b@mail.mipt.ru> Message-ID: > fun fact: weirdly enough after BDFL1 took a vac (for life?), google made > it's appearance on the mailing list > As the Googler who appeared on the mailing list, I can say this was just a coincidence. I was a bit nervous no one would respond though, given Guido's vacation :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From eryksun at gmail.com Tue Jul 24 03:15:17 2018 From: eryksun at gmail.com (eryk sun) Date: Tue, 24 Jul 2018 07:15:17 +0000 Subject: [Python-Dev] [Windows] how to prevent the wrong version of zlib1.dll to be used by lib-dynload modules In-Reply-To: <20180723163112.59a2677d@dulcinea.localdomain> References: <20180723163112.59a2677d@dulcinea.localdomain> Message-ID: On Mon, Jul 23, 2018 at 2:31 PM, Eric Le Lay wrote: > > I encountered a problem with the Windows packaging of gPodder[1] > using msys2: Are you using regular Windows Python with msys2, or their custom port? I installed msys2 and used pacman to install Python 3.6. The msys2 environment names libraries with an "msys-" prefix in the "/usr/bin" directory, such as msys-python3.6m.dll, msys-readline7.dll, and msys-z.dll (zlib). This is also the application directory of the msys2 build of Python (i.e. "/usr/bin/python.exe"), so it's the first directory in the default DLL search path (ahead of system directories and PATH). Unlike Windows Python, msys2 Python does not use the alternate search path that replaces the application directory with the DLL directory in the search path. A way to implement this that allows multiple versions of a DLL to be loaded in the same process is to use an assembly that includes the DLL file in its ".manifest" file. Add the assembly to the extension module's #2 manifest (typically embedded, but can be ".2"). The system looks for the "" subdirectory in the module directory. In Windows 7+ you can also add a probing path in a config file (i.e. ".config") [1] that extends the SxS search path with up to 9 relative paths, which can be up to two levels above the module directory (i.e. "..\.."). [1]: https://docs.microsoft.com/en-us/windows/desktop/SbsCs/application-configuration-files From mingw.android at gmail.com Tue Jul 24 05:01:15 2018 From: mingw.android at gmail.com (Ray Donnelly) Date: Tue, 24 Jul 2018 10:01:15 +0100 Subject: [Python-Dev] [Windows] how to prevent the wrong version of zlib1.dll to be used by lib-dynload modules In-Reply-To: References: <20180723163112.59a2677d@dulcinea.localdomain> Message-ID: MSYS2 has two Python ports, msys2 and mingw-w64. I believe Eric was referring to the mingw-w64 one? zlib1.dll in C:\Windows\System32 is a packaging error on the part of whatever put it there and that is what needs to be fixed here. ISVs need to stop putting anything in that directory as it leads to the worst form of DLL-hell. You should try to remove that file (back it up as something may break) and just move on with your work in my opinion. On Tue, Jul 24, 2018 at 8:15 AM, eryk sun wrote: > On Mon, Jul 23, 2018 at 2:31 PM, Eric Le Lay wrote: >> >> I encountered a problem with the Windows packaging of gPodder[1] >> using msys2: > > Are you using regular Windows Python with msys2, or their custom port? > > I installed msys2 and used pacman to install Python 3.6. The msys2 > environment names libraries with an "msys-" prefix in the "/usr/bin" > directory, such as msys-python3.6m.dll, msys-readline7.dll, and > msys-z.dll (zlib). This is also the application directory of the msys2 > build of Python (i.e. "/usr/bin/python.exe"), so it's the first > directory in the default DLL search path (ahead of system directories > and PATH). Unlike Windows Python, msys2 Python does not use the > alternate search path that replaces the application directory with the > DLL directory in the search path. > > A way to implement this that allows multiple versions of a DLL to be > loaded in the same process is to use an assembly that includes the DLL > file in its ".manifest" file. Add the assembly to the > extension module's #2 manifest (typically embedded, but can be > ".2"). The system looks for the "" > subdirectory in the module directory. In Windows 7+ you can also add a > probing path in a config file (i.e. ".config") [1] that > extends the SxS search path with up to 9 relative paths, which can be > up to two levels above the module directory (i.e. "..\.."). > > [1]: https://docs.microsoft.com/en-us/windows/desktop/SbsCs/application-configuration-files > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/mingw.android%40gmail.com From contact at elelay.fr Tue Jul 24 04:24:56 2018 From: contact at elelay.fr (Eric Le Lay) Date: Tue, 24 Jul 2018 10:24:56 +0200 Subject: [Python-Dev] [Windows] how to prevent the wrong version of zlib1.dll to be used by lib-dynload modules In-Reply-To: References: <20180723163112.59a2677d@dulcinea.localdomain> Message-ID: <20180724102456.71678a1f@dulcinea.localdomain> Le Mon, 23 Jul 2018 10:47:33 -0700, Nathaniel Smith a ?crit : > On Mon, Jul 23, 2018, 08:22 Eric Le Lay wrote: > > > Hello list, > > > > I encountered a problem with the Windows packaging of gPodder[1] > > using msys2: > > > > basic libraries (zlib, openssl) depended upon by python > > platform-specific modules are loaded preferably : > > 1. from lib-dynload (where they are not) > > 2. from the Windows directory (can be any version) > > 3. from the binary directory, next to gpodder.exe (where they are) > > > > So an old zlib1.dll installed by another application in c:\Windows > > was loaded, incompatible with libpng and gPodder couldn't start. > > > > I don't know what's the best approach to solve it: > > - copy those libraries to lib\pythonxx\lib-dynload (works) > > - preload them in my main script before they are loaded by the > > module (works) > > - patch something in python (dynload_win.c ?) to search first in > > the executable directory (not tried) > > > > The way we avoid these kinds of issues on Linux is to rename included > libraries to unique names (e.g. zlib1-aef3742bc3e.dll), and patch the > extension modules to look for the new names. There's a script > (auditwheel) that does the heavy lifting. On MacOS there's a similar > script people use, called "delocate". > > Unfortunately no one has written a script like this for Windows yet. I > think it'd be neat if eventually we could consolidate all this into > auditwheel, but there aren't that many people interested in working > on it. If you wanted to help, the hardest part is already done: > https://github.com/njsmith/machomachomangler/blob/master/README.rst#pe-features > > Alternatively if you just want a hack that works quick, I'd do the > preload thing. In general you need to do this anyway, if you don't > have all your extension modules in the same directory. (Unfortunately > Windows doesn't have the equivalent of RPATH.) > > -n > > > Thanks Steve, Nathaniel, eryk for your detailed answers. I use the mingw64-python3 package in msys2: https://github.com/Alexpux/MINGW-packages/tree/master/mingw-w64-python3 It's not the same as msys2 python3 or Python for Windows. It has quite a few patches. While looking into it, I found a similar issue - on the package https://github.com/Alexpux/MINGW-packages/issues/3381 - on quodlibet (the program I adapted windows packaging from) https://github.com/quodlibet/quodlibet/issues/2817 For the moment I cowardly favor the quick hack of preloading over patching dlls or using application-configuration-files. Regarding auditwheel, isn't it only handling wheel files (*.whl), not basic dll modules like binascii-cpython-36m.dll? Thanks for the link to machomachomangler. -- Eric -------------- next part -------------- A non-text attachment was scrubbed... Name: not available Type: application/pgp-signature Size: 833 bytes Desc: Signature digitale OpenPGP URL: From arj.python at gmail.com Tue Jul 24 15:27:10 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Tue, 24 Jul 2018 23:27:10 +0400 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: <9e936880-1aaa-5b4e-b1d4-82c4a57c921b@mail.mipt.ru> Message-ID: not googler i mean google. they requested a change to a pep Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ Mauritius > As the Googler who appeared on the mailing list, I can say this was just a > coincidence. I was a bit nervous no one would respond though, given Guido's > vacation :) > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Tue Jul 24 16:32:32 2018 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 24 Jul 2018 13:32:32 -0700 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: <9e936880-1aaa-5b4e-b1d4-82c4a57c921b@mail.mipt.ru> Message-ID: On Tue, Jul 24, 2018 at 12:27 PM Abdur-Rahmaan Janhangeer < arj.python at gmail.com> wrote: > not googler i mean google. they requested a change to a pep > "They"? nah. "Google LLC" did not request anything. People who happen to be working for Google on a Google owned project asked a question seeking clarification / codification of some details. There is nothing more to read into that. Adam was merely introducing himself by stating some background info as he hasn't participated on the list much in the past. -gps | destroyer of conspiracies PS The number of core devs / committers who are Googlers has been high for well over a decade (myself included). As is true for many other large open source friendly companies as well. I believe Microsoft has the most *active* committers employed at the moment. > > Abdur-Rahmaan Janhangeer > https://github.com/Abdur-rahmaanJ > Mauritius > > As the Googler who appeared on the mailing list, I can say this was just a >> coincidence. I was a bit nervous no one would respond though, given Guido's >> vacation :) >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/greg%40krypto.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Tue Jul 24 16:45:32 2018 From: eric at trueblade.com (Eric V. Smith) Date: Tue, 24 Jul 2018 16:45:32 -0400 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: <9e936880-1aaa-5b4e-b1d4-82c4a57c921b@mail.mipt.ru> Message-ID: On 7/24/2018 4:32 PM, Gregory P. Smith wrote: > On Tue, Jul 24, 2018 at 12:27 PM Abdur-Rahmaan Janhangeer > > wrote: > > not googler i mean google. they requested a change to a pep > > > "They"?? nah.? "Google LLC" did not request anything. People who > happen to be working for Google on a Google owned project asked a > question seeking clarification / codification of some details.? There > is nothing more to read into that.? Adam was merely introducing > himself by stating some background info as he hasn't participated on > the list much in the past. > > -gps | destroyer of conspiracies > > PS The number of core devs / committers who are Googlers has been high > for well over a decade (myself included).? As is true for many other > large open source friendly companies as well.? I believe Microsoft has > the most /active/ committers employed at the moment. I think we should be concerned that fully 1/3 of True Blade's staff are core developers. Why are they so interested in Python? What nefarious end could possibly motivate them to devote so many company resources to its development? Eric VP, True Blade -------------- next part -------------- An HTML attachment was scrubbed... URL: From arj.python at gmail.com Tue Jul 24 16:52:36 2018 From: arj.python at gmail.com (Abdur-Rahmaan Janhangeer) Date: Wed, 25 Jul 2018 00:52:36 +0400 Subject: [Python-Dev] Finding Guido's replacement In-Reply-To: References: <9e936880-1aaa-5b4e-b1d4-82c4a57c921b@mail.mipt.ru> Message-ID: woops googler is a person working at google saw google-er, someone who googles Abdur-Rahmaan Janhangeer https://github.com/Abdur-rahmaanJ Mauritius -------------- next part -------------- An HTML attachment was scrubbed... URL: From erik.m.bray at gmail.com Wed Jul 25 12:13:24 2018 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 25 Jul 2018 18:13:24 +0200 Subject: [Python-Dev] Update on Cygwin support (was: Clarifying Cygwin support in CPython) In-Reply-To: References: Message-ID: On Wed, Nov 8, 2017 at 3:39 PM Erik Bray wrote: > > Hi folks, > > As some people here know I've been working off and on for a while to > improve CPython's support of Cygwin. I'm motivated in part by a need > to have software working on Python 3.x on Cygwin for the foreseeable > future, preferably with minimal graft. (As an incidental side-effect > Python's test suite--especially of system-level functionality--serves > as an interesting test suite for Cygwin itself too.) > > This is partly what motivated PEP 539 [1], although that PEP had the > advantage of benefiting other POSIX-compatible platforms as well (and > in fact was fixing an aspect of CPython that made it unfriendly to > supporting other platforms). > > As far as I can tell, the first commit to Python to add any kind of > support for Cygwin was made by Guido (committing a contributed patch) > back in 1999 [2]. Since then, bits and pieces have been added for > Cygwin's benefit over time, with varying degrees of impact in terms of > #ifdefs and the like (for the most part Cygwin does not require *much* > in the way of special support, but it does have some differences from > a "normal" POSIX-compliant platform, such as the possibility for > case-insensitive filesystems and executables that end in .exe). I > don't know whether it's ever been "officially supported" but someone > with a longer memory of the project can comment on that. I'm not sure > if it was discussed at all or not in the context of PEP 11. > > I have personally put in a fair amount of effort already in either > fixing issues on Cygwin (many of these issues also impact MinGW), or > more often than not fixing issues in the CPython test suite on > Cygwin--these are mostly tests that are broken due to invalid > assumptions about the platform (for example, that there is always a > "root" user with uid=0; this is not the case on Cygwin). In other > cases some tests need to be skipped or worked around due to > platform-specific bugs, and Cygwin is hardly the only case of this in > the test suite. > > I also have an experimental AppVeyor configuration for running the > tests on Cygwin [3], as well as an experimental buildbot (not > available on the internet, but working). These currently rely on a > custom branch that includes fixes needed for the test suite to run to > completion without crashing or hanging (e.g. > https://bugs.python.org/issue31885). It would be nice to add this as > an official buildbot, but I'm not sure if it makes sense to do that > until it's "green", or at least not crashing. I have several other > patches to the tests toward this goal, and am currently down to ~22 > tests failing. > > Before I do any more work on this, however, it would be best to once > and for all clarify the support for Cygwin in CPython, as it has never > been "officially supported" nor unsupported--this way we can avoid > having this discussion every time a patch related to Cygwin comes up. > I could provide some arguments for why I believe Cygwin should > supported, but before this gets too long I'd just like to float the > idea of having the discussion in the first place. It's also not > exactly clear to me how to meet the standards in PEP 11 for supporting > a platform--in particular it's not clear when a buildbot is considered > "stable", or how to achieve that without getting necessary fixes > merged into the main branch in the first place. > > Thanks, > Erik > > > > [1] https://www.python.org/dev/peps/pep-0539/ > [2] https://github.com/python/cpython/commit/717d1fdf2acbef5e6b47d9b4dcf48ef1829be685 > [3] https://ci.appveyor.com/project/embray/cpython Apologies for responding to a months old post, but rather than repeat myself verbatim I'll just mention that all of the above is still true and relevant, and I am still interested in getting Python somewhere closer to "stable" on Cygwin. Part of the problem with my previous approach is that I was trying to fix every last test failure before asking to add Cygwin to CPython's CI fleet. While I believe all failures *should* be fixed (or skipped as appropriate) this is not practical to do in a short amount of time, and not having CI implemented for a platform means new bugs are added faster than we can fix the existing bugs. For example, between 3.6 and 3.7 two new bugs have caused Python to be unbuildable on Cygwin: https://bugs.python.org/issue34211 https://bugs.python.org/issue34212 This is in addition to an older issue that I was hoping to have fixed in Python 3.7, as the PR was "green" for some time well before its release: https://github.com/python/cpython/pull/4348 However, I fear part of why it was never merged is that lack of CI for Cygwin, which is in turn because it was impossible to get a "stable" test pass on Cygwin in the first place--something of a catch-22. I think a new approach that might be more practical for actually getting this platform re-supported, is to go ahead and add a CI build, and just skip all known failing test modules. This is what I've done in a new PR to add a Cygwin build on AppVeyor: https://github.com/python/cpython/pull/8463 This is far from ideal of course, and should not mean the platform is "supported". But now I and others can go through and fix the remaining test failures, re-enable those modules in the CI configuration, and actually obtain some meaningful results, which will hopefully encourage the core committers to accept fixes for the platform. Once a few more major fixes are accepted (there are many for which I have fixes, but have not yet made PRs) I can also add a buildbot for Cygwin. I already had a working experimental buildbot last year, so now it's just a matter of getting a few critical issues (such as the above-mentioned build issues) fixed, plus more of the major test modules. Thanks for your consideration, E From mail at timgolden.me.uk Wed Jul 25 11:07:09 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 25 Jul 2018 16:07:09 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN Message-ID: I'm just easing back into core development work by trying to get a stable testing environment for Python development on Windows. One problem is that certain tests use support.TESTFN (a local directory constructed from the pid) for output files etc. However this can cause issues on Windows when recreating the folders / files for multiple tests, especially when running in parallel. Here's an example on my laptop deliberately running 3 tests with -j0 which I know will generate an error about one time in three: C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 test_importlib Running Debug|Win32 interpreter... Run tests in parallel using 6 child processes 0:00:23 [1/3/1] test_urllib2 failed test test_urllib2 failed -- Traceback (most recent call last): File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line 821, in test_file f = open(TESTFN, "wb") PermissionError: [Errno 13] Permission denied: '@test_15564_tmp' Although these errors are both intermittent and fairly easily spotted, the effect is that I rarely get a clean test run when I'm applying a patch. I started to address this years ago but things stalled. I'm happy to pick this up again and have another go, but I wanted to ask first whether there was any objection to my converting tests to using tempfile functions which should avoid the problem? TJG From mail at timgolden.me.uk Wed Jul 25 13:32:02 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Wed, 25 Jul 2018 18:32:02 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN Message-ID: [trying again; sorry if it shows up twice] I'm just easing back into core development work by trying to get a stable testing environment for Python development on Windows. One problem is that certain tests use support.TESTFN (a local directory constructed from the pid) for output files etc. However this can cause issues on Windows when recreating the folders / files for multiple tests, especially when running in parallel. Here's an example on my laptop deliberately running 3 tests with -j0 which I know will generate an error about one time in three: C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 test_importlib Running Debug|Win32 interpreter... Run tests in parallel using 6 child processes 0:00:23 [1/3/1] test_urllib2 failed test test_urllib2 failed -- Traceback (most recent call last): File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line 821, in test_file f = open(TESTFN, "wb") PermissionError: [Errno 13] Permission denied: '@test_15564_tmp' Although these errors are both intermittent and fairly easily spotted, the effect is that I rarely get a clean test run when I'm applying a patch. I started to address this years ago but things stalled. I'm happy to pick this up again and have another go, but I wanted to ask first whether there was any objection to my converting tests to using tempfile functions which should avoid the problem? TJG From erik.m.bray at gmail.com Wed Jul 25 07:01:46 2018 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 25 Jul 2018 13:01:46 +0200 Subject: [Python-Dev] Benchmarks why we need PEP 576/579/580 In-Reply-To: <5B535D78.5000307@UGent.be> References: <5B535D78.5000307@UGent.be> Message-ID: On Sat, Jul 21, 2018 at 6:30 PM Jeroen Demeyer wrote: > > Hello, > > I finally managed to get some real-life benchmarks for why we need a > faster C calling protocol (see PEPs 576, 579, 580). > > I focused on the Cython compilation of SageMath. By default, a function > in Cython is an instance of builtin_function_or_method (analogously, > method_descriptor for a method), which has special optimizations in the > CPython interpreter. But the option "binding=True" changes those to a > custom class which is NOT optimized. > > I ran the full SageMath testsuite several times without and with > binding=True to find out any significant differences. The most dramatic > difference is multiplication for generic matrices. More precisely, with > the following command: > > python -m timeit -s "from sage.all import MatrixSpace, GF; M = > MatrixSpace(GF(9), 200).random_element()" "M * M" > > With binding=False, I got > 10 loops, best of 3: 692 msec per loop > > With binding=True, I got > 10 loops, best of 3: 1.16 sec per loop > > This is a big regression which should be gone completely with PEP 580. > > I should mention that this was done on Python 2.7.15 (SageMath is not > yet ported to Python 3) but I see no reason why the conclusions > shouldn't be valid for newer Python versions. I used SageMath 8.3.rc1 > and Cython 0.28.4. I haven't fully caught up on the thread yet so this might already be a moot point. But just in case it isn't, the Python 3 port of Sage works well enough (at least on my branch) that the above benchmark works, and would probably be worth repeating there (it's currently Python 3.6.1, but upgrading to 3.7 probably wouldn't break the example benchmark either). From redstone-cold at 163.com Wed Jul 25 22:30:02 2018 From: redstone-cold at 163.com (Zhao Lee) Date: Thu, 26 Jul 2018 10:30:02 +0800 (CST) Subject: [Python-Dev] Fw:[issue34221] Any plans to combine collections.OrderedDict with dict Message-ID: <49e89182.76.164d46bf41e.Coremail.redstone-cold@163.com> Since Python 3.7?dicts remember the order that items were inserted, so any plans to combine collections.OrderedDict with dict? https://docs.python.org/3/library/collections.html?#collections.OrderedDict https://docs.python.org/3/library/stdtypes.html#dict BTW, I think it would be better to move "Dictionaries preserve insertion order" part at the end of the doc of https://docs.python.org/3/library/stdtypes.html#dict up below the doc for class dict(**kwarg) class dict(mapping, **kwarg) class dict(iterable, **kwarg) so that people won't miss the feature when consulting the doc _______________________________________ -------------- next part -------------- An HTML attachment was scrubbed... URL: From songofacandy at gmail.com Wed Jul 25 23:23:12 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Thu, 26 Jul 2018 12:23:12 +0900 Subject: [Python-Dev] Fw:[issue34221] Any plans to combine collections.OrderedDict with dict In-Reply-To: <49e89182.76.164d46bf41e.Coremail.redstone-cold@163.com> References: <49e89182.76.164d46bf41e.Coremail.redstone-cold@163.com> Message-ID: On Thu, Jul 26, 2018 at 12:04 PM Zhao Lee wrote: > > > Since Python 3.7?dicts remember the order that items were inserted, so any plans to combine collections.OrderedDict with dict? > https://docs.python.org/3/library/collections.html?#collections.OrderedDict > https://docs.python.org/3/library/stdtypes.html#dict No. There are some major difference. * d1 == d2 ignores order / od1 == od2 compares order * OrderedDict has move_to_end() method. * OrderedDict.pop() takes `last=True` keyword. Regards, -- INADA Naoki From raymond.hettinger at gmail.com Thu Jul 26 02:15:59 2018 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Wed, 25 Jul 2018 23:15:59 -0700 Subject: [Python-Dev] [issue34221] Any plans to combine collections.OrderedDict with dict In-Reply-To: References: <49e89182.76.164d46bf41e.Coremail.redstone-cold@163.com> Message-ID: <53227076-EE51-44C1-AC21-8AE6B87D2088@gmail.com> > On Jul 25, 2018, at 8:23 PM, INADA Naoki wrote: > > On Thu, Jul 26, 2018 at 12:04 PM Zhao Lee wrote: >> >> >> Since Python 3.7?dicts remember the order that items were inserted, so any plans to combine collections.OrderedDict with dict? >> https://docs.python.org/3/library/collections.html?#collections.OrderedDict >> https://docs.python.org/3/library/stdtypes.html#dict > > No. There are some major difference. > > * d1 == d2 ignores order / od1 == od2 compares order > * OrderedDict has move_to_end() method. > * OrderedDict.pop() takes `last=True` keyword. In addition to the API differences noted by Naoki, there are also implementation differences. The regular dict implements a low-cost solution for common cases. The OrderedDict has a more complex scheme that can handle frequent rearrangements (move_to_end operations) without touching, resizing, or reordering the underlying dictionary. Roughly speaking, regular dicts emphasize fast, space-efficient core dictionary operations over ordering requirements while OrderedDicts prioritize ordering operations over other considerations. That said, now that regular dicts are ordered by default, the need for collections.OrderedDict() should diminish quite a bit. Mostly, I think people will ignore OrderedDict unless their application heavily exercises move to end operations. Raymond From victor.stinner at gmail.com Thu Jul 26 05:22:32 2018 From: victor.stinner at gmail.com (Victor Stinner) Date: Thu, 26 Jul 2018 10:22:32 +0100 Subject: [Python-Dev] What's the status of the PEP 572 implementation? Message-ID: Hi, Is the PEP 572 implemented? If no, who is working on that? Is there a WIP pull request? An open issue? One month ago, I tried Chis Angelo's implementation but it implemented an old version of the PEP which evolved in the meanwhile. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Thu Jul 26 10:30:44 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 26 Jul 2018 07:30:44 -0700 Subject: [Python-Dev] What's the status of the PEP 572 implementation? In-Reply-To: References: Message-ID: Emily Morehouse is working on it, and I'm helping out. We probably won't get to the PR phase until the core dev sprint in September though. On Thu, Jul 26, 2018 at 2:22 AM, Victor Stinner wrote: > Hi, > > Is the PEP 572 implemented? If no, who is working on that? Is there a WIP > pull request? An open issue? > > One month ago, I tried Chis Angelo's implementation but it implemented an > old version of the PEP which evolved in the meanwhile. > > Victor > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/ > guido%40python.org > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Thu Jul 26 13:11:57 2018 From: brett at python.org (Brett Cannon) Date: Thu, 26 Jul 2018 10:11:57 -0700 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: Message-ID: On Wed, 25 Jul 2018 at 15:16 Tim Golden wrote: > I'm just easing back into core development work by trying to get a > stable testing environment for Python development on Windows. > > One problem is that certain tests use support.TESTFN (a local directory > constructed from the pid) for output files etc. However this can cause > issues on Windows when recreating the folders / files for multiple > tests, especially when running in parallel. > > Here's an example on my laptop deliberately running 3 tests with -j0 > which I know will generate an error about one time in three: > > C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 > test_importlib > > Running Debug|Win32 interpreter... > Run tests in parallel using 6 child processes > 0:00:23 [1/3/1] test_urllib2 failed > test test_urllib2 failed -- Traceback (most recent call last): > File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line > 821, in test_file > f = open(TESTFN, "wb") > PermissionError: [Errno 13] Permission denied: '@test_15564_tmp' > > Although these errors are both intermittent and fairly easily spotted, > the effect is that I rarely get a clean test run when I'm applying a patch. > > I started to address this years ago but things stalled. I'm happy to > pick this up again and have another go, but I wanted to ask first > whether there was any objection to my converting tests to using tempfile > functions which should avoid the problem? > I personally don't see any reason to block a move away from TESTFN since it obviously has some inherent issues in parallel test running which means flaky tests. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Thu Jul 26 13:23:06 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Thu, 26 Jul 2018 13:23:06 -0400 Subject: [Python-Dev] [issue34221] Any plans to combine collections.OrderedDict with dict In-Reply-To: <53227076-EE51-44C1-AC21-8AE6B87D2088@gmail.com> References: <49e89182.76.164d46bf41e.Coremail.redstone-cold@163.com> <53227076-EE51-44C1-AC21-8AE6B87D2088@gmail.com> Message-ID: On 7/26/2018 2:15 AM, Raymond Hettinger wrote: >On Jul 25, 2018, at 8:23 PM, INADA Naoki wrote: >> On Thu, Jul 26, 2018 at 12:04 PM Zhao Lee wrote: >>> Since Python 3.7?dicts remember the order that items were inserted, so any plans to combine collections.OrderedDict with dict? >>> https://docs.python.org/3/library/collections.html?#collections.OrderedDict >>> https://docs.python.org/3/library/stdtypes.html#dict >> No. There are some major difference. >> * d1 == d2 ignores order / od1 == od2 compares order >> * OrderedDict has move_to_end() method. >> * OrderedDict.pop() takes `last=True` keyword. > In addition to the API differences noted by Naoki, there are also implementation differences. The regular dict implements a low-cost solution for common cases. The OrderedDict has a more complex scheme that can handle frequent rearrangements (move_to_end operations) without touching, resizing, or reordering the underlying dictionary. Roughly speaking, regular dicts emphasize fast, space-efficient core dictionary operations over ordering requirements while OrderedDicts prioritize ordering operations over other considerations. > > That said, now that regular dicts are ordered by default, the need for collections.OrderedDict() should diminish quite a bit. Mostly, I think people will ignore OrderedDict unless their application heavily exercises move to end operations. On python-idea, Miro Hron?ok asked today whether we can change the OrderedDict repr from, for instance, OrderedDict([('a', '1'), ('b', '2')]) # to OrderedDict({'a': '1', 'b': '2'}) I am not sure what our repr change policy is, as there is a back-compatibility issue but I remember there being changes. -- Terry Jan Reedy From storchaka at gmail.com Fri Jul 27 01:05:51 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Fri, 27 Jul 2018 08:05:51 +0300 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: Message-ID: 25.07.18 18:07, Tim Golden ????: > I'm just easing back into core development work by trying to get a > stable testing environment for Python development on Windows. > > One problem is that certain tests use support.TESTFN (a local directory > constructed from the pid) for output files etc. However this can cause > issues on Windows when recreating the folders / files for multiple > tests, especially when running in parallel. > > Here's an example on my laptop deliberately running 3 tests with -j0 > which I know will generate an error about one time in three: > > C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 > test_importlib > > Running Debug|Win32 interpreter... > Run tests in parallel using 6 child processes > 0:00:23 [1/3/1] test_urllib2 failed > test test_urllib2 failed -- Traceback (most recent call last): > ? File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line > 821, in test_file > ??? f = open(TESTFN, "wb") > PermissionError: [Errno 13] Permission denied: '@test_15564_tmp' > > Although these errors are both intermittent and fairly easily spotted, > the effect is that I rarely get a clean test run when I'm applying a patch. Try to use test.support.unlink() instead of os.unlink(). It is purposed for solving this issue. > I started to address this years ago but things stalled. I'm happy to > pick this up again and have another go, but I wanted to ask first > whether there was any objection to my converting tests to using tempfile > functions which should avoid the problem? I think the only concern is that it is a pretty large change across many files, and there is a risk of introducing bugs. Also, creating a temporary file in setUp() may slowdown testing, especially if TESTFN is not needed in every test. From chris.jerdonek at gmail.com Fri Jul 27 01:25:43 2018 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Thu, 26 Jul 2018 22:25:43 -0700 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: Message-ID: On Wed, Jul 25, 2018 at 8:07 AM, Tim Golden wrote: > One problem is that certain tests use support.TESTFN (a local directory > constructed from the pid) for output files etc. However this can cause > issues on Windows when recreating the folders / files for multiple tests, > especially when running in parallel. > > Here's an example on my laptop deliberately running 3 tests with -j0 which I > know will generate an error about one time in three: > > C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 > test_importlib > > Running Debug|Win32 interpreter... > Run tests in parallel using 6 child processes > 0:00:23 [1/3/1] test_urllib2 failed > test test_urllib2 failed -- Traceback (most recent call last): > File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line 821, in > test_file > f = open(TESTFN, "wb") > PermissionError: [Errno 13] Permission denied: '@test_15564_tmp' > > Although these errors are both intermittent and fairly easily spotted, the > effect is that I rarely get a clean test run when I'm applying a patch. > > I started to address this years ago but things stalled. I'm happy to pick > this up again and have another go, but I wanted to ask first whether there > was any objection to my converting tests to using tempfile functions which > should avoid the problem? Do you know what's causing the issue on Windows? I thought TESTFN was designed to work for parallel testing, so it would surprise me if there was a problem with it. Alternatively, if TESTFN should be okay, I wonder if it's an issue with another test or tests not cleaning up after itself correctly, in which case it seems like this is an opportunity to track down and fix that issue. Switching to something else would just serve to hide / mask the issue with those other tests. --Chris From raymond.hettinger at gmail.com Fri Jul 27 02:35:42 2018 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Thu, 26 Jul 2018 23:35:42 -0700 Subject: [Python-Dev] [issue34221] Any plans to combine collections.OrderedDict with dict In-Reply-To: References: <49e89182.76.164d46bf41e.Coremail.redstone-cold@163.com> <53227076-EE51-44C1-AC21-8AE6B87D2088@gmail.com> Message-ID: > On Jul 26, 2018, at 10:23 AM, Terry Reedy wrote: > > On python-idea, Miro Hron?ok asked today whether we can change the OrderedDict repr from, for instance, > > OrderedDict([('a', '1'), ('b', '2')]) # to > OrderedDict({'a': '1', 'b': '2'}) > > I am not sure what our repr change policy is, as there is a back-compatibility issue but I remember there being changes. We are allowed to change the repr in future versions of the language. Doing so does come at a cost though. There is a small performance penalty (see the timings below). Some doctests will break. And Python 3.8 printed output in books and blog posts would get shuffled if typed in to Python 3.5 -- this is problematic because one of the few remaining use cases for OrderedDict is to write code that is compatible with older Pythons. The proposed repr does look pretty but probably isn't worth the disruption. Raymond ------------------ $ python3.7 -m timeit -r 7 'from collections import OrderedDict' "OrderedDict([('a', '1'), ('b', '2')])" 200000 loops, best of 7: 1.12 usec per loop $ python3.7 -m timeit -r 7 'from collections import OrderedDict' "OrderedDict({'a': '1', 'b': '2'})" 200000 loops, best of 7: 1.22 usec per loop $ python3.7 -m timeit -r 7 'from collections import OrderedDict' "OrderedDict([('a', '1'), ('b', '2')])" 200000 loops, best of 7: 1.13 usec per loop $ python3.7 -m timeit -r 7 'from collections import OrderedDict' "OrderedDict({'a': '1', 'b': '2'})" 200000 loops, best of 7: 1.2 usec per loop $ python3.7 -m timeit -r 7 'from collections import OrderedDict' "OrderedDict([('a', '1'), ('b', '2')])" 200000 loops, best of 7: 1.12 usec per loop $ python3.7 -m timeit -r 7 'from collections import OrderedDict' "OrderedDict({'a': '1', 'b': '2'})" 200000 loops, best of 7: 1.2 usec per loop From J.Demeyer at UGent.be Fri Jul 27 09:35:14 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Fri, 27 Jul 2018 15:35:14 +0200 Subject: [Python-Dev] PEP 576/579/580 benchmark: mistune Message-ID: <5B5B1F92.8090102@UGent.be> Hello all, since my latest benchmark for PEP 580 [1] involved SageMath, which is quite a big project, I instead propose a much simpler benchmark involving mistune. mistune [2] is a Markdown parser implemented in the Python language. It optionally allows Cython compilation. It doesn't use any kind of optimization beyond that, but I created a branch [3] to use extension types instead of Python classes. Cython can either use built-in functions/methods or a custom class (which is not optimized but which would be optimized with PEP 580). I benchmarked mistune with custom classes [3] (binding=True, the default) and with built-in functions/methods [4] (binding=False). This is the median time of 5 runs: Binding=True: 9.063s Binding=False: 8.658s So this shows again that PEP 580 improves performance in actual real-world use cases. Jeroen. [1] https://mail.python.org/pipermail/python-dev/2018-July/154740.html [2] https://github.com/lepture/mistune [3] https://github.com/jdemeyer/mistune/tree/cython_pxd [4] https://github.com/jdemeyer/mistune/tree/cython_pxd_nobinding From g.rodola at gmail.com Fri Jul 27 09:41:50 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 27 Jul 2018 15:41:50 +0200 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: Message-ID: On Thu, Jul 26, 2018 at 12:16 AM Tim Golden wrote: > > I'm just easing back into core development work by trying to get a > stable testing environment for Python development on Windows. > > One problem is that certain tests use support.TESTFN (a local directory > constructed from the pid) for output files etc. However this can cause > issues on Windows when recreating the folders / files for multiple > tests, especially when running in parallel. > > Here's an example on my laptop deliberately running 3 tests with -j0 > which I know will generate an error about one time in three: > > C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 > test_importlib > > Running Debug|Win32 interpreter... > Run tests in parallel using 6 child processes > 0:00:23 [1/3/1] test_urllib2 failed > test test_urllib2 failed -- Traceback (most recent call last): > File "C:\work-in-progress\cpython\lib\test\test_urllib2.py", line > 821, in test_file > f = open(TESTFN, "wb") > PermissionError: [Errno 13] Permission denied: '@test_15564_tmp' > > Although these errors are both intermittent and fairly easily spotted, > the effect is that I rarely get a clean test run when I'm applying a patch. > > I started to address this years ago but things stalled. I'm happy to > pick this up again and have another go, but I wanted to ask first > whether there was any objection to my converting tests to using tempfile > functions which should avoid the problem? > > TJG >From my experience open() raising PermissionDenied on Windows often means "file is already in use" as I think is this case. The same issue exists for directories: https://bugs.python.org/issue33240. Being TESTFN a global name it appears not suited for parallel testing. Windows makes this more noticeable than POSIX as it's more strict, but accessing the same file from 2 unit tests is technically a problem regardless of the platform. I don't think that means we should ditch TESTFN in favor of tempfile.mktemp() though. Instead the file cleanup functions (support.unlink() and support.rmtree()) may be more clever and (important) they should always be used in setUp / tearDown. For instance, the traceback you pasted refers to a test class which doesn't do this. In psutil I've had occasional Windows failures like this for years then I got tired of it and came up with this: https://github.com/giampaolo/psutil/blob/1b09b5fff78f705dfb42458726ff9789c26f6f21/psutil/tests/__init__.py#L686 ...which basically aggressively retries os.unlink or shutil.rmtree for 1 sec in case of (any) error, and I haven't had this problem since then. I suppose test.support's unlink() and rmtree() can do something similar, maybe just by using a better exception handling, and all unit tests should use them in setUp / tearDown. I think this will diminish the occasional failures on Windows, although not completely. -- Giampaolo - http://grodola.blogspot.com From chris.jerdonek at gmail.com Fri Jul 27 10:48:47 2018 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Fri, 27 Jul 2018 07:48:47 -0700 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: Message-ID: On Fri, Jul 27, 2018 at 6:41 AM, Giampaolo Rodola' wrote: > > Being TESTFN a global name it appears not suited for parallel testing. It was designed for parallel testing though: # Disambiguate TESTFN for parallel testing, while letting it remain a valid # module name. TESTFN = "{}_{}_tmp".format(TESTFN, os.getpid()) https://github.com/python/cpython/blob/aee632dfbb0abbc0d2bcc988c43a736afd568c55/Lib/test/support/__init__.py#L807-L809 > Windows makes this more noticeable than POSIX as it's more strict, but > accessing the same file from 2 unit tests is technically a problem > regardless of the platform. I don't think that means we should ditch > TESTFN in favor of tempfile.mktemp() though. Instead the file cleanup > functions (support.unlink() and support.rmtree()) may be more clever > and (important) they should always be used in setUp / tearDown. For > instance, the traceback you pasted refers to a test class which > doesn't do this. The "test_file" test method referenced in the traceback calls os.remove(TESTFN) in finally blocks preceding its calls to open(TESTFN, "wb"), and inspecting the method shows that it must have been able to open TESTFN earlier in the method (the same test method uses TESTFN multiple times): https://github.com/python/cpython/blob/aee632dfbb0abbc0d2bcc988c43a736afd568c55/Lib/test/test_urllib2.py#L811-L830 So I think one should investigate what can be causing the error / how it can be happening. TESTFN uses the pid of the process, so it doesn't seem like another test case could be interfering and opening the same TESTFN while the "test_file" test method is running. On Stack Overflow, there are some comments suggesting that in some cases os.remove() doesn't always complete right away (e.g. because of anti-malware software briefly holding a second reference). --Chris > > In psutil I've had occasional Windows failures like this for years > then I got tired of it and came up with this: > https://github.com/giampaolo/psutil/blob/1b09b5fff78f705dfb42458726ff9789c26f6f21/psutil/tests/__init__.py#L686 > ...which basically aggressively retries os.unlink or shutil.rmtree for > 1 sec in case of (any) error, and I haven't had this problem since > then. > > I suppose test.support's unlink() and rmtree() can do something > similar, maybe just by using a better exception handling, and all unit > tests should use them in setUp / tearDown. I think this will diminish > the occasional failures on Windows, although not completely. > > -- > Giampaolo - http://grodola.blogspot.com > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com From g.rodola at gmail.com Fri Jul 27 10:58:57 2018 From: g.rodola at gmail.com (Giampaolo Rodola') Date: Fri, 27 Jul 2018 16:58:57 +0200 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: Message-ID: On Fri, Jul 27, 2018 at 4:48 PM Chris Jerdonek wrote: > > On Fri, Jul 27, 2018 at 6:41 AM, Giampaolo Rodola' wrote: > > > > Being TESTFN a global name it appears not suited for parallel testing. > > It was designed for parallel testing though: > > # Disambiguate TESTFN for parallel testing, while letting it remain a valid > # module name. > TESTFN = "{}_{}_tmp".format(TESTFN, os.getpid()) > > https://github.com/python/cpython/blob/aee632dfbb0abbc0d2bcc988c43a736afd568c55/Lib/test/support/__init__.py#L807-L809 Oh, nice, I didn't notice that, sorry. From michael.a.wilson at atos.net Fri Jul 27 11:12:55 2018 From: michael.a.wilson at atos.net (WILSON, MICHAEL) Date: Fri, 27 Jul 2018 15:12:55 +0000 Subject: [Python-Dev] Exporting Python functions on AIX Message-ID: All, My excuse if this is not the appropriate list for a question essentially concerning the AIX port of Python. The current port of Python for AIX includes composing an export file (/lib/python2.7/config/python.exp) in which there are a number of functions starting "Py_" or "_Py_". The Vim package for AIX is built referencing the python.exp file and unfortunately, when functions are removed from libpython, even those which are not called, the vim command detects missing symbols. The most recent case (May 2017), functions _Py_hgidentity, _Py_hgversion and _Py_svnversion were replaced/removed, see "bpo-27593: Get SCM build info from git instead of hg (#1327)". Is it correct to assume that the "_Py_" functions are internal (Python name space) that should/must not be called by or made visible to application code ? Could you indicate a URL to the authoritative API documentation ? Thanks for your replies. Mike Wilson -------------- next part -------------- An HTML attachment was scrubbed... URL: From status at bugs.python.org Fri Jul 27 12:10:04 2018 From: status at bugs.python.org (Python tracker) Date: Fri, 27 Jul 2018 18:10:04 +0200 (CEST) Subject: [Python-Dev] Summary of Python tracker Issues Message-ID: <20180727161004.35F79566E7@psf.upfronthosting.co.za> ACTIVITY SUMMARY (2018-07-20 - 2018-07-27) Python tracker at https://bugs.python.org/ To view or respond to any of the issues listed below, click on the issue. Do NOT respond to this message. Issues counts and deltas: open 6740 ( +6) closed 39260 (+73) total 46000 (+79) Open issues with patches: 2681 Issues opened (50) ================== #34126: Profiling certain invalid calls crashes Python https://bugs.python.org/issue34126 reopened by vstinner #34157: NamedTemporaryFile can leave temporary files behind https://bugs.python.org/issue34157 reopened by r.david.murray #34171: Lib/trace.cover not removed by the clean target https://bugs.python.org/issue34171 opened by doko #34172: multiprocessing.Pool and ThreadPool leak resources after being https://bugs.python.org/issue34172 opened by tzickel #34173: [3.7] deadlock in /usr/lib/python3.7/concurrent/futures/thread https://bugs.python.org/issue34173 opened by corey.bryant #34176: Asyncio StreamReader fails to close Transport https://bugs.python.org/issue34176 opened by JDLH #34178: test_tcl fails on the 3.7 branch https://bugs.python.org/issue34178 opened by doko #34182: Lib/test/test_pydoc.py failed when ran as a script https://bugs.python.org/issue34182 opened by serhiy.storchaka #34185: Lib/test/test_bdb.py failed when ran as a script https://bugs.python.org/issue34185 opened by serhiy.storchaka #34187: Issues with lazy fd support in _WindowsConsoleIO fileno() and https://bugs.python.org/issue34187 opened by eryksun #34188: Allow dict choices to "transform" values in argpagse https://bugs.python.org/issue34188 opened by porton #34191: argparse: Missing subparser error message should be more clear https://bugs.python.org/issue34191 opened by porton #34192: FunctionType.__new__ can generate functions that immediately c https://bugs.python.org/issue34192 opened by bup #34193: Fix pluralization in TypeError messages in getargs.c https://bugs.python.org/issue34193 opened by xtreak #34194: test_ssl, AIX, and defaults for _ssl connections https://bugs.python.org/issue34194 opened by Michael.Felt #34198: Additional encoding options to tkinter.filedialog https://bugs.python.org/issue34198 opened by narito #34199: Add support for delete logger in log module. https://bugs.python.org/issue34199 opened by chetankolhe #34200: importlib: python -m test test_pkg -m test_7 fails randomly https://bugs.python.org/issue34200 opened by vstinner #34203: documentation: recommend Python 3 over 2 in faq https://bugs.python.org/issue34203 opened by abcdef #34204: Bump the default pickle protocol in shelve https://bugs.python.org/issue34204 opened by serhiy.storchaka #34205: Ansible: _PyImport_LoadDynamicModuleWithSpec() crash on an inv https://bugs.python.org/issue34205 opened by mdk #34206: Move and clarify Py_Main documentation https://bugs.python.org/issue34206 opened by ncoghlan #34207: test_cmd_line test_utf8_mode test_warnings fail in AMD64 FreeB https://bugs.python.org/issue34207 opened by pablogsal #34211: Cygwin build broken due to use of &PyType_Type in static decla https://bugs.python.org/issue34211 opened by erik.bray #34212: Cygwin link failure with builtin modules since issue30860 https://bugs.python.org/issue34212 opened by erik.bray #34213: Frozen dataclass __init__ fails for "object" property" https://bugs.python.org/issue34213 opened by Omenien #34214: Pylint recusion stack overflow abort https://bugs.python.org/issue34214 opened by nickdrozd #34215: streams.py:readuntil IncompleteReadError is message is incorre https://bugs.python.org/issue34215 opened by mrbell321 at gmail.com #34216: python platform no child error https://bugs.python.org/issue34216 opened by sskamble619 #34219: distutils: build_ext -D wrongly assume defining a symbol with https://bugs.python.org/issue34219 opened by monson #34222: Email message serialization enters an infinite loop when foldi https://bugs.python.org/issue34222 opened by altvod #34223: PYTHONDUMPREFS=1 ./python -c pass does crash https://bugs.python.org/issue34223 opened by vstinner #34224: python 3.7 inside venv tries to write back to read-only instal https://bugs.python.org/issue34224 opened by ajung #34226: cgi.parse_multipart() requires undocumented CONTENT-LENGTH in https://bugs.python.org/issue34226 opened by yan12125 #34231: PYTHONBREAKPOINT is not documented with python --help https://bugs.python.org/issue34231 opened by matrixise #34232: Python3.7.0 exe installers (32 and 64 bit) failing on Windows7 https://bugs.python.org/issue34232 opened by wolma #34234: Use _PyAnyInt_Check() and _PyAnyInt_CheckExact() in 2.7 https://bugs.python.org/issue34234 opened by serhiy.storchaka #34235: PyArg_ParseTupleAndKeywords: support required keyword argument https://bugs.python.org/issue34235 opened by serhiy.storchaka #34236: Test6012 in test_capi is not run as part of make test https://bugs.python.org/issue34236 opened by xtreak #34238: When BROWSER is set on Mac webbrowser.register_standard_browse https://bugs.python.org/issue34238 opened by ograycode #34239: Convert test_bz2 to use tempfile https://bugs.python.org/issue34239 opened by tim.golden #34240: Convert test_mmap to use tempfile https://bugs.python.org/issue34240 opened by tim.golden #34242: configparser: SectionProxy.get is silent on missing options https://bugs.python.org/issue34242 opened by Stig Johan Berggren #34243: pty.spawn: inconsistent interface https://bugs.python.org/issue34243 opened by statix #34244: Add support of check logger https://bugs.python.org/issue34244 opened by chetankolhe #34245: Python library should be installed writable https://bugs.python.org/issue34245 opened by jdemeyer #34246: Gentoo Refleaks 3.7: test_smtplib has dangling threads https://bugs.python.org/issue34246 opened by pablogsal #34247: PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher https://bugs.python.org/issue34247 opened by ronaldoussoren #34248: dbm errors should contain file names https://bugs.python.org/issue34248 opened by sam-s #34249: Full set of format codes applies to strftime only https://bugs.python.org/issue34249 opened by v kats Most recent 15 issues with no replies (15) ========================================== #34248: dbm errors should contain file names https://bugs.python.org/issue34248 #34247: PYTHONOPTIMZE ignored in 3.7.0 when using custom launcher https://bugs.python.org/issue34247 #34244: Add support of check logger https://bugs.python.org/issue34244 #34243: pty.spawn: inconsistent interface https://bugs.python.org/issue34243 #34236: Test6012 in test_capi is not run as part of make test https://bugs.python.org/issue34236 #34235: PyArg_ParseTupleAndKeywords: support required keyword argument https://bugs.python.org/issue34235 #34232: Python3.7.0 exe installers (32 and 64 bit) failing on Windows7 https://bugs.python.org/issue34232 #34226: cgi.parse_multipart() requires undocumented CONTENT-LENGTH in https://bugs.python.org/issue34226 #34222: Email message serialization enters an infinite loop when foldi https://bugs.python.org/issue34222 #34219: distutils: build_ext -D wrongly assume defining a symbol with https://bugs.python.org/issue34219 #34216: python platform no child error https://bugs.python.org/issue34216 #34215: streams.py:readuntil IncompleteReadError is message is incorre https://bugs.python.org/issue34215 #34214: Pylint recusion stack overflow abort https://bugs.python.org/issue34214 #34212: Cygwin link failure with builtin modules since issue30860 https://bugs.python.org/issue34212 #34211: Cygwin build broken due to use of &PyType_Type in static decla https://bugs.python.org/issue34211 Most recent 15 issues waiting for review (15) ============================================= #34249: Full set of format codes applies to strftime only https://bugs.python.org/issue34249 #34245: Python library should be installed writable https://bugs.python.org/issue34245 #34242: configparser: SectionProxy.get is silent on missing options https://bugs.python.org/issue34242 #34240: Convert test_mmap to use tempfile https://bugs.python.org/issue34240 #34239: Convert test_bz2 to use tempfile https://bugs.python.org/issue34239 #34234: Use _PyAnyInt_Check() and _PyAnyInt_CheckExact() in 2.7 https://bugs.python.org/issue34234 #34231: PYTHONBREAKPOINT is not documented with python --help https://bugs.python.org/issue34231 #34219: distutils: build_ext -D wrongly assume defining a symbol with https://bugs.python.org/issue34219 #34213: Frozen dataclass __init__ fails for "object" property" https://bugs.python.org/issue34213 #34212: Cygwin link failure with builtin modules since issue30860 https://bugs.python.org/issue34212 #34211: Cygwin build broken due to use of &PyType_Type in static decla https://bugs.python.org/issue34211 #34206: Move and clarify Py_Main documentation https://bugs.python.org/issue34206 #34198: Additional encoding options to tkinter.filedialog https://bugs.python.org/issue34198 #34193: Fix pluralization in TypeError messages in getargs.c https://bugs.python.org/issue34193 #34182: Lib/test/test_pydoc.py failed when ran as a script https://bugs.python.org/issue34182 Top 10 most discussed issues (10) ================================= #34170: Py_Initialize(): computing path configuration must not have si https://bugs.python.org/issue34170 12 msgs #34172: multiprocessing.Pool and ThreadPool leak resources after being https://bugs.python.org/issue34172 10 msgs #34115: code.InteractiveConsole.interact() closes stdin https://bugs.python.org/issue34115 7 msgs #34246: Gentoo Refleaks 3.7: test_smtplib has dangling threads https://bugs.python.org/issue34246 7 msgs #29710: Incorrect representation caveat on bitwise operation docs https://bugs.python.org/issue29710 6 msgs #34154: Tkinter __init__ documentations sometimes missing valid keywor https://bugs.python.org/issue34154 6 msgs #34173: [3.7] deadlock in /usr/lib/python3.7/concurrent/futures/thread https://bugs.python.org/issue34173 6 msgs #29750: smtplib doesn't handle unicode passwords https://bugs.python.org/issue29750 5 msgs #33089: Add multi-dimensional Euclidean distance function to the math https://bugs.python.org/issue33089 5 msgs #34178: test_tcl fails on the 3.7 branch https://bugs.python.org/issue34178 5 msgs Issues closed (68) ================== #12743: C API marshalling doc contains XXX https://bugs.python.org/issue12743 closed by berker.peksag #13041: argparse: terminal width is not detected properly https://bugs.python.org/issue13041 closed by berker.peksag #15127: Supressing warnings with -w "whether gcc supports ParseTuple" https://bugs.python.org/issue15127 closed by serhiy.storchaka #20260: Argument Clinic: add unsigned integers converters https://bugs.python.org/issue20260 closed by serhiy.storchaka #21446: Update reload fixer to use importlib instead of imp https://bugs.python.org/issue21446 closed by berker.peksag #21826: Performance issue (+fix) AIX ctypes.util with no /sbin/ldconfi https://bugs.python.org/issue21826 closed by berker.peksag #22374: Replace contextmanager example and improve explanation https://bugs.python.org/issue22374 closed by berker.peksag #23927: getargs.c skipitem() doesn't skip 'w*' https://bugs.python.org/issue23927 closed by serhiy.storchaka #24085: large memory overhead when pyc is recompiled https://bugs.python.org/issue24085 closed by inada.naoki #25094: [EASY][Windows] test_tools fails on Windows when passing https://bugs.python.org/issue25094 closed by vstinner #25156: shutil.copyfile should internally use os.sendfile when possibl https://bugs.python.org/issue25156 closed by giampaolo.rodola #25943: Integer overflow in _bsddb leads to heap corruption https://bugs.python.org/issue25943 closed by serhiy.storchaka #27717: sqlite documentation bug https://bugs.python.org/issue27717 closed by berker.peksag #28536: Show the qualified name when a call fails https://bugs.python.org/issue28536 closed by berker.peksag #28677: difficult to parse sentence structure in "When an instance att https://bugs.python.org/issue28677 closed by taleinat #30270: Remove sqlite3.Cache display method https://bugs.python.org/issue30270 closed by berker.peksag #31166: null pointer deref and segfault in _PyObject_Alloc (obmalloc.c https://bugs.python.org/issue31166 closed by inada.naoki #31935: subprocess.run() timeout not working with grandchildren and st https://bugs.python.org/issue31935 closed by martin.panter #32083: sqlite3 Cursor.description can't return column types https://bugs.python.org/issue32083 closed by berker.peksag #32663: SMTPUTF8SimTests are not actually being run https://bugs.python.org/issue32663 closed by r.david.murray #33102: get the nth folder of a given path https://bugs.python.org/issue33102 closed by taleinat #33336: [imaplib] MOVE is a legal command https://bugs.python.org/issue33336 closed by vstinner #33468: Add try-finally contextlib.contextmanager example https://bugs.python.org/issue33468 closed by taleinat #33719: Test failures on Python 3.7 beta 5 and Windows 10 https://bugs.python.org/issue33719 closed by vstinner #33720: test_marshal: crash in Python 3.7b5 on Windows 10 https://bugs.python.org/issue33720 closed by serhiy.storchaka #34008: Do we support calling Py_Main() after Py_Initialize()? https://bugs.python.org/issue34008 closed by ncoghlan #34084: possible free statically allocated string in compiler when eas https://bugs.python.org/issue34084 closed by serhiy.storchaka #34127: Gramatically incorrect error message for some calls with wrong https://bugs.python.org/issue34127 closed by rhettinger #34136: Del on class __annotations__ regressed, failing test https://bugs.python.org/issue34136 closed by serhiy.storchaka #34149: Behavior of the min/max with key=None https://bugs.python.org/issue34149 closed by rhettinger #34159: asyncio basic event loop stuck with no tasks scheduled or read https://bugs.python.org/issue34159 closed by Sheng Zhong #34161: (good first issue) Tutorial 7.1 str.format() code example synt https://bugs.python.org/issue34161 closed by Mariatta #34163: Python latest release 2.7 shows SSL error https://bugs.python.org/issue34163 closed by brett.cannon #34164: base64.b32decode() leads into UnboundLocalError and OverflowEr https://bugs.python.org/issue34164 closed by serhiy.storchaka #34166: Tools/msgfmt.py emits a DeprecationWarning under Python 3.7 https://bugs.python.org/issue34166 closed by serhiy.storchaka #34167: Standard library docs: prev/next skip right over 16.11 https://bugs.python.org/issue34167 closed by inada.naoki #34169: itertools.repeat does not accept None as an explicit count https://bugs.python.org/issue34169 closed by rhettinger #34174: argparse formatter_class issue for RawDescriptionHelpFormatter https://bugs.python.org/issue34174 closed by Mariatta #34175: typing.NamedTuple: type-checking error when "index" used as me https://bugs.python.org/issue34175 closed by gvanrossum #34177: test_site fails in macOS-PR VSTS builds for 3.7 branch https://bugs.python.org/issue34177 closed by inada.naoki #34179: test_statistics fails after test_time https://bugs.python.org/issue34179 closed by skrah #34180: bool(Q) always return True for a priority queue Q https://bugs.python.org/issue34180 closed by rhettinger #34181: Lib/test/test_typing.py failed when ran as a script https://bugs.python.org/issue34181 closed by serhiy.storchaka #34183: Lib/test/test_contextlib_async.py failed when ran as a script https://bugs.python.org/issue34183 closed by serhiy.storchaka #34184: Lib/test/test_dataclasses.py failed when ran as a script https://bugs.python.org/issue34184 closed by serhiy.storchaka #34186: [3.6.6 on macOS] os.listdir replacing "/" with ":" https://bugs.python.org/issue34186 closed by zach.ware #34189: Add simple tests for new Tk widget options https://bugs.python.org/issue34189 closed by serhiy.storchaka #34190: x86 Gentoo Refleaks 3.x: test_sys_setprofile leaked [1, 1, 1] https://bugs.python.org/issue34190 closed by vstinner #34195: test_nt_helpers fails with case difference in drive letter https://bugs.python.org/issue34195 closed by tim.golden #34196: @staticmethod mangles name of outside class with __ (it is not https://bugs.python.org/issue34196 closed by rhettinger #34197: Make csv.Dialect attributes skipinitialspace, doublequote and https://bugs.python.org/issue34197 closed by serhiy.storchaka #34201: Make ndarray.readonly a bool https://bugs.python.org/issue34201 closed by serhiy.storchaka #34202: 3.6 ZipFile fails with Path https://bugs.python.org/issue34202 closed by berker.peksag #34208: Change in 3.7 expression evaluation? https://bugs.python.org/issue34208 closed by eryksun #34209: racecondition https://bugs.python.org/issue34209 closed by inada.naoki #34210: Small improvements in heapq (refactoring) https://bugs.python.org/issue34210 closed by rhettinger #34217: windows: cross compilation fails due to headers with uppercase https://bugs.python.org/issue34217 closed by inada.naoki #34218: test_xml_etree_c leaked [68, 68, 74] references, sum=210 https://bugs.python.org/issue34218 closed by serhiy.storchaka #34220: Serialization of email message without header line length limi https://bugs.python.org/issue34220 closed by r.david.murray #34221: Any plans to combine collections.OrderedDict with dict https://bugs.python.org/issue34221 closed by rhettinger #34225: distutils._msvccompiler should trim INCLUDE/LIB directories https://bugs.python.org/issue34225 closed by steve.dower #34227: Weighted random.sample() (weighted sampling without replacemen https://bugs.python.org/issue34227 closed by rhettinger #34228: Allow PYTHONTRACEMALLOC=0 and -X tracemalloc=0 to disable expl https://bugs.python.org/issue34228 closed by vstinner #34229: Possible access to unintended variable in "cpython/Objects/sli https://bugs.python.org/issue34229 closed by serhiy.storchaka #34230: crashes in pysqlite_build_row_cast_map() on memory allocation https://bugs.python.org/issue34230 closed by serhiy.storchaka #34233: subprocess.check_output goes to infinte loop https://bugs.python.org/issue34233 closed by eryksun #34237: faq/design: PEP 572 adds assignment expressions https://bugs.python.org/issue34237 closed by ammar2 #34241: Segfault while Django template rendering https://bugs.python.org/issue34241 closed by vhelke From vstinner at redhat.com Fri Jul 27 12:42:55 2018 From: vstinner at redhat.com (Victor Stinner) Date: Fri, 27 Jul 2018 18:42:55 +0200 Subject: [Python-Dev] Exporting Python functions on AIX In-Reply-To: References: Message-ID: Yes, all symbols starting with _Py are private and must not be used outside CPython internals. Victor Le vendredi 27 juillet 2018, WILSON, MICHAEL a ?crit : > All, > > My excuse if this is not the appropriate list for a question essentially concerning the AIX port of Python. > > The current port of Python for AIX includes composing an export file (/lib/python2.7/config/python.exp) in which there are a number of functions starting ?Py_? or ?_Py_?. > > The Vim package for AIX is built referencing the python.exp file and unfortunately, when functions are removed from libpython, even those which are not called, the vim command detects missing symbols. > > The most recent case (May 2017), functions _Py_hgidentity, _Py_hgversion and _Py_svnversion were replaced/removed, see ?bpo-27593: Get SCM build info from git instead of hg (#1327)?. > > Is it correct to assume that the ?_Py_? functions are internal (Python name space) that should/must not be called by or made visible to application code ? > > Could you indicate a URL to the authoritative API documentation ? > > Thanks for your replies. > > Mike Wilson > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robb at datalogics.com Fri Jul 27 15:23:01 2018 From: robb at datalogics.com (Rob Boehne) Date: Fri, 27 Jul 2018 19:23:01 +0000 Subject: [Python-Dev] [DLFILTER] Exporting Python functions on AIX In-Reply-To: References: Message-ID: <54CCBECB-00A1-4CF8-8F9E-0CEC524D59D7@datalogics.com> Why would a VIM build refer to the export file for python? From: Python-Dev on behalf of "WILSON, MICHAEL" Date: Friday, July 27, 2018 at 10:27 AM To: "python-dev at python.org" Cc: "WILSON, MICHAEL" Subject: [DLFILTER] [Python-Dev] Exporting Python functions on AIX All, My excuse if this is not the appropriate list for a question essentially concerning the AIX port of Python. The current port of Python for AIX includes composing an export file (/lib/python2.7/config/python.exp) in which there are a number of functions starting ?Py_? or ?_Py_?. The Vim package for AIX is built referencing the python.exp file and unfortunately, when functions are removed from libpython, even those which are not called, the vim command detects missing symbols. The most recent case (May 2017), functions _Py_hgidentity, _Py_hgversion and _Py_svnversion were replaced/removed, see ?bpo-27593: Get SCM build info from git instead of hg (#1327)?. Is it correct to assume that the ?_Py_? functions are internal (Python name space) that should/must not be called by or made visible to application code ? Could you indicate a URL to the authoritative API documentation ? Thanks for your replies. Mike Wilson -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Jul 27 15:56:33 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 27 Jul 2018 20:56:33 +0100 Subject: [Python-Dev] [DLFILTER] Exporting Python functions on AIX In-Reply-To: <54CCBECB-00A1-4CF8-8F9E-0CEC524D59D7@datalogics.com> References: <54CCBECB-00A1-4CF8-8F9E-0CEC524D59D7@datalogics.com> Message-ID: On 27 July 2018 at 20:23, Rob Boehne wrote: > Why would a VIM build refer to the export file for python? Because vim includes an optional embedded Python interpreter. Paul From mail at timgolden.me.uk Sat Jul 28 10:38:23 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Sat, 28 Jul 2018 15:38:23 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: Message-ID: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> On 25/07/2018 16:07, Tim Golden wrote: > One problem is that certain tests use support.TESTFN (a local directory > constructed from the pid) for output files etc. However this can cause > issues on Windows when recreating the folders / files for multiple > tests, especially when running in parallel. > > Here's an example on my laptop deliberately running 3 tests with -j0 > which I know will generate an error about one time in three: > > C:\work-in-progress\cpython>python -mtest -j0 test_urllib2 test_bz2 > test_importlib > [... snip ...] > [...] I wanted to ask first > whether there was any objection to my converting tests to using tempfile > functions which should avoid the problem? Thanks to those who responded here and/or on the two bpo issues I've already raised: https://bugs.python.org/issue34239 -- test_bz2 https://bugs.python.org/issue34240 -- test_mmap Two key questions have been posed in different ways: 1) Why are these errors occurring? ie are we dodging a root cause issue 2) Isn't this what test.support.unlink is there to solve? For (1) I'm putting together a test run using code which I originally wrote for https://bugs.python.org/issue7443 to force the issues out into the open. For (2), yes: test.support.unlink is supposed to solve that. But it's either not doing enough retries etc. or it's missing a trick. To be clear: my motivation here isn't some housekeeping or modernisation exercise. I want to get a clear test run on a fresh build on Windows. At present I never get that. I'd be interested to hear from other Windows devs whether they have the same experience? TJG From ronaldoussoren at mac.com Sat Jul 28 10:50:17 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Sat, 28 Jul 2018 15:50:17 +0100 Subject: [Python-Dev] USE_STACKCHECK and running out of stack Message-ID: <9EE08C5E-1ACC-45D3-8894-1E01BF57F4BF@mac.com> Hi, I?m looking at PyOS_CheckStack because this feature might be useful on macOS (and when I created bpo-33955 for this someone ran with it and created a patch). Does anyone remember why the interpreter raises MemoryError and not RecursionError when PyOS_CheckStack detects that we?re about to run out of stack space? The reason I?m looking into this is that the default stack size on macOS is fairly small and I?d like to avoid crashing the interpreter when running out of stackspace on threads created by the system (this is less of a risk on threads created by Python itself because we can arrange for a large enough stack in that case). Ronald From jeremy.kloth at gmail.com Sat Jul 28 12:27:24 2018 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Sat, 28 Jul 2018 10:27:24 -0600 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018 at 8:41 AM Tim Golden wrote: > 1) Why are these errors occurring? ie are we dodging a root cause issue The root cause is how Windows handles file deletions. When a file is removed, it is not immediately removed from the directory, instead, it is simply marked for deletion. Only once all open handles to the file are closed, is it removed from the directory. The most common cause of additional open handles to a file is a antivirus scanner. > 2) Isn't this what test.support.unlink is there to solve? Yes. If test.support.unlink doesn't succeed in removing the file, a RuntimeWarning will be emitted. > For (1) I'm putting together a test run using code which I originally > wrote for https://bugs.python.org/issue7443 to force the issues out into > the open. These intermittent errors are tough as it is really just a matter of timing. The faster the disk and the more loaded the CPU, the more often these can occur, however. > For (2), yes: test.support.unlink is supposed to solve that. But it's > either not doing enough retries etc. or it's missing a trick. If you are not seeing the RuntimeWarnings, then something else is amiss. -- Jeremy Kloth From mail at timgolden.me.uk Sat Jul 28 13:20:18 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Sat, 28 Jul 2018 18:20:18 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> Message-ID: <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> On 28/07/2018 17:27, Jeremy Kloth wrote: > On Sat, Jul 28, 2018 at 8:41 AM Tim Golden wrote: >> 1) Why are these errors occurring? ie are we dodging a root cause issue > > The root cause is how Windows handles file deletions. When a file is > removed, it is not immediately removed from the directory, instead, it > is simply marked for deletion. Only once all open handles to the file > are closed, is it removed from the directory. The most common cause > of additional open handles to a file is a antivirus scanner. Thanks, Jeremy. I'm already aware of that. (If you look back at https://bugs.python.org/issue7443 you'll find me explaining the same to MvL some years ago). [In case the tone isn't clear, there's absolutely no sarcasm implied here. I just wanted to make it clear that I'm at least partly au fait with the situation :)]. Although things have moved on since that discussion and test.support.unlink has grown some extra legs, all it's done really is to push the bump along the carpet for a bit. I've got a newly-installed Win10 machine with the typical MS Antivirus & TortoiseGitCache vying for locks with every other process. I've just done a full test run: python -mtest -j0 -v > test.log and I've got a mixture of Permission (winerror 13) & Access errors (winerror 5). The former are usually attempting to open a TESTFN file; the latter are attempting to unlink one. Most of the Error 5 are os.unlink, but at least one is from test.support.unlink. I understand the unable-to-recreate (virus checkers with share-delete handles) but I'm not so clear on what's blocking the unlink. IOW I think we have more than one issue. Here's the thing. The TESTFN approach creates a directory per process test_python_ and some variant of @test__tmp inside that directory. But the same filename in the same directory is used for every test in that process. Now, leaving aside the particular mechanism by which Windows processes might be holding locks which prevent removal or re-addition, that already seems like an odd choice. I think that was my starting point: rather than develop increasingly involved and still somewhat brittle mechanisms, why not do what you'd naturally do with a new test and use tempfile? I was expecting someone to come forward to highlight some particular reason why the TESTFN approach is superior, but apart from a reference to the possibly cost of creating a new temporary file per test, no-one really has. > If you are not seeing the RuntimeWarnings, then something else is amiss. I think I've seen one RuntimeWarning in the many, many times I've been running through tests today. TJG From jeremy.kloth at gmail.com Sat Jul 28 17:17:50 2018 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Sat, 28 Jul 2018 15:17:50 -0600 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018 at 11:20 AM Tim Golden wrote: > Although things have moved on since that discussion and > test.support.unlink has grown some extra legs, all it's done really is > to push the bump along the carpet for a bit. I've got a newly-installed > Win10 machine with the typical MS Antivirus & TortoiseGitCache vying for > locks with every other process. I've just done a full test run: > > python -mtest -j0 -v > test.log I, for one, would like to see that log. The issues you are have are fairly unique. Just check out the buildbot status page. I know that some of the workers are somewhat limited, but my worker (https://buildbot.python.org/all/#/workers/12) is running on dedicated hardware. Before https://bugs.python.org/issue15496 was applied, the errors you describe were indeed happening, but no longer. > Here's the thing. The TESTFN approach creates a directory per process > test_python_ and some variant of @test__tmp inside that > directory. But the same filename in the same directory is used for every > test in that process. Now, leaving aside the particular mechanism by > which Windows processes might be holding locks which prevent removal or > re-addition, that already seems like an odd choice. Well, since every test (well, test file) is run in its own process, a directory per process doesn't seem that odd. You seem to be focusing on 2 tests in particular, both of which have not been converted to use test.support.unlink for removing. The "trick" with test.support.unlink now is that it waits until the file has truly removed from the directory before continuing. Using this, in turn, would mean that following create *should* succeed without error. If test.support.unlink returns without warning, the removed file is really gone. > I think that was my starting point: rather than develop increasingly > involved and still somewhat brittle mechanisms, why not do what you'd > naturally do with a new test and use tempfile? I was expecting someone > to come forward to highlight some particular reason why the TESTFN > approach is superior, but apart from a reference to the possibly cost of > creating a new temporary file per test, no-one really has. *PLEASE*, don't use tempfile to create files/directories in tests. It is unfriendly to (Windows) buildbots. The current approach of directory-per-process ensures no test turds are left behind, whereas the tempfile solution slowly fills up my buildbot. Windows doesn't natively clean out the temp directory. Additionally, I'm working on a solution to allow the test machinery to use a given directory for test outputs to allow for a RAM-backed filesystem for temporary files. With this, I've had the full test suite (with -uall including bigmem) take ~5min. Having some tests use the Windows default test directory would break this. It is simply not feasible to have the entire Windows TEMP in RAM, but just Python's test suite usage is small enough (<6Gb). Another point, some tests require that the temporary filename resides on the same drive as the source directory (discovered in developing the above). This means that, Windows development would be restricted to the same drive as the user directory, if the per-directory approach isn't used. -- Jeremy Kloth From chris.jerdonek at gmail.com Sat Jul 28 17:34:13 2018 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 28 Jul 2018 14:34:13 -0700 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018 at 10:20 AM, Tim Golden wrote: > > Here's the thing. The TESTFN approach creates a directory per process > test_python_ and some variant of @test__tmp inside that directory. I filed an issue some years back about this (still open): https://bugs.python.org/issue15305 The pid is unnecessarily being used twice. Once the directory is created for the process, there shouldn't be a need to disambiguate within the directory further. > But the same filename in the same directory is used for every test in that > process. Now, leaving aside the particular mechanism by which Windows > processes might be holding locks which prevent removal or re-addition, that > already seems like an odd choice. > > I think that was my starting point: rather than develop increasingly > involved and still somewhat brittle mechanisms, why not do what you'd > naturally do with a new test and use tempfile? I was expecting someone to > come forward to highlight some particular reason why the TESTFN approach is > superior, but apart from a reference to the possibly cost of creating a new > temporary file per test, no-one really has. I think there is value in having the files used during test runs in a known location. How about a middle ground where, once the process-specific directory is created in a known location, unique files are created within that directory (e.g. using a random suffix, or a perhaps a suffix that is a function of the test name / test id)? This would address both the issue you're experiencing, and, if the directory is deleted at the end of the test run, it would address the issue Jeremy mentions of growing leftover files. There could also be a check at the end of each test run making sure that the directory is empty (to check for tests that aren't cleaning up after themselves). --Chris From eryksun at gmail.com Sat Jul 28 18:02:20 2018 From: eryksun at gmail.com (eryk sun) Date: Sat, 28 Jul 2018 22:02:20 +0000 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018 at 5:20 PM, Tim Golden wrote: > > I've got a mixture of Permission (winerror 13) & Access errors (winerror 5) EACCES (13) is a CRT errno value. Python raises PermissionError for EACCES and EPERM (1, not used). It also does the reverse mapping for WinAPI calls, so PermissionError is raised either way. About 25 WinAPI error codes map to EACCES. Commonly it's due to either ERROR_ACCESS_DENIED (5) or ERROR_SHARING_VIOLATION (32). open() uses read-write sharing but not delete sharing. In this case trying to either delete an already open file or open a file that's already open with delete access (e.g. an O_TEMPORARY open) both fail with a sharing violation. An access-denied error could be due to a range of causes. Over 20 NTAPI status codes map to ERROR_ACCESS_DENIED. Commonly for a file it's due to one of the following status codes: STATUS_ACCESS_DENIED (0xc0000022) The file security doesn't grant the requested access to the caller. STATUS_DELETE_PENDING (0xc0000056) The file's delete disposition is set, i.e. it's flagged to be deleted when the last handle is closed. Opening a new handle is disallowed for any access. STATUS_FILE_IS_A_DIRECTORY (0xc00000ba) Except when using backup semantics, CreateFile calls NtCreateFile with the flag FILE_NON_DIRECTORY_FILE, so only non-directory files/devices can be opened. STATUS_CANNOT_DELETE (0xc0000121) The file is either readonly or memory mapped. From eryksun at gmail.com Sat Jul 28 18:12:25 2018 From: eryksun at gmail.com (eryk sun) Date: Sat, 28 Jul 2018 22:12:25 +0000 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018 at 9:17 PM, Jeremy Kloth wrote: > > *PLEASE*, don't use tempfile to create files/directories in tests. It > is unfriendly to (Windows) buildbots. The current approach of > directory-per-process ensures no test turds are left behind, whereas > the tempfile solution slowly fills up my buildbot. Windows doesn't > natively clean out the temp directory. FYI, Windows 10 storage sense (under system->storage) can be configured to delete temporary files on a schedule. Of course that doesn't help with older systems. From chris.jerdonek at gmail.com Sat Jul 28 18:54:41 2018 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 28 Jul 2018 15:54:41 -0700 Subject: [Python-Dev] [Python-checkins] bpo-34239: Convert test_bz2 to use tempfile (#8485) In-Reply-To: <41c4PF3BG6zFqv9@mail.python.org> References: <41c4PF3BG6zFqv9@mail.python.org> Message-ID: On Thu, Jul 26, 2018 at 2:05 PM, Tim Golden wrote: > https://github.com/python/cpython/commit/6a62e1d365934de82ff7c634981b3fbf218b4d5f > commit: 6a62e1d365934de82ff7c634981b3fbf218b4d5f > branch: master > author: Tim Golden > committer: GitHub > date: 2018-07-26T22:05:00+01:00 > summary: > > bpo-34239: Convert test_bz2 to use tempfile (#8485) > > * bpo-34239: Convert test_bz2 to use tempfile > > test_bz2 currently uses the test.support.TESTFN functionality which creates a temporary file local to the test directory named around the pid. > > This can give rise to race conditions where tests are competing with each other to delete and recreate the file. Per the other thread-- https://mail.python.org/pipermail/python-dev/2018-July/154762.html this seems like a wrong statement of the problem as tests are properly cleaning up after themselves. The leading hypothesis is that unrelated Windows processes are delaying the deletion (e.g. virus scanners). --Chris > > This change converts the tests to use tempfile.mkstemp which gives a different file every time from the system's temp area > > files: > M Lib/test/test_bz2.py > > diff --git a/Lib/test/test_bz2.py b/Lib/test/test_bz2.py > index 003497f28b16..e62729a5a2f8 100644 > --- a/Lib/test/test_bz2.py > +++ b/Lib/test/test_bz2.py > @@ -6,6 +6,7 @@ > import os > import pickle > import glob > +import tempfile > import pathlib > import random > import shutil > @@ -76,11 +77,14 @@ class BaseTest(unittest.TestCase): > BIG_DATA = bz2.compress(BIG_TEXT, compresslevel=1) > > def setUp(self): > - self.filename = support.TESTFN > + fd, self.filename = tempfile.mkstemp() > + os.close(fd) > > def tearDown(self): > - if os.path.isfile(self.filename): > + try: > os.unlink(self.filename) > + except FileNotFoundError: > + pass > > > class BZ2FileTest(BaseTest): > > _______________________________________________ > Python-checkins mailing list > Python-checkins at python.org > https://mail.python.org/mailman/listinfo/python-checkins From brett at python.org Sat Jul 28 20:40:11 2018 From: brett at python.org (Brett Cannon) Date: Sat, 28 Jul 2018 17:40:11 -0700 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018, 15:13 eryk sun, wrote: > On Sat, Jul 28, 2018 at 9:17 PM, Jeremy Kloth > wrote: > > > > *PLEASE*, don't use tempfile to create files/directories in tests. It > > is unfriendly to (Windows) buildbots. The current approach of > > directory-per-process ensures no test turds are left behind, whereas > > the tempfile solution slowly fills up my buildbot. Windows doesn't > > natively clean out the temp directory. > > FYI, Windows 10 storage sense (under system->storage) can be > configured to delete temporary files on a schedule. Of course that > doesn't help with older systems. > If Windows doesn't clean up its temp directory on a regular basis then that doesn't suggest to me not to use tempfile, but instead that the use of tempfile still needs to clean up after itself. And if there is a lacking feature in tempfile then we should add it instead of a avoiding the module. -Brett _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy.kloth at gmail.com Sat Jul 28 21:04:07 2018 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Sat, 28 Jul 2018 19:04:07 -0600 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018 at 6:43 PM Brett Cannon wrote: > If Windows doesn't clean up its temp directory on a regular basis then that doesn't suggest to me not to use tempfile, but instead that the use of tempfile still needs to clean up after itself. And if there is a lacking feature in tempfile then we should add it instead of a avoiding the module. Mind you, this is mentioned in the confines of the test harness where just about anything can happen (and usually does!). Something that cannot be coded against using just tempfile is cleanup on process abort. The per-process-directory approach handles this case. I would think it is desired to have no leftovers after running the test harness (especially in regards to the buildbots). Now, I'm not sure the exact cause of all of the leftovers in the TEMP directory, but it is definitely something that is currently happening (and shouldn't be). It is not exactly the easiest of tasks to track the file usage of every test in the test suite. It is certainly easier to replace usages of os.unlink with test.support.unlink within the test suite. -- Jeremy Kloth From chris.jerdonek at gmail.com Sat Jul 28 21:12:35 2018 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Sat, 28 Jul 2018 18:12:35 -0700 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018 at 5:40 PM Brett Cannon wrote: > > On Sat, Jul 28, 2018, 15:13 eryk sun, wrote: > >> On Sat, Jul 28, 2018 at 9:17 PM, Jeremy Kloth >> wrote: >> > >> > *PLEASE*, don't use tempfile to create files/directories in tests. It >> > is unfriendly to (Windows) buildbots. The current approach of >> > directory-per-process ensures no test turds are left behind, whereas >> > the tempfile solution slowly fills up my buildbot. Windows doesn't >> > natively clean out the temp directory. >> >> FYI, Windows 10 storage sense (under system->storage) can be >> configured to delete temporary files on a schedule. Of course that >> doesn't help with older systems. >> > > If Windows doesn't clean up its temp directory on a regular basis then > that doesn't suggest to me not to use tempfile, but instead that the use of > tempfile still needs to clean up after itself. And if there is a lacking > feature in tempfile then we should add it instead of a avoiding the module. > Regardless of whether the tempfile or TESTFN approach is used, I think it would be best for a few reasons if the choice is abstracted behind a uniquely named test function (e.g. make_test_file if not already used). ?Chris > -Brett > > _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> > Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/brett%40python.org >> > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jeremy.kloth at gmail.com Sat Jul 28 21:57:06 2018 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Sat, 28 Jul 2018 19:57:06 -0600 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sat, Jul 28, 2018 at 7:15 PM Chris Jerdonek wrote: > Regardless of whether the tempfile or TESTFN approach is used, I think it would be best for a few reasons if the choice is abstracted behind a uniquely named test function (e.g. make_test_file if not already used). +1, although my particular choice of color would be to add a pair of functions, mkstemp and mkdtemp, to match the style of test.support-wrapped library functions for use in the test harness. -- Jeremy Kloth From mail at timgolden.me.uk Sun Jul 29 02:04:48 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 29 Jul 2018 07:04:48 +0100 Subject: [Python-Dev] [Python-checkins] bpo-34239: Convert test_bz2 to use tempfile (#8485) In-Reply-To: References: <41c4PF3BG6zFqv9@mail.python.org> Message-ID: <81e58544-da2b-af27-49d8-aa7f9b10f82c@timgolden.me.uk> On 28/07/2018 23:54, Chris Jerdonek wrote: > On Thu, Jul 26, 2018 at 2:05 PM, Tim Golden wrote: >> https://github.com/python/cpython/commit/6a62e1d365934de82ff7c634981b3fbf218b4d5f >> commit: 6a62e1d365934de82ff7c634981b3fbf218b4d5f >> branch: master >> author: Tim Golden >> committer: GitHub >> date: 2018-07-26T22:05:00+01:00 >> summary: >> >> bpo-34239: Convert test_bz2 to use tempfile (#8485) >> >> * bpo-34239: Convert test_bz2 to use tempfile >> >> test_bz2 currently uses the test.support.TESTFN functionality which creates a temporary file local to the test directory named around the pid. >> >> This can give rise to race conditions where tests are competing with each other to delete and recreate the file. > > Per the other thread-- > https://mail.python.org/pipermail/python-dev/2018-July/154762.html > this seems like a wrong statement of the problem as tests are properly > cleaning up after themselves. The leading hypothesis is that unrelated > Windows processes are delaying the deletion (e.g. virus scanners). Thanks, Chris. I'm tracking the other thread and, if needs be, I'll revert this change in favour of whatever other approach is preferred TJG From mail at timgolden.me.uk Sun Jul 29 04:58:15 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 29 Jul 2018 09:58:15 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On 29/07/2018 02:04, Jeremy Kloth wrote: > On Sat, Jul 28, 2018 at 6:43 PM Brett Cannon > wrote: >> If Windows doesn't clean up its temp directory on a regular basis >> then that doesn't suggest to me not to use tempfile, but instead >> that the use of tempfile still needs to clean up after itself. And >> if there is a lacking feature in tempfile then we should add it >> instead of a avoiding the module. > > Mind you, this is mentioned in the confines of the test harness > where just about anything can happen (and usually does!). Something > that cannot be coded against using just tempfile is cleanup on > process abort. The per-process-directory approach handles this > case. > > I would think it is desired to have no leftovers after running the > test harness (especially in regards to the buildbots). > > Now, I'm not sure the exact cause of all of the leftovers in the > TEMP directory, but it is definitely something that is currently > happening (and shouldn't be). It is not exactly the easiest of tasks > to track the file usage of every test in the test suite. It is > certainly easier to replace usages of os.unlink with > test.support.unlink within the test suite. In the interests of trying to keep a focus to the changes I'm making, I propose to start again by, as you suggest, making use of test.support.unlink where it's not currently used. From the evidence I don't believe that will solve every problem I'm seeing but it should certainly reduce them. I do there there's mileage in a wider change to revamp the test suite's naming and cleanup of temporary files but I'm very wary of trying to undertake what would undoubtedly be a sprawling and probably contentious change. TJG From mail at timgolden.me.uk Sun Jul 29 05:13:38 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 29 Jul 2018 10:13:38 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On 28/07/2018 22:17, Jeremy Kloth wrote: > On Sat, Jul 28, 2018 at 11:20 AM Tim Golden wrote: >> Although things have moved on since that discussion and >> test.support.unlink has grown some extra legs, all it's done really is >> to push the bump along the carpet for a bit. I've got a newly-installed >> Win10 machine with the typical MS Antivirus & TortoiseGitCache vying for >> locks with every other process. I've just done a full test run: >> >> python -mtest -j0 -v > test.log > > I, for one, would like to see that log. The issues you are have are > fairly unique. Just check out the buildbot status page. I know that > some of the workers are somewhat limited, but my worker > (https://buildbot.python.org/all/#/workers/12) is running on dedicated > hardware. Before https://bugs.python.org/issue15496 was applied, the > errors you describe were indeed happening, but no longer. For an example: http://tjg.org.uk/test.log Thinkpad T420, 4Gb, i5, SSD Recently rebuilt and reinstalled: Win10, VS2017, TortoiseGit, standard Windows Antimalware, usual developer tools. That particular run was done with the laptop unattended (ie nothing else going on at the front end). But the problem is certainly not specific to this laptop. TJG From storchaka at gmail.com Sun Jul 29 07:53:10 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 29 Jul 2018 14:53:10 +0300 Subject: [Python-Dev] Testing C API Message-ID: Currently C API is not completely covered by tests. Tests for particular parts of C API are scattered through different files. There are files completely purposed for testing C API (like test_capi.py, test_getargs2.py), there are classes (usually having "CAPI" in the name) in different files for testing C API specific for unicode, marshal. Argument parsing tests are split between two files, test_capi.py, test_getargs2.py. I need to add new tests for new features, and I'm going to add new tests for existing C API. But first I'm going to reorganize tests. Add a new directory Lib/test/test_capi, and move all C API tests into it, grouped by function prefixes. test_getargs.py for testing PyArg_*(), test_unicode.py for testing PyUnicode_*(), etc. Tests that use the _testcapi module, but don't test specific C API, will left on place. The benefit is that it will be easier to run all C API tests at once, and only them, and it will be clearer what C API is covered by tests. The disadvantage is that you will need to run several files for testing marshal for example. What are your thoughts? From steve.dower at python.org Sun Jul 29 08:35:35 2018 From: steve.dower at python.org (Steve Dower) Date: Sun, 29 Jul 2018 13:35:35 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: <1ddc7783-6516-c70a-0825-e8e0ce847dd0@python.org> On 29Jul2018 0958, Tim Golden wrote: > In the interests of trying to keep a focus to the changes I'm making, I > propose to start again by, as you suggest, making use of > test.support.unlink where it's not currently used. From the evidence I > don't believe that will solve every problem I'm seeing but it should > certainly reduce them. One additional thing that may help (if support.unlink doesn't already do it) is to rename the file before deleting it. Renames are always possible even with open handles, and then you can create a new file at the original name. Cheers, Steve From steve.dower at python.org Sun Jul 29 08:39:32 2018 From: steve.dower at python.org (Steve Dower) Date: Sun, 29 Jul 2018 13:39:32 +0100 Subject: [Python-Dev] Testing C API In-Reply-To: References: Message-ID: On 29Jul2018 1253, Serhiy Storchaka wrote: > The benefit is that it will be easier to run all C API tests at once, > and only them, and it will be clearer what C API is covered by tests. > The disadvantage is that you will need to run several files for testing > marshal for example. Can we make the regular tests import and also run the related C API tests? So that a normal run wouldn't normally include the entire C API test directory, but would include test classes in the related Python test modules? (Maybe there's a way to decorate the test classes for this?) I agree with the intent, but also think that's quite a disadvantage. It would be good to avoid it. Cheers, Steve From storchaka at gmail.com Sun Jul 29 09:43:46 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 29 Jul 2018 16:43:46 +0300 Subject: [Python-Dev] Testing C API In-Reply-To: References: Message-ID: 29.07.18 15:39, Steve Dower ????: > On 29Jul2018 1253, Serhiy Storchaka wrote: >> The benefit is that it will be easier to run all C API tests at once, >> and only them, and it will be clearer what C API is covered by tests. >> The disadvantage is that you will need to run several files for >> testing marshal for example. > > Can we make the regular tests import and also run the related C API > tests? So that a normal run wouldn't normally include the entire C API > test directory, but would include test classes in the related Python > test modules? (Maybe there's a way to decorate the test classes for this?) There are many ways of running tests: ./python -m test test_capi ./python -m test.test_capi ./python -m unittest test.test_capi ./python -m unittest discover Lib/test/test_capi/ They need different solutions for making them disabled by default. Seems that the simplest way is to move test_capi out of the test directory. But I think that in any case this will complicate testing code. > I agree with the intent, but also think that's quite a disadvantage. It > would be good to avoid it. Actually this disadvantage is not very large. There are not much C API tests for now. Testing unicode requires running not just test_unicode, but test_codecs, test_codeccallbacks, test_format, and yet few test. test_bytes itself is a mess, it needs significant rewriting. Marshal C API is outdated (it is based on FILE*), it is mostly unused in CPython. In any case Python tests should be enough for testing Python API. From eryksun at gmail.com Sun Jul 29 10:05:30 2018 From: eryksun at gmail.com (eryk sun) Date: Sun, 29 Jul 2018 14:05:30 +0000 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sun, Jul 29, 2018 at 9:13 AM, Tim Golden wrote: > > For an example: > > http://tjg.org.uk/test.log > > Thinkpad T420, 4Gb, i5, SSD > > Recently rebuilt and reinstalled: Win10, VS2017, TortoiseGit, standard > Windows Antimalware, usual developer tools. That particular run was done > with the laptop unattended (ie nothing else going on at the front end). > But the problem is certainly not specific to this laptop. On my last run I had one test directory that wasn't removed properly, but nothing like the flood of EACCES and ERROR_ACCES_DENIED errors you have in that log. Then again, I had Defender disabled by policy. I'll enable it and add exceptions for my source and build directories, and see how it goes. It would be nice if OSError instances always captured the last Windows error and NT status values when instantiated. We have no guarantees that these values are valid, but in many contexts they are. In the case of a test log, it would certainly help to clarify errors without having to individually investigate each one. For example, trying to open a directory as a file is a common error, but all Python tells us on Windows is that it failed with EACCES. In this case the last Windows error is ERROR_ACCESS_DENIED, which doesn't help, but the last NT status code is STATUS_FILE_IS_A_DIRECTORY (0xc00000ba). Here's a file opener that adds last_winerror and last_ntstatus values. import os ntdll = ctypes.WinDLL('ntdll') kernel32 = ctypes.WinDLL('kernel32') def nt_opener(path, flags): try: return os.open(path, flags) except OSError as e: last_ntstatus = ntdll.RtlGetLastNtStatus() last_winerror = kernel32.GetLastError() e.last_ntstatus = last_ntstatus & 2**32 - 1 e.last_winerror = (last_winerror if e.winerror is None else e.winerror) if e.errno is not None or e.winerror is not None: # hack the last error/status into the error message e.strerror = '[Last NtStatus {:#08x}] {}'.format( e.last_ntstatus, e.strerror or '') if e.winerror is None: e.strerror = '[Last WinError {}] {}'.format( e.last_winerror, e.strerror or '') raise e from None Opening a directory as a file: >>> open('C:/Windows', opener=nt_opener) Traceback (most recent call last): File "", line 1, in File "", line 17, in nt_opener File "", line 3, in nt_opener PermissionError: [Errno 13] [Last WinError 5] [Last NtStatus 0xc00000ba] Permission denied: 'C:/Windows' From jeremy.kloth at gmail.com Sun Jul 29 10:21:16 2018 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Sun, 29 Jul 2018 08:21:16 -0600 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sun, Jul 29, 2018 at 3:13 AM Tim Golden wrote: > For an example: > > http://tjg.org.uk/test.log Thank you! After inspecting all the errors, it does seem that they are ALL caused by "bare" os.unlink/rmdir calls. So it seems that a massive undertaking of ferreting out these locations and replacing them with their support equivalents would be needed to have a passing test suite for you. Unfortunately, test_mailbox is failing due to the same fate, but within the stdlib itself. The fix for that will probably need to be a rename/delete dance. diff --git a/Lib/mailbox.py b/Lib/mailbox.py index 056251d..23662f2 100644 --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -701,8 +701,10 @@ class _singlefileMailbox(Mailbox): try: os.rename(new_file.name, self._path) except FileExistsError: - os.remove(self._path) + temp_name = _create_temporary_name(self._path) + os.rename(self._path, temp_name) os.rename(new_file.name, self._path) + os.remove(temp_name) self._file = open(self._path, 'rb+') self._toc = new_toc self._pending = False @@ -2112,11 +2114,14 @@ def _create_carefully(path): finally: os.close(fd) +def _create_temporary_name(path): + """Create a temp filename based on path.""" + return '%s.%s.%s.%s' % (path, int(time.time()), socket.gethostname(), + os.getpid()) + def _create_temporary(path): """Create a temp file based on path and open for reading and writing.""" - return _create_carefully('%s.%s.%s.%s' % (path, int(time.time()), - socket.gethostname(), - os.getpid())) + return _create_carefully(_create_temporary_name(path)) def _sync_flush(f): """Ensure changes to file f are physically on disk.""" -- Jeremy Kloth From eryksun at gmail.com Sun Jul 29 10:22:03 2018 From: eryksun at gmail.com (eryk sun) Date: Sun, 29 Jul 2018 14:22:03 +0000 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: <1ddc7783-6516-c70a-0825-e8e0ce847dd0@python.org> References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> <1ddc7783-6516-c70a-0825-e8e0ce847dd0@python.org> Message-ID: On Sun, Jul 29, 2018 at 12:35 PM, Steve Dower wrote: > > One additional thing that may help (if support.unlink doesn't already do it) > is to rename the file before deleting it. Renames are always possible even > with open handles, and then you can create a new file at the original name. Renaming open files typically fails with a sharing violation (32). Most programs open files with read and write sharing but not delete sharing. This applies to Python, except temporary files (i.e. os.O_TEMPORARY) do share delete access. Renaming a file is effectively adding a new link and deleting the old link, so it requires opening the file with delete access. Also, renaming a directory that has open files in the tree fails with access denied (5). From eryksun at gmail.com Sun Jul 29 10:54:39 2018 From: eryksun at gmail.com (eryk sun) Date: Sun, 29 Jul 2018 14:54:39 +0000 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sun, Jul 29, 2018 at 2:21 PM, Jeremy Kloth wrote: > > try: > os.rename(new_file.name, self._path) > except FileExistsError: > - os.remove(self._path) > + temp_name = _create_temporary_name(self._path) > + os.rename(self._path, temp_name) > os.rename(new_file.name, self._path) > + os.remove(temp_name) This should call os.replace to allow the file system to replace the existing file. From mail at timgolden.me.uk Sun Jul 29 11:31:23 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Sun, 29 Jul 2018 16:31:23 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On 29/07/2018 15:21, Jeremy Kloth wrote: > On Sun, Jul 29, 2018 at 3:13 AM Tim Golden wrote: >> For an example: >> >> http://tjg.org.uk/test.log > > Thank you! After inspecting all the errors, it does seem that they > are ALL caused by "bare" os.unlink/rmdir calls. So it seems that a > massive undertaking of ferreting out these locations and replacing > them with their support equivalents would be needed to have a passing > test suite for you. Thanks for checking. As I mentioned elsewhere in this thread, I propose to go through and apply support.unlink where necessary. It won't make things any worse and will hopefully improve in some areas. For test_mailbox I've experimentally implemented a hybrid tempfile / local directory solution. ie I've created a new file on each run, but only within the python_ folder which already exists. As long as the directory cleans up there should be no leftovers. That's certainly helped although my re-run harness has provoked at least one error. TJG From baratharon at caesar.elte.hu Sun Jul 29 10:44:46 2018 From: baratharon at caesar.elte.hu (Barath Aron) Date: Sun, 29 Jul 2018 16:44:46 +0200 Subject: [Python-Dev] Using Python on a fork-less POSIX-like OS Message-ID: <0f77a21c-c804-d296-2b04-469091d14baa@caesar.elte.hu> Hello Python list, I intend to cross-compile Python v3.6.6 to Threos ( https://threos.io ) operating system. Threos is supports a quite large set from POSIX and C89/C99. Unfortunately, Threos lacks fork(2), but provides posix_spawn(3) instead. I already made some local changes in posixmodule.c to compile due to some features are detected as present but actually not supported, like HAVE_FORK -- I blame autotools for this :-). I don't know, however, whether the Python shall cross-compile without issues. My question is that the _posixsubprocess.c can be prepared to use posix_spawn(3) instead of fork(2)? Maybe the UNIX/Linux version can also benefit from it, see: https://salsa.debian.org/ruby-team/ruby-posix-spawn Best regards, Aron From jeremy.kloth at gmail.com Sun Jul 29 11:40:45 2018 From: jeremy.kloth at gmail.com (Jeremy Kloth) Date: Sun, 29 Jul 2018 09:40:45 -0600 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On Sun, Jul 29, 2018 at 9:34 AM Tim Golden wrote: > For test_mailbox I've experimentally implemented a hybrid tempfile / > local directory solution. ie I've created a new file on each run, but > only within the python_ folder which already exists. As long as the > directory cleans up there should be no leftovers. That's certainly > helped although my re-run harness has provoked at least one error. As Eryk noted, the fix for mailbox.py (yes, the stdlib needs fixing in this case) is quite simple: diff --git a/Lib/mailbox.py b/Lib/mailbox.py index 056251d..eb85df1 100644 --- a/Lib/mailbox.py +++ b/Lib/mailbox.py @@ -701,8 +701,7 @@ class _singlefileMailbox(Mailbox): try: os.rename(new_file.name, self._path) except FileExistsError: - os.remove(self._path) - os.rename(new_file.name, self._path) + os.replace(new_file.name, self._path) self._file = open(self._path, 'rb+') self._toc = new_toc self._pending = False -- Jeremy Kloth From berker.peksag at gmail.com Sun Jul 29 12:02:44 2018 From: berker.peksag at gmail.com (=?UTF-8?Q?Berker_Peksa=C4=9F?=) Date: Sun, 29 Jul 2018 19:02:44 +0300 Subject: [Python-Dev] Using Python on a fork-less POSIX-like OS In-Reply-To: <0f77a21c-c804-d296-2b04-469091d14baa@caesar.elte.hu> References: <0f77a21c-c804-d296-2b04-469091d14baa@caesar.elte.hu> Message-ID: On Sun, Jul 29, 2018 at 5:44 PM, Barath Aron wrote: > My question is that the _posixsubprocess.c can be prepared to use > posix_spawn(3) instead of fork(2)? Maybe the UNIX/Linux version can also > benefit from it, see: > https://salsa.debian.org/ruby-team/ruby-posix-spawn There is an open issue to add os.posix_spawn() at https://bugs.python.org/issue20104 --Berker From steve.dower at python.org Sun Jul 29 13:02:06 2018 From: steve.dower at python.org (Steve Dower) Date: Sun, 29 Jul 2018 18:02:06 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> <1ddc7783-6516-c70a-0825-e8e0ce847dd0@python.org> Message-ID: If the problem is AV scanners, then they should be opening them properly for this (and since the delete is not failing but is being deferred, I assume it's allowing deletes). If the problem is elsewhere in our code base then we have a different bug. Top-posted from my Windows 10 phone From: eryk sun Sent: Sunday, 29 July 2018 15:28 To: python-dev at python.org Subject: Re: [Python-Dev] Tests failing on Windows with TESTFN On Sun, Jul 29, 2018 at 12:35 PM, Steve Dower wrote: > > One additional thing that may help (if support.unlink doesn't already do it) > is to rename the file before deleting it. Renames are always possible even > with open handles, and then you can create a new file at the original name. Renaming open files typically fails with a sharing violation (32). Most programs open files with read and write sharing but not delete sharing. This applies to Python, except temporary files (i.e. os.O_TEMPORARY) do share delete access. Renaming a file is effectively adding a new link and deleting the old link, so it requires opening the file with delete access. Also, renaming a directory that has open files in the tree fails with access denied (5). _______________________________________________ Python-Dev mailing list Python-Dev at python.org https://mail.python.org/mailman/listinfo/python-dev Unsubscribe: https://mail.python.org/mailman/options/python-dev/steve.dower%40python.org -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Sun Jul 29 15:45:32 2018 From: brett at python.org (Brett Cannon) Date: Sun, 29 Jul 2018 12:45:32 -0700 Subject: [Python-Dev] Testing C API In-Reply-To: References: Message-ID: On Sun, Jul 29, 2018, 06:44 Serhiy Storchaka, wrote: > 29.07.18 15:39, Steve Dower ????: > > On 29Jul2018 1253, Serhiy Storchaka wrote: > >> The benefit is that it will be easier to run all C API tests at once, > >> and only them, and it will be clearer what C API is covered by tests. > >> The disadvantage is that you will need to run several files for > >> testing marshal for example. > > > > Can we make the regular tests import and also run the related C API > > tests? So that a normal run wouldn't normally include the entire C API > > test directory, but would include test classes in the related Python > > test modules? (Maybe there's a way to decorate the test classes for > this?) > > There are many ways of running tests: > > ./python -m test test_capi > ./python -m test.test_capi > ./python -m unittest test.test_capi > ./python -m unittest discover Lib/test/test_capi/ > > They need different solutions for making them disabled by default. Seems > that the simplest way is to move test_capi out of the test directory. > But I think that in any case this will complicate testing code. > > > I agree with the intent, but also think that's quite a disadvantage. It > > would be good to avoid it. > > Actually this disadvantage is not very large. There are not much C API > tests for now. For now, but if are successful, Serhiy, there will be a lot more tests. ? Testing unicode requires running not just test_unicode, > but test_codecs, test_codeccallbacks, test_format, and yet few test. > test_bytes itself is a mess, it needs significant rewriting. Marshal C > API is outdated (it is based on FILE*), it is mostly unused in CPython. > In any case Python tests should be enough for testing Python API. > I think it depends on how you mentally group tests. Do you want to run all tests relating to marshal when you run test_marshal, or should you run test_marshal just for the Python code and test_capi.test_marshal for just the C API depending on what you changed? One perk of putting the C API tests in under test_capi is it makes it easier for other implementations to ignore those tests if they want to (although I'm assuming all the tests will also be marked as appropriate as CPython-specific). > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Sun Jul 29 15:47:51 2018 From: vstinner at redhat.com (Victor Stinner) Date: Sun, 29 Jul 2018 21:47:51 +0200 Subject: [Python-Dev] Let's change to C API! Message-ID: Hi, I just sent an email to the capi-sig mailing list. Since this mailing list was idle for months, I copy my email here to get a wider audience. But if possible, I would prefer that you join me on capi-sig to reply ;-) -- Hi, Last year, I gave a talk at the Language Summit (during Pycon) to explain that CPython should become 2x faster to remain competitive. IMHO all attempts to optimize Python (CPython forks) have failed because they have been blocked by the C API which implies strict constraints. I started to write a proposal to change the C API to hide implementation details, to prepare CPython for future changes. It allows to experimental optimization ideas without loosing support for C extensions. C extensions are a large part of the Python success. They are also the reason why PyPy didn't replace CPython yet. PyPy cpyext remains slower than CPython because PyPy has to mimick CPython which adds a significant overhead (even if PyPy developers are working *hard* to optimize it). I created a new to discuss how to introduce backward incompatible changes in the C API without breaking all C extensions: http://pythoncapi.readthedocs.io/ The source can be found at: https://github.com/vstinner/pythoncapi/ I would like to create a team of people who want to work on this project: CPython, PyPy, Cython and anyone who depends on the C API. Contact me in private if you want to be added to the GitHub project. I propose to discuss on the capi-sig mailing list since I would like to involve people from various projects, and I don't want to bother you with the high traffic of python-dev. Victor PS: I added some people as BCC ;-) From baratharon at caesar.elte.hu Sun Jul 29 16:22:29 2018 From: baratharon at caesar.elte.hu (Barath Aron) Date: Sun, 29 Jul 2018 22:22:29 +0200 Subject: [Python-Dev] Using Python on a fork-less POSIX-like OS In-Reply-To: References: <0f77a21c-c804-d296-2b04-469091d14baa@caesar.elte.hu> Message-ID: On 07/29/2018 06:02 PM, Berker Peksa? wrote: > There is an open issue to add os.posix_spawn() at > https://bugs.python.org/issue20104 Seems promising, but 3.7 does not support it. And I don't see whether Python will work without fork(). - bpo-20104: Expose posix_spawn as a low level API in the os module. ? (removed before 3.7.0rc1) > --Berker Aron From J.Demeyer at UGent.be Sun Jul 29 17:41:47 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sun, 29 Jul 2018 23:41:47 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: Message-ID: <5B5E349B.5000703@UGent.be> My first impression is that making things faster and hiding implementation details in the ABI are contrary goals. I agree with hiding implementation details in the API but not in the ABI. For example, you mention that you want to make Py_INCREF() a function call instead of a macro. But since Py_INCREF is very common, I would guess that this would make performance worse (not by much maybe but surely measurable). Jeroen. From vstinner at redhat.com Sun Jul 29 20:46:22 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 30 Jul 2018 02:46:22 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <5B5E349B.5000703@UGent.be> References: <5B5E349B.5000703@UGent.be> Message-ID: 2018-07-29 23:41 GMT+02:00 Jeroen Demeyer : > For example, you mention that you want to make Py_INCREF() a function call > instead of a macro. But since Py_INCREF is very common, I would guess that > this would make performance worse (not by much maybe but surely measurable). For the very specific case of Py_INCREF(), yes, I agree that performance is an issue. But I don't see how I would hide completely the PyObject structure without converting Py_INCREF() macro with a function call. (I have reasons to want to hide everything, explained in the project.) The open question is if the cost of using function calls for Py_INCREF/DECREF versus the benefit of having the ability to modify deeply CPython internals. I'm not sure that it's worth to bet at this point, it's too early, and we can decide that later. Moreover, it's also possible to keep Py_INCREF() as a macro in the "backward compatible" mode, but require a function call in the mode which hides all implementation details (the one where you can experiment deep CPython internals changes). Victor From rosuav at gmail.com Sun Jul 29 21:18:37 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 30 Jul 2018 11:18:37 +1000 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <5B5E349B.5000703@UGent.be> Message-ID: On Mon, Jul 30, 2018 at 10:46 AM, Victor Stinner wrote: > 2018-07-29 23:41 GMT+02:00 Jeroen Demeyer : >> For example, you mention that you want to make Py_INCREF() a function call >> instead of a macro. But since Py_INCREF is very common, I would guess that >> this would make performance worse (not by much maybe but surely measurable). > > For the very specific case of Py_INCREF(), yes, I agree that > performance is an issue. But I don't see how I would hide completely > the PyObject structure without converting Py_INCREF() macro with a > function call. (I have reasons to want to hide everything, explained > in the project.) > > The open question is if the cost of using function calls for > Py_INCREF/DECREF versus the benefit of having the ability to modify > deeply CPython internals. > > I'm not sure that it's worth to bet at this point, it's too early, and > we can decide that later. Moreover, it's also possible to keep > Py_INCREF() as a macro in the "backward compatible" mode, but require > a function call in the mode which hides all implementation details > (the one where you can experiment deep CPython internals changes). > If the macro and function are absolutely 100% compatible, it would be possible to set compilation to use the function by default, and have a directive that switches to using the macro. It'd improve performance at the price of locking you to the exact CPython build. So within CPython itself, there'd be no performance cost (ergo if you mess with the internals, you have to recompile all of CPython), most extension libraries would pay a small (probably immeasurable) price for compatibility, and a small handful could opt to improve performance at the price of breaking if anything changes. ChrisA From raymond.hettinger at gmail.com Mon Jul 30 02:46:23 2018 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Sun, 29 Jul 2018 23:46:23 -0700 Subject: [Python-Dev] Testing C API In-Reply-To: References: Message-ID: <5791BC6D-4150-4313-AA42-04A4C8B12BAE@gmail.com> > On Jul 29, 2018, at 4:53 AM, Serhiy Storchaka wrote: > > The benefit is that it will be easier to run all C API tests at once, and only them, and it will be clearer what C API is covered by tests. The disadvantage is that you will need to run several files for testing marshal for example. > > What are your thoughts? I prefer the current organization that keeps the various tests together with the category being tested. I almost never need to run the C API tests all at once, but I do need to see all the tests for an object in one place. When maintaining something like marshal, it would be easy to miss some of the tests if they are in a separate file. IMO, the proposed change would hinder future maintenance and fly in the face of our traditional code organization. Raymond From storchaka at gmail.com Mon Jul 30 03:06:32 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Mon, 30 Jul 2018 10:06:32 +0300 Subject: [Python-Dev] Testing C API In-Reply-To: <5791BC6D-4150-4313-AA42-04A4C8B12BAE@gmail.com> References: <5791BC6D-4150-4313-AA42-04A4C8B12BAE@gmail.com> Message-ID: 30.07.18 09:46, Raymond Hettinger ????: > I prefer the current organization that keeps the various tests together with the category being tested. I almost never need to run the C API tests all at once, but I do need to see all the tests for an object in one place. When maintaining something like marshal, it would be easy to miss some of the tests if they are in a separate file. IMO, the proposed change would hinder future maintenance and fly in the face of our traditional code organization. What about moving just test_capi.py, test_getargs2.py and test_structmembers.py into Lib/test/test_capi? They are not related to specific types or modules. For references: An issue on the tracker: https://bugs.python.org/issue34272. A PR that moves C API into the test_capi subdirectory: https://github.com/python/cpython/pull/8551. Currently it just moves test_capi.py, test_getargs2.py and test_structmembers.py. An initial version that moved also other C API tests: https://github.com/python/cpython/pull/8551/commits/eb16b9ee9eb36c9965c2d852461bb7838f3f6dfa. From vstinner at redhat.com Mon Jul 30 04:20:06 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 30 Jul 2018 10:20:06 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <5B5E349B.5000703@UGent.be> Message-ID: The API leaking all implementation details will remain available as an opt-in option for Cython, cffi and debug tools. But this API will only be usable on the "slow" Python runtime, the one which keeps maximum backward compatibility. To get new optimizations, you have to use Py_INCREF() and avoid accessing C strucuture fields, which may or may not need to modify your code. Hum, please, join the capi-sig mailing list, since I already explained that in my long reply to Stefan on capi-sig ;-) Victor Le lundi 30 juillet 2018, Chris Angelico a ?crit : > On Mon, Jul 30, 2018 at 10:46 AM, Victor Stinner wrote: >> 2018-07-29 23:41 GMT+02:00 Jeroen Demeyer : >>> For example, you mention that you want to make Py_INCREF() a function call >>> instead of a macro. But since Py_INCREF is very common, I would guess that >>> this would make performance worse (not by much maybe but surely measurable). >> >> For the very specific case of Py_INCREF(), yes, I agree that >> performance is an issue. But I don't see how I would hide completely >> the PyObject structure without converting Py_INCREF() macro with a >> function call. (I have reasons to want to hide everything, explained >> in the project.) >> >> The open question is if the cost of using function calls for >> Py_INCREF/DECREF versus the benefit of having the ability to modify >> deeply CPython internals. >> >> I'm not sure that it's worth to bet at this point, it's too early, and >> we can decide that later. Moreover, it's also possible to keep >> Py_INCREF() as a macro in the "backward compatible" mode, but require >> a function call in the mode which hides all implementation details >> (the one where you can experiment deep CPython internals changes). >> > > If the macro and function are absolutely 100% compatible, it would be > possible to set compilation to use the function by default, and have a > directive that switches to using the macro. It'd improve performance > at the price of locking you to the exact CPython build. So within > CPython itself, there'd be no performance cost (ergo if you mess with > the internals, you have to recompile all of CPython), most extension > libraries would pay a small (probably immeasurable) price for > compatibility, and a small handful could opt to improve performance at > the price of breaking if anything changes. > > ChrisA > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Mon Jul 30 04:23:55 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 30 Jul 2018 10:23:55 +0200 Subject: [Python-Dev] Using Python on a fork-less POSIX-like OS In-Reply-To: References: <0f77a21c-c804-d296-2b04-469091d14baa@caesar.elte.hu> Message-ID: Python 3.8 will support os.posix_spawn(). I would like to see it used whenever possible instead of fork+exec, since it's faster and it can be safer on some platforms. Pablo Salgado is your guy for that. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From raymond.hettinger at gmail.com Mon Jul 30 04:25:16 2018 From: raymond.hettinger at gmail.com (Raymond Hettinger) Date: Mon, 30 Jul 2018 01:25:16 -0700 Subject: [Python-Dev] Testing C API In-Reply-To: References: <5791BC6D-4150-4313-AA42-04A4C8B12BAE@gmail.com> Message-ID: <9586F573-9561-43FF-8F77-DD2F622E33CF@gmail.com> > On Jul 30, 2018, at 12:06 AM, Serhiy Storchaka wrote: > > 30.07.18 09:46, Raymond Hettinger ????: >> I prefer the current organization that keeps the various tests together with the category being tested. I almost never need to run the C API tests all at once, but I do need to see all the tests for an object in one place. When maintaining something like marshal, it would be easy to miss some of the tests if they are in a separate file. IMO, the proposed change would hinder future maintenance and fly in the face of our traditional code organization. > > What about moving just test_capi.py, test_getargs2.py and test_structmembers.py into Lib/test/test_capi? They are not related to specific types or modules That would be reasonable. Raymond From baratharon at caesar.elte.hu Mon Jul 30 04:26:30 2018 From: baratharon at caesar.elte.hu (Barath Aron) Date: Mon, 30 Jul 2018 10:26:30 +0200 Subject: [Python-Dev] Using Python on a fork-less POSIX-like OS In-Reply-To: References: <0f77a21c-c804-d296-2b04-469091d14baa@caesar.elte.hu> Message-ID: <70785bf4-6f39-af27-cce6-c15dc332a667@caesar.elte.hu> On 07/30/2018 10:23 AM, Victor Stinner wrote: > Python 3.8 will support os.posix_spawn(). I would like to see it used > whenever possible instead of fork+exec, since it's faster and it can > be safer on some platforms. Pablo Salgado is your guy for that. > > Victor Awesome! Will this backported to 2.7? Or people should forget 2.7? Aron From vstinner at redhat.com Mon Jul 30 04:32:07 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 30 Jul 2018 10:32:07 +0200 Subject: [Python-Dev] Using Python on a fork-less POSIX-like OS In-Reply-To: <70785bf4-6f39-af27-cce6-c15dc332a667@caesar.elte.hu> References: <0f77a21c-c804-d296-2b04-469091d14baa@caesar.elte.hu> <70785bf4-6f39-af27-cce6-c15dc332a667@caesar.elte.hu> Message-ID: Supporting a new platform requires a lot work. It would be more reasonable for you to first try to get a good support of the master branch before start thinking how to support Python versions. Python 2.7 in 2018? Really? Tick tock: https://pythonclock.org/ http://python3statement.org/ Usually, we don't support new platforms in CPython without strong support of a core developer. See the PEP 11 for more conditions like buildbot requirement. I suggest you to start working on a fork of CPython and maintain your changes in a branch. Git rebase makes it easy. Victor Le lundi 30 juillet 2018, Barath Aron a ?crit : > On 07/30/2018 10:23 AM, Victor Stinner wrote: >> >> Python 3.8 will support os.posix_spawn(). I would like to see it used whenever possible instead of fork+exec, since it's faster and it can be safer on some platforms. Pablo Salgado is your guy for that. >> >> Victor > > Awesome! Will this backported to 2.7? Or people should forget 2.7? > > Aron > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Jul 30 05:01:20 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 30 Jul 2018 11:01:20 +0200 Subject: [Python-Dev] Let's change to C API! References: Message-ID: <20180730110120.6d03e6d8@fsol> Hi Victor, On Sun, 29 Jul 2018 21:47:51 +0200 Victor Stinner wrote: > > I just sent an email to the capi-sig mailing list. Since this mailing > list was idle for months, I copy my email here to get a wider > audience. But if possible, I would prefer that you join me on capi-sig > to reply ;-) > > -- > > Hi, > > Last year, I gave a talk at the Language Summit (during Pycon) to > explain that CPython should become 2x faster to remain competitive. > IMHO all attempts to optimize Python (CPython forks) have failed > because they have been blocked by the C API which implies strict > constraints. Well, that's your opinion, but did you prove it? Which CPython forks did you examine that failed because of the C API? The one area I know of where the C API is a strong barrier to improvement is removing the GIL, and I'd rather let Larry speak about it. Regards Antoine. From tjreedy at udel.edu Mon Jul 30 05:04:15 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 30 Jul 2018 05:04:15 -0400 Subject: [Python-Dev] Using Python on a fork-less POSIX-like OS In-Reply-To: <70785bf4-6f39-af27-cce6-c15dc332a667@caesar.elte.hu> References: <0f77a21c-c804-d296-2b04-469091d14baa@caesar.elte.hu> <70785bf4-6f39-af27-cce6-c15dc332a667@caesar.elte.hu> Message-ID: On 7/30/2018 4:26 AM, Barath Aron wrote: > On 07/30/2018 10:23 AM, Victor Stinner wrote: >> Python 3.8 will support os.posix_spawn(). I would like to see it used >> whenever possible instead of fork+exec, since it's faster and it can >> be safer on some platforms. Pablo Salgado is your guy for that. >> >> Victor > > Awesome! Will this backported to 2.7? Or people should forget 2.7? Normally, enhancements are not backported anywhere. New API == new version of Python. You can, and people and organizations do, compile your own customized version. Official (pydev) bugfixes for 2.7 end 1 Jan 2020. Many projects have already stopped or will stop before then fixing 2.7 versions of their packages. But some people will probably use it for at least another decade. Do what is best for you. -- Terry Jan Reedy From songofacandy at gmail.com Mon Jul 30 07:11:58 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 30 Jul 2018 20:11:58 +0900 Subject: [Python-Dev] PEP 576/579/580 benchmark: mistune In-Reply-To: <5B5B1F92.8090102@UGent.be> References: <5B5B1F92.8090102@UGent.be> Message-ID: Like previous SageMath bench, this is caused by Cython's specialization; __Pyx_PyObject_CallOneArg. It specializing calling PyFunction and PyCFunction, but it doesn't specialized for calling CyFunction. Cython can optimize both benchmark with binding, but without PEP 576 nor 580, by adding calling code specialized for CyFunction to __Pyx_PyObject_CallOneArg. Of course, this specialization doesn't help (1) calling CyFunction from Python case and (2) calling CyFunction from C case. Please don't attack me about it. I just survey and report where the overhead in both benchmark with binding=True in this mail. I will send another mail about PEP 576 vs 580. But I'm not good English writer. It's difficult to explain my opinion on such sensitive topic. Regards, On Fri, Jul 27, 2018 at 10:38 PM Jeroen Demeyer wrote: > > Hello all, > > since my latest benchmark for PEP 580 [1] involved SageMath, which is > quite a big project, I instead propose a much simpler benchmark > involving mistune. > > mistune [2] is a Markdown parser implemented in the Python language. It > optionally allows Cython compilation. It doesn't use any kind of > optimization beyond that, but I created a branch [3] to use extension > types instead of Python classes. > > Cython can either use built-in functions/methods or a custom class > (which is not optimized but which would be optimized with PEP 580). > > I benchmarked mistune with custom classes [3] (binding=True, the > default) and with built-in functions/methods [4] (binding=False). This > is the median time of 5 runs: > > Binding=True: 9.063s > Binding=False: 8.658s > > So this shows again that PEP 580 improves performance in actual > real-world use cases. > > > Jeroen. > > > > [1] https://mail.python.org/pipermail/python-dev/2018-July/154740.html > [2] https://github.com/lepture/mistune > [3] https://github.com/jdemeyer/mistune/tree/cython_pxd > [4] https://github.com/jdemeyer/mistune/tree/cython_pxd_nobinding > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/songofacandy%40gmail.com -- INADA Naoki From vstinner at redhat.com Mon Jul 30 07:19:57 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 30 Jul 2018 13:19:57 +0200 Subject: [Python-Dev] Testing C API In-Reply-To: References: Message-ID: Buildbots have a timeout of 15 min per test. I suggest to use multiple test_capi_.py files rather than a directory which behaves as a single test. Or regrtest should be modified to implement timeout differently. Victor Le dimanche 29 juillet 2018, Serhiy Storchaka a ?crit : > Currently C API is not completely covered by tests. Tests for particular parts of C API are scattered through different files. There are files completely purposed for testing C API (like test_capi.py, test_getargs2.py), there are classes (usually having "CAPI" in the name) in different files for testing C API specific for unicode, marshal. Argument parsing tests are split between two files, test_capi.py, test_getargs2.py. > > I need to add new tests for new features, and I'm going to add new tests for existing C API. But first I'm going to reorganize tests. Add a new directory Lib/test/test_capi, and move all C API tests into it, grouped by function prefixes. test_getargs.py for testing PyArg_*(), test_unicode.py for testing PyUnicode_*(), etc. Tests that use the _testcapi module, but don't test specific C API, will left on place. > > The benefit is that it will be easier to run all C API tests at once, and only them, and it will be clearer what C API is covered by tests. The disadvantage is that you will need to run several files for testing marshal for example. > > What are your thoughts? > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Mon Jul 30 07:23:37 2018 From: vstinner at redhat.com (Victor Stinner) Date: Mon, 30 Jul 2018 13:23:37 +0200 Subject: [Python-Dev] Testing C API In-Reply-To: References: Message-ID: Or maybe test__capi.py so you can more easily discover test_unicode_cami while working on Unicode. You can use -m "test_*_capi" to run all C API tests. Victor Le lundi 30 juillet 2018, Victor Stinner a ?crit : > Buildbots have a timeout of 15 min per test. I suggest to use multiple test_capi_.py files rather than a directory which behaves as a single test. Or regrtest should be modified to implement timeout differently. > > Victor > > Le dimanche 29 juillet 2018, Serhiy Storchaka a ?crit : >> Currently C API is not completely covered by tests. Tests for particular parts of C API are scattered through different files. There are files completely purposed for testing C API (like test_capi.py, test_getargs2.py), there are classes (usually having "CAPI" in the name) in different files for testing C API specific for unicode, marshal. Argument parsing tests are split between two files, test_capi.py, test_getargs2.py. >> >> I need to add new tests for new features, and I'm going to add new tests for existing C API. But first I'm going to reorganize tests. Add a new directory Lib/test/test_capi, and move all C API tests into it, grouped by function prefixes. test_getargs.py for testing PyArg_*(), test_unicode.py for testing PyUnicode_*(), etc. Tests that use the _testcapi module, but don't test specific C API, will left on place. >> >> The benefit is that it will be easier to run all C API tests at once, and only them, and it will be clearer what C API is covered by tests. The disadvantage is that you will need to run several files for testing marshal for example. >> >> What are your thoughts? >> >> _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com >> -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Mon Jul 30 08:12:22 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 30 Jul 2018 14:12:22 +0200 Subject: [Python-Dev] Error message for wrong number of arguments Message-ID: <5B5F00A6.8070802@UGent.be> Hello, I noticed an inconsistency in the error messages for the number of arguments to a method call. For Python methods, the "self" argument is counted. For built-in methods, the "self" argument is *not* counted: >>> class mylist(list): ... def append(self, val): super().append(val) >>> f = list().append >>> f(1,2) Traceback (most recent call last): File "", line 1, in TypeError: append() takes exactly one argument (2 given) >>> g = mylist().append >>> g(1,2) Traceback (most recent call last): File "", line 1, in TypeError: append() takes 2 positional arguments but 3 were given I think it has been argued before that it's a feature that self is counted. So I consider the error message for list().append a bug. This is one of the many oddities I noticed while working on improving built-in functions. Would you agree to change the error message for built-in methods to be closer to Python methods? Jeroen. From ronaldoussoren at mac.com Mon Jul 30 08:24:25 2018 From: ronaldoussoren at mac.com (Ronald Oussoren) Date: Mon, 30 Jul 2018 14:24:25 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <5B5E349B.5000703@UGent.be> Message-ID: > On 30 Jul 2018, at 10:20, Victor Stinner wrote: > > The API leaking all implementation details will remain available as an opt-in option for Cython, cffi and debug tools. But this API will only be usable on the "slow" Python runtime, the one which keeps maximum backward compatibility. To get new optimizations, you have to use Py_INCREF() and avoid accessing C strucuture fields, which may or may not need to modify your code. > > Hum, please, join the capi-sig mailing list, since I already explained that in my long reply to Stefan on capi-sig ;-) Interesting. I didn?t know that list exists. I?ll respond to your message on that list. Ronald -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Mon Jul 30 08:50:30 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 30 Jul 2018 14:50:30 +0200 Subject: [Python-Dev] PEP 576/579/580 benchmark: mistune In-Reply-To: <4009dcc08e074a148fdae1ad288b9281@xmail101.UGent.be> References: <5B5B1F92.8090102@UGent.be> <4009dcc08e074a148fdae1ad288b9281@xmail101.UGent.be> Message-ID: <5B5F0996.1010200@UGent.be> On 2018-07-30 13:11, INADA Naoki wrote: > Like previous SageMath bench, this is caused by Cython's > specialization; __Pyx_PyObject_CallOneArg. > > It specializing calling PyFunction and PyCFunction, but it doesn't > specialized for calling CyFunction. Yes, I saw that too. But this is exactly what CPython does (it optimizes PyFunction and PyCFunction but not CyFunction), so I would still argue that the benchmark is fair. From songofacandy at gmail.com Mon Jul 30 09:35:54 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Mon, 30 Jul 2018 22:35:54 +0900 Subject: [Python-Dev] [PEP 576/580] Reserve one type slot for Cython Message-ID: As repeatedly said, PEP 580 is very complicated protocol when just implementing callable object. It is optimized for implementing custom method object, although almost only Cython want the custom method type. I'm not sure adding such complicated protocol almost only for Cython. If CyFunction can be implemented behind PEP 576, it may be better. On the other hand, most complexity of PEP 580 is not newly added. Most of them are in PyCFunction, method_descriptor, and some calling APIs already. PEP 580 just restructure them completely to be reusable from Cython. So I agree that PEP 580 is better when thinking from Cython's side. --- I'm not sure which way we should go yet. But my current idea is: * Implement PEP 580 as semi-public APIs only for tools like Cython. * Other Python implementation may not support it in foreseeable future. So such tools should support legacy implementation too. * PEP 576 and 580 are not strictly mutually exclusive; PEP 576 may be accepted in addition to PEP 580, for simpler FASTCALL-able object support. Especially for extension author prefer C to Cython (including stdlib). * If this happened, PEP 580 can remove one abstraction; tp_ccalloffset is offset of PyCCallRoot instead of pointer to it. Py_TPFLAGS_FUNCTION_DESCRIPTOR will be removed from PEP 576 too. Regards, -- INADA Naoki From J.Demeyer at UGent.be Mon Jul 30 10:40:00 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 30 Jul 2018 16:40:00 +0200 Subject: [Python-Dev] [PEP 576/580] Reserve one type slot for Cython In-Reply-To: <31a0ccacffdb4dce8d7d73513e87175b@xmail101.UGent.be> References: <31a0ccacffdb4dce8d7d73513e87175b@xmail101.UGent.be> Message-ID: <5B5F2340.9090105@UGent.be> On 2018-07-30 15:35, INADA Naoki wrote: > As repeatedly said, PEP 580 is very complicated protocol > when just implementing callable object. Can you be more concrete what you find complicated? Maybe I can improve the PEP to explain it more. Also, I'm open to suggestions to make it less complicated. > It is optimized for implementing custom method object, although > almost only Cython want the custom method type. For the record, Numba also seems interested in the PEP: https://groups.google.com/a/continuum.io/d/msg/numba-users/2G6k2R92MIM/P-cFKW7xAgAJ > I'm not sure adding such complicated protocol almost only for Cython. > If CyFunction can be implemented behind PEP 576, it may be better. I recall my post https://mail.python.org/pipermail/python-dev/2018-July/154238.html explaining the main difference between PEP 576 and PEP 580. I would like to stress that PEP 580 was designed for maximum performance, both today and for future extensions (such as calling with native C types). > * PEP 576 and 580 are not strictly mutually exclusive; PEP 576 may be > accepted in addition to PEP 580 I don't think that this is a good idea: you will mostly end up with the disadvantages of both approaches. Jeroen. From ncoghlan at gmail.com Mon Jul 30 11:11:12 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 Jul 2018 01:11:12 +1000 Subject: [Python-Dev] Fuzzing the Python standard library In-Reply-To: References: <1531845897.661372.1443802960.08AEA2F5@webmail.messagingengine.com> <1531860904.733490.1444082664.1CAF20DA@webmail.messagingengine.com> Message-ID: On 18 July 2018 at 17:49, Steve Holden wrote: > On Tue, Jul 17, 2018 at 11:44 PM, Paul G wrote: >> >> In many languages numeric types can't hold arbitrarily large values, and I >> for one hadn't really previously recognized that if you read in a numeric >> value with an exponent that it would be represented *exactly* in memory (and >> thus one object with a very compact representation can take up huge amounts >> of memory). It's also not *inconceivable* that under the hood Python would >> represent fractions.Fraction("1.64E6646466664") "lazily" in some fashion so >> that it did not consume all the memory on disk. >> > Sooner or later you are going to need the digits of the number to perform a > computation. Exactly when would you propose the deferred evaluation should > take place? > > There are already occasional inquiries about the effects of creation of such > large numbers and their unexpected effects, so they aren't completely > unknown. At the same time, this isn't exactly a mainstream "bug", as > evidenced by the fact that such issues are relatively rare. It does mean that if Fraction is being used with untrusted inputs though, it *does* make sense to put a reasonable upper bound on permitted exponents. The default decimal context caps expression results at an exponent of less than 1 million for example: >>> +decimal.Decimal("1e999_999") Decimal('1E+999999') >>> +decimal.Decimal("1e1_000_000") Traceback (most recent call last): File "", line 1, in decimal.Overflow: [] That's already large enough to result in a ~415k integer that takes a minute or so for my machine to create if I call int() on it. So I think it does make sense to at least describe how to use the decimal module to do some initial sanity checking on potentially exponential inputs, even if the fractions module itself never gains native support for processing untrusted inputs. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jul 30 11:26:18 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 Jul 2018 01:26:18 +1000 Subject: [Python-Dev] Update on Cygwin support (was: Clarifying Cygwin support in CPython) In-Reply-To: References: Message-ID: On 26 July 2018 at 02:13, Erik Bray wrote: > I think a new approach that might be more practical for actually > getting this platform re-supported, is to go ahead and add a CI build, > and just skip all known failing test modules. This is what I've done > in a new PR to add a Cygwin build on AppVeyor: > > https://github.com/python/cpython/pull/8463 > > This is far from ideal of course, and should not mean the platform is > "supported". But now I and others can go through and fix the > remaining test failures, re-enable those modules in the CI > configuration, and actually obtain some meaningful results, which will > hopefully encourage the core committers to accept fixes for the > platform. I believe the PEP 538 & 540 locale handling tests are amongst those that are still a bit sketchy (or outright broken?) on Cygwin, and I think having an advisory CI bot would definitely help with that. (Cygwin/MinGW are an interesting hybrid that really highlight the fact that neither "POSIX implies not Windows" nor "Windows implies the Win32 API" are entirely valid assumptions) So your suggested approach seems like a plausible way forward to me. The main potentially viable alternative I see would be to set up the *buildbot* first, and then devote the custom builder branch to the task of Cygwin testing for a while: https://devguide.python.org/buildbots/#custom-builders However, I think the overall UX of that would be worse than going down the advisory CI path (especially since it wouldn't really help with the aspect of parallel development introducing new Cygwin failures). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jul 30 11:28:28 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 Jul 2018 01:28:28 +1000 Subject: [Python-Dev] Error message for wrong number of arguments In-Reply-To: <5B5F00A6.8070802@UGent.be> References: <5B5F00A6.8070802@UGent.be> Message-ID: On 30 July 2018 at 22:12, Jeroen Demeyer wrote: > I think it has been argued before that it's a feature that self is counted. > So I consider the error message for list().append a bug. This is one of the > many oddities I noticed while working on improving built-in functions. > > Would you agree to change the error message for built-in methods to be > closer to Python methods? I would, and I think it would make sense for the PEP to cite improving consistency (and reducing code duplication?) in that regard as an advantage of the PEP. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Mon Jul 30 11:41:59 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 Jul 2018 01:41:59 +1000 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: On 29 July 2018 at 03:20, Tim Golden wrote: > I think that was my starting point: rather than develop increasingly > involved and still somewhat brittle mechanisms, why not do what you'd > naturally do with a new test and use tempfile? I was expecting someone to > come forward to highlight some particular reason why the TESTFN approach is > superior, but apart from a reference to the possibly cost of creating a new > temporary file per test, no-one really has. For higher level modules, "just use tempfile to create a new temporary directory, then unlink it at the end" is typically going to be a good answer (modulo the current cleanup issues that Jeremy is reporting, but ideally those will be fixed rather than avoided, either by improving the way the module is being used, or fixing any underlying defects). For lower level modules though, adding a test suite dependency on tempfile introduces a non-trivial chance of adding an operational dependency from a module's test suite back to the module itself. When that happens, module regressions may show up as secondary failures in tempfile that then need to be debugged, rather than as specific unit test failures that point you towards exactly what you broke. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From mail at timgolden.me.uk Mon Jul 30 11:46:47 2018 From: mail at timgolden.me.uk (Tim Golden) Date: Mon, 30 Jul 2018 16:46:47 +0100 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> Message-ID: <738a82ba-a145-76ba-4447-763d59834379@timgolden.me.uk> On 30/07/2018 16:41, Nick Coghlan wrote: > On 29 July 2018 at 03:20, Tim Golden wrote: >> I think that was my starting point: rather than develop increasingly >> involved and still somewhat brittle mechanisms, why not do what you'd >> naturally do with a new test and use tempfile? I was expecting someone to >> come forward to highlight some particular reason why the TESTFN approach is >> superior, but apart from a reference to the possibly cost of creating a new >> temporary file per test, no-one really has. > > For higher level modules, "just use tempfile to create a new temporary > directory, then unlink it at the end" is typically going to be a good > answer (modulo the current cleanup issues that Jeremy is reporting, > but ideally those will be fixed rather than avoided, either by > improving the way the module is being used, or fixing any underlying > defects). > > For lower level modules though, adding a test suite dependency on > tempfile introduces a non-trivial chance of adding an operational > dependency from a module's test suite back to the module itself. When > that happens, module regressions may show up as secondary failures in > tempfile that then need to be debugged, rather than as specific unit > test failures that point you towards exactly what you broke. > > Cheers, > Nick. > Thanks Nick; I hadn't thought about the possible interdependency issue. I think for the moment my approach will be to switch to support.unlink wherever possible to start with. Before introducing other (eg tempfile) changes, this should at least narrow the issues down. I've made a start on that (before inadvertently blowing away all the changes since my hours-previous commit!) If further changes are necessary then I'll probably look case-by-case to see whether a tempfile or some other solution would help. That said, that's potentially quite a lot of change -- at least in terms of files changed if not strictly of functionality. So I'm thinking of trickle-feeding the changes through as people will understandably baulk at a patchbomb (PR-bomb?) hitting the codebase all at once. TJG From ncoghlan at gmail.com Mon Jul 30 11:53:59 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 31 Jul 2018 01:53:59 +1000 Subject: [Python-Dev] Testing C API In-Reply-To: References: Message-ID: On 30 July 2018 at 21:23, Victor Stinner wrote: > Or maybe test__capi.py so you can more easily discover > test_unicode_cami while working on Unicode. You can use -m "test_*_capi" to > run all C API tests. Missing word there: -m test "test*_capi" I think between this approach and expanding test_capi to be a test package, not just a test file, I think it would be possible to make these parts of the test a fair bit more discoverable and maintainable. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From stefan_ml at behnel.de Mon Jul 30 14:03:54 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Mon, 30 Jul 2018 20:03:54 +0200 Subject: [Python-Dev] [PEP 576/580] Reserve one type slot for Cython In-Reply-To: <5B5F2340.9090105@UGent.be> References: <31a0ccacffdb4dce8d7d73513e87175b@xmail101.UGent.be> <5B5F2340.9090105@UGent.be> Message-ID: Jeroen Demeyer schrieb am 30.07.2018 um 16:40: > On 2018-07-30 15:35, INADA Naoki wrote: >> As repeatedly said, PEP 580 is very complicated protocol >> when just implementing callable object. > > Can you be more concrete what you find complicated? Maybe I can improve the > PEP to explain it more. Also, I'm open to suggestions to make it less > complicated. > >> It is optimized for implementing custom method object, although >> almost only Cython want the custom method type. > > For the record, Numba also seems interested in the PEP: > https://groups.google.com/a/continuum.io/d/msg/numba-users/2G6k2R92MIM/P-cFKW7xAgAJ To add to that record, I (briefly) talked to Ronan Lamy about this at EuroPython and PyPy could also be interested in generalising the call protocol, especially with the future goal of extending it into a C level call protocol that their JIT could understand and build a cffi-like interface on. Stefan From chris.barker at noaa.gov Mon Jul 30 14:22:48 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 30 Jul 2018 11:22:48 -0700 Subject: [Python-Dev] Error message for wrong number of arguments In-Reply-To: <5B5F00A6.8070802@UGent.be> References: <5B5F00A6.8070802@UGent.be> Message-ID: On Mon, Jul 30, 2018 at 5:12 AM, Jeroen Demeyer wrote: > I think it has been argued before that it's a feature that self is > counted. I suppose it is, as it's technically correct, but it's also a HUGE source of confusion, particularly for newbies. IF this is being touched anyway, is it possible for the interpreter to know when this error is generated that this is a bound method expecting a "self", rather than an arbitrary function with n parameters? In which case, it would be really nice if the error message replaced that somehow, maybe something like: >>> g(1,2) Traceback (most recent call last): File "", line 1, in TypeError: append() takes 1 positional argument in addition to the automatically added instance object -- but 2 were given in addition to the object instance. Man -- hard to come up with good wording for that -- but SOMETHING that lets users know what they actually did wrong would be good :-) If it's not do-able, then still +1 on making builtins consistent. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Mon Jul 30 14:39:42 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Mon, 30 Jul 2018 20:39:42 +0200 Subject: [Python-Dev] Error message for wrong number of arguments In-Reply-To: References: <5B5F00A6.8070802@UGent.be> Message-ID: <5B5F5B6E.1020300@UGent.be> On 2018-07-30 20:22, Chris Barker wrote: > is it possible for the interpreter to know when this error is > generated that this is a bound method expecting a "self", rather than an > arbitrary function with n parameters? That would be quite hard. The error message is generated by the underlying function. At that point, the information of how it was called (as bound method or not) is already gone. Jeroen. From chris.jerdonek at gmail.com Mon Jul 30 15:45:16 2018 From: chris.jerdonek at gmail.com (Chris Jerdonek) Date: Mon, 30 Jul 2018 12:45:16 -0700 Subject: [Python-Dev] Tests failing on Windows with TESTFN In-Reply-To: <738a82ba-a145-76ba-4447-763d59834379@timgolden.me.uk> References: <18227962-e2e6-d1d7-0fda-597e814de0d2@timgolden.me.uk> <3e5ff496-649d-5107-e242-448993c8fa43@timgolden.me.uk> <738a82ba-a145-76ba-4447-763d59834379@timgolden.me.uk> Message-ID: On Mon, Jul 30, 2018 at 8:46 AM, Tim Golden wrote: > On 30/07/2018 16:41, Nick Coghlan wrote: >> >> On 29 July 2018 at 03:20, Tim Golden wrote: >>> >>> I think that was my starting point: rather than develop increasingly >>> involved and still somewhat brittle mechanisms, why not do what you'd >>> naturally do with a new test and use tempfile? I was expecting someone to >>> come forward to highlight some particular reason why the TESTFN approach >>> is >>> superior, but apart from a reference to the possibly cost of creating a >>> new >>> temporary file per test, no-one really has. >> >> >> For higher level modules, "just use tempfile to create a new temporary >> directory, then unlink it at the end" is typically going to be a good >> answer (modulo the current cleanup issues that Jeremy is reporting, >> but ideally those will be fixed rather than avoided, either by >> improving the way the module is being used, or fixing any underlying >> defects). If there's a desire to use tempfile, another option is to have tempfile create the temp files inside the temporary directory the test harness creates specifically for testing -- using the "dir" argument to many of tempfile's functions. Here is where the process-specific temp directory is created for testing (inside test.libregrtest.main): https://github.com/python/cpython/blob/9045199c5aaeac9b52537581be127d999b5944ee/Lib/test/libregrtest/main.py#L511 This would also facilitate the clean-up of any leftover temp files. Again, I think it would be best to use any tempfile functions behind one or more test-support functions, so the choice of location, etc. can be changed centrally without needing to modify code everywhere. --Chris >> >> For lower level modules though, adding a test suite dependency on >> tempfile introduces a non-trivial chance of adding an operational >> dependency from a module's test suite back to the module itself. When >> that happens, module regressions may show up as secondary failures in >> tempfile that then need to be debugged, rather than as specific unit >> test failures that point you towards exactly what you broke. >> >> Cheers, >> Nick. >> > > Thanks Nick; I hadn't thought about the possible interdependency issue. > > I think for the moment my approach will be to switch to support.unlink > wherever possible to start with. Before introducing other (eg tempfile) > changes, this should at least narrow the issues down. I've made a start on > that (before inadvertently blowing away all the changes since my > hours-previous commit!) > > If further changes are necessary then I'll probably look case-by-case to see > whether a tempfile or some other solution would help. > > That said, that's potentially quite a lot of change -- at least in terms of > files changed if not strictly of functionality. So I'm thinking of > trickle-feeding the changes through as people will understandably baulk at a > patchbomb (PR-bomb?) hitting the codebase all at once. > > TJG > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/chris.jerdonek%40gmail.com From chris.barker at noaa.gov Mon Jul 30 16:19:06 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Mon, 30 Jul 2018 13:19:06 -0700 Subject: [Python-Dev] Error message for wrong number of arguments In-Reply-To: <5B5F5B6E.1020300@UGent.be> References: <5B5F00A6.8070802@UGent.be> <5B5F5B6E.1020300@UGent.be> Message-ID: On Mon, Jul 30, 2018 at 11:39 AM, Jeroen Demeyer wrote: > On 2018-07-30 20:22, Chris Barker wrote: > >> is it possible for the interpreter to know when this error is >> generated that this is a bound method expecting a "self", rather than an >> arbitrary function with n parameters? >> > > That would be quite hard. The error message is generated by the underlying > function. At that point, the information of how it was called (as bound > method or not) is already gone. Thanks, I figured as much. Oh well. This is a serious usability issue -- but what can you do? -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From bpurvy at gmail.com Mon Jul 30 16:40:37 2018 From: bpurvy at gmail.com (Bob Purvy) Date: Mon, 30 Jul 2018 13:40:37 -0700 Subject: [Python-Dev] Accessing mailing list archives Message-ID: hi all, I've been trying to figure out how to access the archives programmatically. I'm sure this is easy once you know, but googling various things hasn't worked. What I want to do is graph the number of messages about PEP 572 by time. (or has someone already done that?) I installed GNU Mailman, and downloaded the gzip'ed archives for a number of months and unzipped them, and I suspect that there's some way to get them all into a single database, but it hasn't jumped out at me. If I count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines in one of those text files, I get slightly different numbers for each. Alternatively, they're maybe *already* in a database, and I just need API access to do the querying? Can someone help me out? Bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Mon Jul 30 18:26:57 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 31 Jul 2018 00:26:57 +0200 Subject: [Python-Dev] Accessing mailing list archives In-Reply-To: References: Message-ID: Hi Bob, I wrote a basic script to compute the number of emails per PEP. It requires to download gzipped mbox files from the web page of archives per month, then ungzip them: https://github.com/vstinner/misc/blob/master/python/parse_mailman_mbox_peps.py Results: https://mail.python.org/pipermail/python-committers/2018-April/005310.html Victor Le lundi 30 juillet 2018, Bob Purvy a ?crit : > hi all, > I've been trying to figure out how to access the archives programmatically. I'm sure this is easy once you know, but googling various things hasn't worked. What I want to do is graph the number of messages about PEP 572 by time. (or has someone already done that?) > I installed GNU Mailman, and downloaded the gzip'ed archives for a number of months and unzipped them, and I suspect that there's some way to get them all into a single database, but it hasn't jumped out at me. If I count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines in one of those text files, I get slightly different numbers for each. > Alternatively, they're maybe already in a database, and I just need API access to do the querying? Can someone help me out? > Bob -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Mon Jul 30 18:32:00 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 31 Jul 2018 00:32:00 +0200 Subject: [Python-Dev] Testing C API In-Reply-To: References: Message-ID: Actually, I mean -m test --match 'test_*_capi' where --match can also be written -m. It may catch false positive since the filter is also applied to test case names and test method names. Maybe 'test_*_capi.*' is better, I didn't try. You may using use "ls Lib/test/test_*_capi.py > tests; ./python -m test --fromfile tests". There are different options. By the way, running the full test suite just takes 5 min on my laptop, it isn't so long ;-) Victor Le lundi 30 juillet 2018, Nick Coghlan a ?crit : > On 30 July 2018 at 21:23, Victor Stinner wrote: >> Or maybe test__capi.py so you can more easily discover >> test_unicode_cami while working on Unicode. You can use -m "test_*_capi" to >> run all C API tests. > > Missing word there: -m test "test*_capi" > > I think between this approach and expanding test_capi to be a test > package, not just a test file, I think it would be possible to make > these parts of the test a fair bit more discoverable and maintainable. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Mon Jul 30 20:25:06 2018 From: greg at krypto.org (Gregory P. Smith) Date: Mon, 30 Jul 2018 17:25:06 -0700 Subject: [Python-Dev] USE_STACKCHECK and running out of stack In-Reply-To: <9EE08C5E-1ACC-45D3-8894-1E01BF57F4BF@mac.com> References: <9EE08C5E-1ACC-45D3-8894-1E01BF57F4BF@mac.com> Message-ID: On Sat, Jul 28, 2018 at 7:51 AM Ronald Oussoren via Python-Dev < python-dev at python.org> wrote: > Hi, > > I?m looking at PyOS_CheckStack because this feature might be useful on > macOS (and when I created bpo-33955 for this someone ran with it and > created a patch). > > Does anyone remember why the interpreter raises MemoryError and not > RecursionError when PyOS_CheckStack detects that we?re about to run out of > stack space? > Running out of C stack makes more sense to me as a MemoryError, you don't know that recursion was the primary cause and the fundamental problem is that you are (about to) run out of a finite memory resource. All you know is that at least one C frame on the C call stack may have required a lot of local stack space. Just as likely as there being tons of regular sized frames. RecursionError is only raised when CPython's arbitrary Python recursion depth limit is reached. It is not raised for memory reasons. In small thread stack space environments (64k, etc) I tended to find problematic code to be C code with local arrays that consumed several kilobytes each rather than actual recursion. Python 2's own import system was notorious for that so if your server is calling service handlers in threads with a small stack and some of those are doing new imports at handler time rather than having done all imports before serving began: boom. stack overflow. -gps -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Mon Jul 30 20:29:42 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 31 Jul 2018 02:29:42 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <20180730110120.6d03e6d8@fsol> References: <20180730110120.6d03e6d8@fsol> Message-ID: >> Last year, I gave a talk at the Language Summit (during Pycon) to >> explain that CPython should become 2x faster to remain competitive. >> IMHO all attempts to optimize Python (CPython forks) have failed >> because they have been blocked by the C API which implies strict >> constraints. > > Well, that's your opinion, but did you prove it? I discussed with many Python developers who agree with me that the C API blocked CPython forks. For example, I heard that Pyston was very fast and very promising before starting to support the C API. The C API requires that your implementations make almost all the same design choices that CPython made 25 years ago (C structures, memory allocators, reference couting, specific GC implementation, GIL, etc.). More efficient technonogies appeared in the meanwhile. Multiple PyPy developers told me that cpyext remains a blocker issue to use PyPy. I am not sure how I am supposed to "prove" these facts. Oh, by the way, I will not promise anything about any potential performance gain. When I write "2x faster", I mean that our current approach for optimization failed to make Python 2x faster over the last 10 years. Python 3 is more or less as fast, or a little bit faster, than Python 2. But Python 2 must not be used as an example of performance. People hesitate between Go, Javascript and Python. And Python is not the winner in term of performance. > The one area I know of where the C API is a strong barrier to > improvement is removing the GIL, and I'd rather let Larry speak about > it. IMHO Gilectomy is going to fail without our help to change the C API. Again, I don't want to promise anything here. Removing reference counting inside CPython is a giant project. But at least, I know that borrowed references are very complex to support if CPython doesn't use reference counting. I heard that PyPy has issues to implement borrowed references. If we succeed to remove them, PyPy should benefit directly of that work. Note: PyPy will still have to support borrowed references for C extensions usong the old C API. But I expect that PyPy will be more reliable, maybe even faster, to run C extensions using the new C API without reference counting. I have to confess that helping Larry is part of my overall plan. But I dislke making promise that I cannot do, and I dislike working on giant multiyear Python roject. My hope is to have a working new C API next year which will hide some implementation details, but not all of them. I want to work incrementally using popular C extensions in the feedback loop. Building a new C API is useless if nobody can use it. But it will take time to adjust the backward compatibility cursor. Victor -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Tue Jul 31 02:58:16 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 31 Jul 2018 08:58:16 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <20180730110120.6d03e6d8@fsol> Message-ID: <20180731085816.7bf79326@fsol> On Tue, 31 Jul 2018 02:29:42 +0200 Victor Stinner wrote: > >> Last year, I gave a talk at the Language Summit (during Pycon) to > >> explain that CPython should become 2x faster to remain competitive. > >> IMHO all attempts to optimize Python (CPython forks) have failed > >> because they have been blocked by the C API which implies strict > >> constraints. > > > > Well, that's your opinion, but did you prove it? > > I discussed with many Python developers who agree with me that the C API > blocked CPython forks. For example, I heard that Pyston was very fast and > very promising before starting to support the C API. What exactly in the C API made it slow or non-promising? > The C API requires that your implementations make almost all the same > design choices that CPython made 25 years ago (C structures, memory > allocators, reference couting, specific GC implementation, GIL, etc.). Yes, but those choices are not necessarily bad. > Multiple PyPy developers told me that cpyext remains a blocker issue to use > PyPy. Probably, but we're talking about speeding up CPython here, right? If we're talking about making more C extensions PyPy-compatible, that's a different discussion, and one where I think Stefan is right that we should push people towards Cython and alternatives, rather than direct use of the C API (which people often fail to use correctly, in my experience). But the C API is still useful for specialized uses, *including* for development tools such as Cython. > Oh, by the way, I will not promise anything about any potential performance > gain. When I write "2x faster", I mean that our current approach for > optimization failed to make Python 2x faster over the last 10 years. Python > 3 is more or less as fast, or a little bit faster, than Python 2. But > Python 2 must not be used as an example of performance. People hesitate > between Go, Javascript and Python. And Python is not the winner in term of > performance. I agree about the overall diagnosis. I just disagree that changing the C API will open up easy optimization opportunities. Actually I'd like to see a list of optimizations that you think are held up by the C API. > I have to confess that helping Larry is part of my overall plan. Which is why I'd like to see Larry chime in here. Regards Antoine. From J.Demeyer at UGent.be Tue Jul 31 03:27:03 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 31 Jul 2018 09:27:03 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> References: <20180730110120.6d03e6d8@fsol> <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> Message-ID: <5B600F47.3090503@UGent.be> On 2018-07-31 08:58, Antoine Pitrou wrote: > I think Stefan is right that we > should push people towards Cython and alternatives, rather than direct > use of the C API (which people often fail to use correctly, in my > experience). I know this probably isn't the correct place to bring it up, but I'm sure that CPython itself could benefit from using Cython. For example, most of the C extensions in Modules/ could be written in Cython. From songofacandy at gmail.com Tue Jul 31 03:36:27 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 31 Jul 2018 16:36:27 +0900 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 Message-ID: First of all, I'm sorry to I forgot change my mail title. (I though about reserving one more slot for Cython for further Cython-to-Cython call optimization, but I rejected my idea because I'm not sure it really help Cython.) On Mon, Jul 30, 2018 at 11:55 PM Jeroen Demeyer wrote: > > On 2018-07-30 15:35, INADA Naoki wrote: > > As repeatedly said, PEP 580 is very complicated protocol > > when just implementing callable object. > > Can you be more concrete what you find complicated? Maybe I can improve > the PEP to explain it more. Also, I'm open to suggestions to make it > less complicated. When thinking from extension writer's point of view, almost all of PEP 580 is complicated comparing PEP 576. Remember they don't need custom method/function type. So PEP 576/580 are needed only when implementing callable object, like itemgetter or lru_cache in stdlib. * We continue to use PyMethodDef and METH_* when writing tp_methods. They should learn PyCCallDef and CCALL_* flags in addition to PyMethodDef and METH_*. * In PEP 576, just put function pointer to type slot. On the other hand, when implementing callable object with PEP 580, (1) Put PyCCallDef somewhere, (2) Put CCallRoot in instance, (3) put offset of (2) to tp_ccall. * Difference between cc_parent and cc_self are unclear too. I think PEP 580 is understandable only for people who tried to implement method objects. It's complete rewrite of PyCFunction and method_descriptor. But extension author can write extension without knowing implementation of them. > > > It is optimized for implementing custom method object, although > > almost only Cython want the custom method type. > > For the record, Numba also seems interested in the PEP: > https://groups.google.com/a/continuum.io/d/msg/numba-users/2G6k2R92MIM/P-cFKW7xAgAJ > OK, Numba developer interested in: * Supporting FASTCALL for Dispatcher type: PEP 576 is more simple for it as I described above. * Direct C function calling (skip PyObject calling abstraction). While it's not part of PEP 580, it's strong motivation for PEP 580. I want to see PoC of direct C calling. And I think PoC can be implemented without waiting PEP 580. * Cython can have specialization for CyFunction, like it have for CFunction. (Note that Cython doesn't utilize LOAD_METHOD / CALL_METHOD for CFunction too. So lacking support for LOAD_METHOD / CALL_METHOD is not a big problem for now.) * Cython can implement own C signature and embed it in CyFunction. After that, we (including Numba, Cython, and PyPy developers) can discuss how portable C signature can be embedded in PyCCallDef. > > I'm not sure adding such complicated protocol almost only for Cython. > > If CyFunction can be implemented behind PEP 576, it may be better. > > I recall my post > https://mail.python.org/pipermail/python-dev/2018-July/154238.html > explaining the main difference between PEP 576 and PEP 580. I wrote my mail after reading the mail, of course. But it was unclear without reading PEP and implementation carefully. For example, "hook which part" seems meta-discussion to me before reading your implementation. I think only way to understand PEP 580 is reading implementation and imagine how Cython and Numba use it. > > I would like to stress that PEP 580 was designed for maximum > performance, both today and for future extensions (such as calling with > native C types). > I don't know what the word *stress* mean here. (Sorry, I'm not good at English enough for such hard discussion). But I want to see PoC of real benefit of PEP 580, as I said above. > > * PEP 576 and 580 are not strictly mutually exclusive; PEP 576 may be > > accepted in addition to PEP 580 > > I don't think that this is a good idea: you will mostly end up with the > disadvantages of both approaches. > Hm, My point was providing easy and simple way to support FASTCALL in callable object like functools.partial or functools.lru_cache. But it should be discussed after PEP 580. -- INADA Naoki From solipsis at pitrou.net Tue Jul 31 03:45:28 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 31 Jul 2018 09:45:28 +0200 Subject: [Python-Dev] Let's change to C API! References: <20180730110120.6d03e6d8@fsol> <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> <5B600F47.3090503@UGent.be> Message-ID: <20180731094528.118471f9@fsol> On Tue, 31 Jul 2018 09:27:03 +0200 Jeroen Demeyer wrote: > On 2018-07-31 08:58, Antoine Pitrou wrote: > > I think Stefan is right that we > > should push people towards Cython and alternatives, rather than direct > > use of the C API (which people often fail to use correctly, in my > > experience). > > I know this probably isn't the correct place to bring it up, but I'm > sure that CPython itself could benefit from using Cython. For example, > most of the C extensions in Modules/ could be written in Cython. We don't depend on any third-party Python modules. Adding a Cython dependency for CPython development would be a tough sell. Also, a C extension can be built-in (linked statically into the interpreter), which I think would be hard to do with Cython. Regards Antoine. From J.Demeyer at UGent.be Tue Jul 31 04:46:39 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 31 Jul 2018 10:46:39 +0200 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: References: Message-ID: <5B6021EF.5080908@UGent.be> On 2018-07-31 09:36, INADA Naoki wrote: > I want to see PoC of direct C calling. To be honest, there is no implementation plan for this yet. I know that several people want this feature, so it makes sense to think about it. For me personally, the main open problem is how to deal with arguments which may be passed both as Python object or as native C type. For example, when doing a function call like f(1,2,3), it may happen that the first argument is really a Python object (so it should be passed as Python int) but that the other two arguments are C integers. > And I think PoC can be implemented without waiting PEP 580. For one particular class (say CyFunction), yes. But this feature would be particularly useful for calling between different kinds of C code, for example between Numba and CPython built-ins, or between Pythran and Cython, ... That is why I think it should be implemented as an extension of PEP 580. Anyway, this is a different subject that we should not mix in the discussion about PEP 580 (that is also why I am replying to this specific point separately). Jeroen. From J.Demeyer at UGent.be Tue Jul 31 05:07:26 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 31 Jul 2018 11:07:26 +0200 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: References: Message-ID: <5B6026CE.70901@UGent.be> On 2018-07-31 09:36, INADA Naoki wrote: > I think PEP 580 is understandable only for people who tried to implement > method objects. Is this really a problem? Do we expect that all Python developers can understand all PEPs, especially on a technical subject like this? To give a different example, I would say that PEP 567 is also quite technical and not understandable by people who don't care about about context variables. If PEP 580 is accepted, we can make it very clear in the documentation that this is only meant for implementing fast function/method classes and that ordinary "extension writers" can safely skip that part. For example, you write > They should learn PyCCallDef and CCALL_* flags in addition > to PyMethodDef and METH_*. but that's not true: they can easily NOT learn those flags, just like they do NOT need to learn about context variables if they don't need them. >> I would like to stress that PEP 580 was designed for maximum >> performance, both today and for future extensions (such as calling with >> native C types). >> > > I don't know what the word *stress* mean here. (Sorry, I'm not good at English > enough for such hard discussion). > But I want to see PoC of real benefit of PEP 580, as I said above. "to stress" = to draw attention to, to make it clear that So, PEP 580 is meant to keep all existing optimizations for functions/methods. It can also be extended in the future (for example, to support direct C calling) by just adding extra flags and structure fields to PyCCallDef. > Hm, My point was providing easy and simple way to support FASTCALL > in callable object like functools.partial or functools.lru_cache. That can be done easily with only PEP 580. Jeroen. From songofacandy at gmail.com Tue Jul 31 05:12:11 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 31 Jul 2018 18:12:11 +0900 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: <5B6021EF.5080908@UGent.be> References: <5B6021EF.5080908@UGent.be> Message-ID: On Tue, Jul 31, 2018 at 5:46 PM Jeroen Demeyer wrote: > > On 2018-07-31 09:36, INADA Naoki wrote: > > I want to see PoC of direct C calling. > > To be honest, there is no implementation plan for this yet. I know that > several people want this feature, so it makes sense to think about it. Yes. It will be the strong benefit of PEP 580. > > For me personally, the main open problem is how to deal with arguments > which may be passed both as Python object or as native C type. For > example, when doing a function call like f(1,2,3), it may happen that > the first argument is really a Python object (so it should be passed as > Python int) but that the other two arguments are C integers. I don't think it's the main point... At first time, we can just skip direct calling path and use normal Python call. > > > And I think PoC can be implemented without waiting PEP 580. > > For one particular class (say CyFunction), yes. But this feature would > be particularly useful for calling between different kinds of C code, > for example between Numba and CPython built-ins, or between Pythran and > Cython, ... > > That is why I think it should be implemented as an extension of PEP 580. Of course, finally it can be implemented based on PEP 580. But I said "Proof of Concept". Optimize only Cython-to-Cython case is enough for "Proof of Concept". PoC makes it easy to discuss concrete API design for cross implementation call. > > Anyway, this is a different subject that we should not mix in the > discussion about PEP 580 (that is also why I am replying to this > specific point separately). > For me, this is the most important benefit of PEP 580. I can't split it from PEP 580. And I didn't say PoC is needed *before* accepting or discussing PEP 580. I just meant PoC can be implemented without waiting PEP 580 accepted, and I want see it early. Any PEP won't be accepted in few month, because we don't have flow to accept PEPs for now. And I think Victor and Serhiy are key person about these PEPs, but I don't know when they have enough time to review / discuss / make decision about these PEPs. So it's worthless that waiting PEP accepted before start PoC. -- INADA Naoki From J.Demeyer at UGent.be Tue Jul 31 05:39:45 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 31 Jul 2018 11:39:45 +0200 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: References: <5B6021EF.5080908@UGent.be> Message-ID: <5B602E61.7020107@UGent.be> On 2018-07-31 11:12, INADA Naoki wrote: > For me, this is the most important benefit of PEP 580. I can't split > it from PEP 580. I want PEP 580 to stand by itself. And you say that it is already complicated enough, so we should not mix native C calling into it. PEP 580 is written to allow future extensions like that, but it should be reviewed without those future extensions. Jeroen. From songofacandy at gmail.com Tue Jul 31 06:10:48 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 31 Jul 2018 19:10:48 +0900 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: <5B6026CE.70901@UGent.be> References: <5B6026CE.70901@UGent.be> Message-ID: On Tue, Jul 31, 2018 at 6:14 PM Jeroen Demeyer wrote: > > On 2018-07-31 09:36, INADA Naoki wrote: > > I think PEP 580 is understandable only for people who tried to implement > > method objects. > > Is this really a problem? Do we expect that all Python developers can > understand all PEPs, especially on a technical subject like this? > > To give a different example, I would say that PEP 567 is also quite > technical and not understandable by people who don't care about about > context variables. > Please don't "straw man" discussion! I said "So PEP 576/580 are needed only when implementing callable object". I showed example "functools.partial or functools.lru_cache". Using FASTCALL for callable object, instead of tp_call is one of important goal of PEP 576 and 580. So, cleary, ordinal extension author should be able to understand and implement PEP 576 and 580. > If PEP 580 is accepted, we can make it very clear in the documentation > that this is only meant for implementing fast function/method classes > and that ordinary "extension writers" can safely skip that part. For > example, you write > > > They should learn PyCCallDef and CCALL_* flags in addition > > to PyMethodDef and METH_*. > > but that's not true: they can easily NOT learn those flags, just like > they do NOT need to learn about context variables if they don't need them. > Surely, they should understand they must use CCALL_* flags instead of METH_* flags when implementing fast-callable object. > >> I would like to stress that PEP 580 was designed for maximum > >> performance, both today and for future extensions (such as calling with > >> native C types). > >> > > I don't know what the word *stress* mean here. (Sorry, I'm not good at English > > enough for such hard discussion). > > But I want to see PoC of real benefit of PEP 580, as I said above. > > "to stress" = to draw attention to, to make it clear that > OK, I'm very drawn attention to that already. > > Hm, My point was providing easy and simple way to support FASTCALL > > in callable object like functools.partial or functools.lru_cache. > > That can be done easily with only PEP 580. > After spent several days to read PEP 580 and your implementation, I think I can implement it. I think it's not easy, but it's not impossible too. But I think we should admit the fact it's complicated than PEP 576. Although I don't think it's important than some benefits of PEP 580. NOTE that I didn't say it's a big problem. I just explained because you said "Can you be more concrete what you find complicated?" -- INADA Naoki From solipsis at pitrou.net Tue Jul 31 06:27:29 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 31 Jul 2018 12:27:29 +0200 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 References: <5B6026CE.70901@UGent.be> Message-ID: <20180731122729.054fb374@fsol> On Tue, 31 Jul 2018 19:10:48 +0900 INADA Naoki wrote: > > I said "So PEP 576/580 are needed only when implementing callable object". > I showed example "functools.partial or functools.lru_cache". > > Using FASTCALL for callable object, instead of tp_call is one of important goal > of PEP 576 and 580. > > So, cleary, ordinal extension author should be able to understand and implement > PEP 576 and 580. No, Jeroen is right here. Ordinary extension authors do not need to understand and implement PEP 576 and 580. First, it's rare for extension types to be callable. Second, even if an extension type is callable, implementing PEP 576 or 580 is only useful if tp_call overhead is really important. In any case, just using tp_call will be as good as it was before. There is no regression, hence it is incorrect to say that we place an additional burden on the shoulders of extension authors. Yes, they can learn an additional PEP to increase tp_call performance; but they don't have to. Regards Antoine. From songofacandy at gmail.com Tue Jul 31 06:44:53 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 31 Jul 2018 19:44:53 +0900 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: <20180731122729.054fb374@fsol> References: <5B6026CE.70901@UGent.be> <20180731122729.054fb374@fsol> Message-ID: On Tue, Jul 31, 2018 at 7:27 PM Antoine Pitrou wrote: > > On Tue, 31 Jul 2018 19:10:48 +0900 > INADA Naoki wrote: > > > > I said "So PEP 576/580 are needed only when implementing callable object". > > I showed example "functools.partial or functools.lru_cache". > > > > Using FASTCALL for callable object, instead of tp_call is one of important goal > > of PEP 576 and 580. > > > > So, cleary, ordinal extension author should be able to understand and implement > > PEP 576 and 580. > > No, Jeroen is right here. Ordinary extension authors do not need to > understand and implement PEP 576 and 580. Ah, my wording "ordinary" was wrong. But Jeroen was wrong too. He said "PEP 567 is also quite technical and not understandable by people who don't care about context variables." People can skip PEP 567 easily when they don't care about contextvars. On the other hand, *if we accept PEP 576 over PEP 580", C extension author will use it easily. So when comparing PEP 576 and 580, understandability for C extension author is not out of the scope. Note that I didn't claim how important it is. I just compare design of PEP 576 and PEP 580. > > First, it's rare for extension types to be callable. > Second, even if an extension type is callable, implementing PEP 576 or > 580 is only useful if tp_call overhead is really important. > > In any case, just using tp_call will be as good as it was before. > There is no regression, hence it is incorrect to say that we place an > additional burden on the shoulders of extension authors. Yes, they > can learn an additional PEP to increase tp_call performance; but they > don't have to. I agree here. I have grepped tp_call usage and about to send mail explaining "Why FASTCALL for tp_call is not so important". > > Regards > > Antoine. > -- INADA Naoki From vstinner at redhat.com Tue Jul 31 06:51:23 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 31 Jul 2018 12:51:23 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <20180731085816.7bf79326@fsol> References: <20180730110120.6d03e6d8@fsol> <20180731085816.7bf79326@fsol> Message-ID: 2018-07-31 8:58 GMT+02:00 Antoine Pitrou : > What exactly in the C API made it slow or non-promising? > >> The C API requires that your implementations make almost all the same >> design choices that CPython made 25 years ago (C structures, memory >> allocators, reference couting, specific GC implementation, GIL, etc.). > > Yes, but those choices are not necessarily bad. I understood that PyPy succeeded to become at least 2x faster than CPython by stopping to use reference counting internally. >> Multiple PyPy developers told me that cpyext remains a blocker issue to use >> PyPy. > > Probably, but we're talking about speeding up CPython here, right? My project has different goals. I would prefer to not make any promise about speed. So speed is not my first motivation, or at least not the only one :-) I also want to make the debug build usable. I also want to allow OS vendors to provide multiple Python versions per OS release: *reduce* the maintenance burden, obviously it will still mean more work. It's a tradeoff depending on the lifetime of your OS and the pressure of customers to get the newest Python :-) FYI Red Hat already provide recent development tools on top of RHEL (and Centos and Fedora) because customers are asking for that. We don't work for free :-) I also want to see more alternatives implementations of Python! I would like to see RustPython succeed! See the latest version of https://pythoncapi.readthedocs.io/ for the full rationale. > If we're talking about making more C extensions PyPy-compatible, that's > a different discussion, For pratical reasons, IMHO it makes sense to put everything in the same "new C API" bag. Obviously, I propose to make many changes, and some of them can be more difficult to implement. My proposal contains many open questions and is made of multiple milestones, with a strong requirement on backward compatibility. > and one where I think Stefan is right that we > should push people towards Cython and alternatives, rather than direct > use of the C API (which people often fail to use correctly, in my > experience). Don't get me wrong: my intent is not to replace Cython. Even if PyPy is pushing hard cffi, many C extensions still use the C API. Maybe if the C API becomes more annoying and require developers to adapt their old code base for the "new C API", some of them will reconsider to use Cython, cffi or something else :-D But backward compatibility is a big part of my plan, and in fact, I expect that porting most C extensions to the new C API will be between "free" and "cheap". Obviously, it depends on how much changes we put in the "new C API" :-) I would like to work incrementally. > But the C API is still useful for specialized uses, *including* for > development tools such as Cython. It seems like http://pythoncapi.readthedocs.io/ didn't explain well my intent. I updated my doc to make it very clear that the "old C API" remains available *on purpose*. The main question is if you will be able to use Cython with the "old C API" on a new "experimental runtime", or if Cython will be stuck at the "regular runtime". https://pythoncapi.readthedocs.io/runtimes.html It's just that for the long term (end of my roadmap), you will have to opt-in for the old C API. > I agree about the overall diagnosis. I just disagree that changing the > C API will open up easy optimization opportunities. Ok, please help me to rephrase the documentation to not make any promise :-) Currently, I wrote: """ Optimization ideas Once the new C API will succeed to hide implementation details, it becomes possible to experiment radical changes in CPython to implement new optimizations. See Experimental runtime. """ https://pythoncapi.readthedocs.io/optimization_ideas.html In my early plan, I wrote "faster runtime". I replaced it with "experimental runtime" :-) Do you think that it's wrong to promise that a smaller C API without implementation details will allow to more easily *experiment* optimizations? > Actually I'd like to see a list of optimizations that you think are > held up by the C API. Hum, let me use the "Tagged pointers" example. Most C functions use "PyObject*" as an opaque C type. Good. But technically, since we give access to fields of C structures, like PyObject.ob_refcnt or PyListObject.ob_item, C extensions currently dereference directly pointers. I'm not convinced that tagged pointers will make CPython way faster. I'm just saying that the C API prevents you to even experiment such change to measure the impact on performance. https://pythoncapi.readthedocs.io/optimization_ideas.html#tagged-pointers-doable For the "Copy-on-Write" idea, the issue is that many macros access directly fields of C structures and so at the machine code, the ABI uses a fixed offset in memory to read data, whereas my plan is to allow each runtime to use a different memory layout, like putting Py_GC elsewhere (or even remove it!!!) and/or put ob_refcnt elsewhere. https://pythoncapi.readthedocs.io/optimization_ideas.html#copy-on-write-cow-doable >> I have to confess that helping Larry is part of my overall plan. > > Which is why I'd like to see Larry chime in here. I already talked a little bit with Larry about my plan, but he wasn't sure that my plan is enough to be able to stop reference counting internally and move to a different garbage collector. I'm only sure that it's possible to keep using reference counting for the C API, since there are solutions for that (ex: maintain a hash table PyObject* => reference count). Honestly, right now, I'm only convinvced of two things: * Larry implementation is very complex and so I doubt that he is going to succeed. I'm talking about solutions to maintain optimize reference counting in multithreaded applications. Like his idea of "logs" of reference counters. * We have to change the C API: it causes troubles to *everybody*. Nobody spoke up because changing the C API is a giant project and it breaks the backward compatibility. But I'm not sure that all victims of the C API are aware that their issues are caused by the design of the current C API. Victor From vstinner at redhat.com Tue Jul 31 06:56:04 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 31 Jul 2018 12:56:04 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <5B600F47.3090503@UGent.be> References: <20180730110120.6d03e6d8@fsol> <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> <5B600F47.3090503@UGent.be> Message-ID: 2018-07-31 9:27 GMT+02:00 Jeroen Demeyer : > On 2018-07-31 08:58, Antoine Pitrou wrote: >> >> I think Stefan is right that we >> should push people towards Cython and alternatives, rather than direct >> use of the C API (which people often fail to use correctly, in my >> experience). > > > I know this probably isn't the correct place to bring it up, but I'm sure > that CPython itself could benefit from using Cython. For example, most of > the C extensions in Modules/ could be written in Cython. CPython build system has very little dependencies. We even include vendored copies of third party libraries to make the build even simpler: http://pythondev.readthedocs.io/cpython.html#vendored-external-libraries We try to make CPython build as simple as possible. I'm quite sure that Cython rely on the stdlib. Would depending on Cython open a chicken-and-egg issue? I would be nice to be able to use something to "generate" C extensions, maybe even from pure Python code. But someone has to work on a full solution to implement that. The statu co is that CPython uses C extensions calling directly the C API. Some people complained that CPython doesn't use its own stable ABI for its own stable ABI. I concur that it's an issue. Because of that, nobody noticed that we broke the stable ABI (we did it, multiple times...). Hum, maybe I should explain that my plan is also try to use the "new C API" for some C extensions of the stdlib. I'm not sure if we can do it for all C extensions, since performance matters, and sometimes we really need to access private fields ("implementation details"). Victor From solipsis at pitrou.net Tue Jul 31 07:55:45 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 31 Jul 2018 13:55:45 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <20180730110120.6d03e6d8@fsol> <20180731085816.7bf79326@fsol> Message-ID: <20180731135545.24c4d427@fsol> On Tue, 31 Jul 2018 12:51:23 +0200 Victor Stinner wrote: > 2018-07-31 8:58 GMT+02:00 Antoine Pitrou : > > What exactly in the C API made it slow or non-promising? > > > >> The C API requires that your implementations make almost all the same > >> design choices that CPython made 25 years ago (C structures, memory > >> allocators, reference couting, specific GC implementation, GIL, etc.). > > > > Yes, but those choices are not necessarily bad. > > I understood that PyPy succeeded to become at least 2x faster than > CPython by stopping to use reference counting internally. "I understood that"... where did you get it from? :-) > I also want to make the debug build usable. So I think that we should ask what the ABI differences between debug and non-debug builds are. AFAIK, the two main ones are Py_TRACE_REFS and Py_REF_DEBUG. Are there any others? Honestly, I don't think Py_TRACE_REFS is useful. I don't remember any bug being discovered thanks to it. Py_REF_DEBUG is much more useful. The main ABI issue with Py_REF_DEBUG is not object structure (it doesn't change object structure), it's when a non-debug extension steals a reference (or calls a reference-stealing C API function), because then increments and decrements are unbalanced. > I also want to allow OS vendors to provide multiple Python versions > per OS release: *reduce* the maintenance burden, obviously it will > still mean more work. It's a tradeoff depending on the lifetime of > your OS and the pressure of customers to get the newest Python :-) FYI > Red Hat already provide recent development tools on top of RHEL (and > Centos and Fedora) because customers are asking for that. We don't > work for free :-) OS vendors seem to be doing a fine job AFAICT. And if I want a recent Python I just download Miniconda/Anaconda. > I also want to see more alternatives implementations of Python! I > would like to see RustPython succeed! As long as RustPython gets 10 commits a year, it has no chance of being a functional Python implementation, let alone a successful one. AFAICS it's just a toy project. > > and one where I think Stefan is right that we > > should push people towards Cython and alternatives, rather than direct > > use of the C API (which people often fail to use correctly, in my > > experience). > > Don't get me wrong: my intent is not to replace Cython. Even if PyPy > is pushing hard cffi, many C extensions still use the C API. cffi is a ctypes replacement. It's nice when you want to bind with foreign C code, not if you want tight interaction with CPython objects. > Maybe if the C API becomes more annoying and require developers to > adapt their old code base for the "new C API", some of them will > reconsider to use Cython, cffi or something else :-D I think you don't realize that the C API is *already* annoying. People started with it mostly because there wasn't a better alternative at the time. You don't need to make it more annoying than it already is ;-) Replacing existing C extensions with something else is entirely a developer time/effort problem, not an attractivity problem. And I'm not sure that porting a C extension to a new C API is more reasonable than porting to Cython entirely. > Do you think that it's wrong to promise that a smaller C API without > implementation details will allow to more easily *experiment* > optimizations? I don't think it's wrong. Though as long as CPython itself uses the internal C API, you'll still have a *lot* of code to change before you can even launch a functional interpreter and standard library... It's just that I disagree that removing the C API will make CPython 2x faster. Actually, important modern optimizations for dynamic languages (such as inlining, type specialization, inline caches, object unboxing) don't seem to depend on the C API at all. > >> I have to confess that helping Larry is part of my overall plan. > > > > Which is why I'd like to see Larry chime in here. > > I already talked a little bit with Larry about my plan, but he wasn't > sure that my plan is enough to be able to stop reference counting > internally and move to a different garbage collector. I'm only sure > that it's possible to keep using reference counting for the C API, > since there are solutions for that (ex: maintain a hash table > PyObject* => reference count). Theoretically possible, but the cost of reference counting will go through the roof if you start using a hash table. > Honestly, right now, I'm only convinvced of two things: > > * Larry implementation is very complex and so I doubt that he is going > to succeed. I'm talking about solutions to maintain optimize reference > counting in multithreaded applications. Like his idea of "logs" of > reference counters. Well, you know, *any* solution is going to be very complex. Switching to a full GC for a runtime (CPython) which can allocate hundreds of thousands of objects per second will require a lot of optimization work as well. > * We have to change the C API: it causes troubles to *everybody*. > Nobody spoke up because changing the C API is a giant project and it > breaks the backward compatibility. But I'm not sure that all victims > of the C API are aware that their issues are caused by the design of > the current C API. I fully agree that the C API is not very nice to play with. The diversity of calling / error return conventions is one annoyance. Borrowed references and reference stealing is another. Getting reference counting right on all code paths is often delicate. So I'm all for sanitizing the C API, and slowly deprecating old patterns. And I think we should push people towards Cython for most current uses of the C API. Regards Antoine. From J.Demeyer at UGent.be Tue Jul 31 08:01:25 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 31 Jul 2018 14:01:25 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <7401f25c721941799a8dc9a288ad1ce8@xmail101.UGent.be> References: <20180730110120.6d03e6d8@fsol> <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> <5B600F47.3090503@UGent.be> <7401f25c721941799a8dc9a288ad1ce8@xmail101.UGent.be> Message-ID: <5B604F95.3040705@UGent.be> On 2018-07-31 12:56, Victor Stinner wrote: > We try to make CPython build as simple as possible. I'm quite sure > that Cython rely on the stdlib. It does rely on modules like "re" and "functools". > Would depending on Cython open a > chicken-and-egg issue? Yes, that's a problem but it's not unsolvable. For example, we could use the PEP 399 pure Python modules for running Cython. Or we could keep certain "core" C modules (which are used by Cython) implemented directly in C. Note that Cython is not all-or-nothing: it is easy to mix pure Python modules, Cython modules and pure C modules. You can also combine pure C code and Cython code in the same module. Anyway, I know that this is probably not going to happen, but I just wanted to bring it up in case people would find it a great idea. But maybe not many CPython core developers actually know and use Cython? > I would be nice to be able to use something to "generate" C > extensions, maybe even from pure Python code. Cython has a "pure Python mode" which does exactly that. There are several ways to include typing information, to ensure that a module remains Python-compatible but can be compiled by Cython in an optimized way. Jeroen. From J.Demeyer at UGent.be Tue Jul 31 08:55:26 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 31 Jul 2018 14:55:26 +0200 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: References: <5B6026CE.70901@UGent.be> Message-ID: <5B605C3E.2070004@UGent.be> On 2018-07-31 12:10, INADA Naoki wrote: > Surely, they should understand they must use CCALL_* flags instead of > METH_* flags when implementing fast-callable object. Yes indeed. But implementing a fast-callable object is quite specialized, not something that ordinary extension writers would care about. And if they don't care about performance, tp_call remains supported. More generally: with PEP 580, everything from the documented C API remains supported. So people can write extensions exactly as before. They only need to care about PEP 580 if they want to use the new features that PEP 580 adds (or if they used undocumented internals). Jeroen. From vstinner at redhat.com Tue Jul 31 09:34:05 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 31 Jul 2018 15:34:05 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <20180731135545.24c4d427@fsol> References: <20180730110120.6d03e6d8@fsol> <20180731085816.7bf79326@fsol> <20180731135545.24c4d427@fsol> Message-ID: Antoine: would you mind to subscribe to the capi-sig mailing list? As expected, they are many interesting points discussed here, but I would like to move all C API discussions to capi-sig. I only continue on python-dev since you started here (and ignored my request to start discussing my idea on capi-sig :-)). 2018-07-31 13:55 GMT+02:00 Antoine Pitrou : >> I understood that PyPy succeeded to become at least 2x faster than >> CPython by stopping to use reference counting internally. > > "I understood that"... where did you get it from? :-) I'm quite sure that PyPy developers told me that, but I don't recall who nor when. I don't think that PyPy became 5x faster just because of a single change. But I understand that to be able to implement some optimizations, you first have to remove constraints caused by a design choice like reference counting. For example, PyPy uses different memory allocators depending on the scope and the lifetime of an object. I'm not sure that you can implement such optimization if you are stuck with reference counting. > So I think that we should ask what the ABI differences between debug > and non-debug builds are. Debug build is one use case. Another use case for OS vendors is to compile a C extension once (ex: on Python 3.6) and use it on multiple Python versions (3.7, 3.8, etc.). > AFAIK, the two main ones are Py_TRACE_REFS and Py_REF_DEBUG. Are there > any others? No idea. > Honestly, I don't think Py_TRACE_REFS is useful. I don't remember > any bug being discovered thanks to it. Py_REF_DEBUG is much more > useful. The main ABI issue with Py_REF_DEBUG is not object structure > (it doesn't change object structure), it's when a non-debug extension > steals a reference (or calls a reference-stealing C API function), > because then increments and decrements are unbalanced. About Py_REF_DEBUG:_Py_RefTotal counter is updated at each INCREF/DECREF. _Py_RefTotal is a popular feature of debug build, and I'm not sure how we can update it without replacing Py_INCREF/DECREF macros with function calls. I'm ok to remove/deprecate Py_TRACE_REFS feature if nobody uses it. > OS vendors seem to be doing a fine job AFAICT. And if I want a recent > Python I just download Miniconda/Anaconda. Is it used in production to deploy services? Or is it more used by developers? I never used Anaconda. > cffi is a ctypes replacement. It's nice when you want to bind with > foreign C code, not if you want tight interaction with CPython objects. I have been told that cffi is a different way to do the same thing. Instead of writing C code with the C API glue, only write C code, and then write a cffi binding for it. But I never used Cython nor cffi, so I'm not sure which one is the most appropriate depending on the use case. > I think you don't realize that the C API is *already* annoying. People > started with it mostly because there wasn't a better alternative at the > time. You don't need to make it more annoying than it already is ;-) > > Replacing existing C extensions with something else is entirely a > developer time/effort problem, not an attractivity problem. And I'm > not sure that porting a C extension to a new C API is more reasonable > than porting to Cython entirely. Do you think that it's doable to port numpy to Cython? It's made of 255K lines of C code. A major "rewrite" of such large code base is very difficult since people want to push new things in parallel. Or is it maybe possible to do it incrementally? > It's just that I disagree that removing the C API will make CPython 2x > faster. How can we make CPython 2x faster? Why everybody, except of PyPy, failed to do that? Victor From vstinner at redhat.com Tue Jul 31 09:34:58 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 31 Jul 2018 15:34:58 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <5B604F95.3040705@UGent.be> References: <20180730110120.6d03e6d8@fsol> <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> <5B600F47.3090503@UGent.be> <7401f25c721941799a8dc9a288ad1ce8@xmail101.UGent.be> <5B604F95.3040705@UGent.be> Message-ID: 2018-07-31 14:01 GMT+02:00 Jeroen Demeyer : > Anyway, I know that this is probably not going to happen, but I just wanted > to bring it up in case people would find it a great idea. But maybe not many > CPython core developers actually know and use Cython? I know that Yury wants to use Cython for "C extensions" of CPython. Victor From J.Demeyer at UGent.be Tue Jul 31 09:35:28 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 31 Jul 2018 15:35:28 +0200 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: References: <5B6026CE.70901@UGent.be> Message-ID: <5B6065A0.8020904@UGent.be> On 2018-07-31 12:10, INADA Naoki wrote: > After spent several days to read PEP 580 and your implementation, I think > I can implement it. I think it's not easy, but it's not impossible too. The signature of "extended_call_ptr" in PEP 576 is almost the same as the signature of a CCALL_FUNCARG|CCALL_FASTCALL|CCALL_KEYWORDS function in PEP 580 (the only difference is a "self" argument which can be ignored if you don't need it). So, if you can implement it using PEP 576, it's not a big step to implement it using PEP 580. From songofacandy at gmail.com Tue Jul 31 10:00:13 2018 From: songofacandy at gmail.com (INADA Naoki) Date: Tue, 31 Jul 2018 23:00:13 +0900 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: <5B605C3E.2070004@UGent.be> References: <5B6026CE.70901@UGent.be> <5B605C3E.2070004@UGent.be> Message-ID: On Tue, Jul 31, 2018 at 9:55 PM Jeroen Demeyer wrote: > > On 2018-07-31 12:10, INADA Naoki wrote: > > Surely, they should understand they must use CCALL_* flags instead of > > METH_* flags when implementing fast-callable object. > > Yes indeed. But implementing a fast-callable object is quite > specialized, not something that ordinary extension writers would care > about. And if they don't care about performance, tp_call remains supported. > > More generally: with PEP 580, everything from the documented C API > remains supported. So people can write extensions exactly as before. > They only need to care about PEP 580 if they want to use the new > features that PEP 580 adds (or if they used undocumented internals). > > > Jeroen. 100% agree. I never claimed fastcall support for callable object is important. >From my first mail in this thread, I prefer PEP 580, although I was not 100% sure PEP 580 is better. My point (why I'm not 100% sure PEP 580 is better) was simplicity / straightforwardness. I just explained it as "understandability for extension author". Method types (PyCFunction / CyFunction / method_descripter, etc) are callable type. So "make simple and fast callable protocol, then implement complicated method types behind the protocol". seems much straightforward than "Add complicated protocol for method types, then callable can be implemented like method type". If there are no drawback in PEP 576, CCall can be used behind it like: .tp_ccalloffset = offsetof(...), .tp_extcall = PyCCall_ExtCall, // filled by PyType_Read() But PEP 576 is not perfect: * FASTCALL | KEYWORDS is not stable yet, and signature of ext_call is not flexible enough. * LOAD_METHOD support is not straightforward * I want something like CCall anyway, especially for "direct call with C type" optimization in the future. That's why I prefer PEP 580. When I wrote first mail in this thread in last week, I'm about 70% sure. Since I can't find performance critical "tp_call" usage (*), I'm 90% sure for now. (*) Actually speaking, I think tp_new is called more often than tp_call. -- INADA Naoki From J.Demeyer at UGent.be Tue Jul 31 10:01:11 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Tue, 31 Jul 2018 16:01:11 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <20180730110120.6d03e6d8@fsol> <20180731085816.7bf79326@fsol> <20180731135545.24c4d427@fsol> Message-ID: <5B606BA7.8040009@UGent.be> On 2018-07-31 15:34, Victor Stinner wrote: > But I never used Cython nor cffi, so I'm not sure which one is the > most appropriate depending on the use case. Cython is a build-time tool, while cffi is a run-time tool. But Cython does a lot more than just FFI. It is a Python->C compiler which can be used for FFI but also for many other things. > A major "rewrite" of such large code base is > very difficult since people want to push new things in parallel. Or is > it maybe possible to do it incrementally? Yes, that's not a problem: you can easily mix pure Python code, Cython code and C code. I think that this kind of mixing is an important part in Cython's philosophy: for stuff where you don't care about performance: use Python. For most stuff where you do care: use Cython. For very specialized code which cannot easily be translated to Cython: use C. Jeroen. From python at mrabarnett.plus.com Tue Jul 31 11:42:10 2018 From: python at mrabarnett.plus.com (MRAB) Date: Tue, 31 Jul 2018 16:42:10 +0100 Subject: [Python-Dev] [PEP 576/580] Comparing PEP 576 and 580 In-Reply-To: <5B6026CE.70901@UGent.be> References: <5B6026CE.70901@UGent.be> Message-ID: <110e25f9-ae34-1daf-5c30-8ca81b7768aa@mrabarnett.plus.com> On 2018-07-31 10:07, Jeroen Demeyer wrote: > On 2018-07-31 09:36, INADA Naoki wrote: [snip] >>> I would like to stress that PEP 580 was designed for maximum >>> performance, both today and for future extensions (such as calling with >>> native C types). >>> >> >> I don't know what the word *stress* mean here. (Sorry, I'm not good at English >> enough for such hard discussion). >> But I want to see PoC of real benefit of PEP 580, as I said above. > > "to stress" = to draw attention to, to make it clear that > In this context, I'd define it as: "to stress" = to emphasize From ericsnowcurrently at gmail.com Tue Jul 31 12:01:51 2018 From: ericsnowcurrently at gmail.com (Eric Snow) Date: Tue, 31 Jul 2018 10:01:51 -0600 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <20180730110120.6d03e6d8@fsol> <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> <5B600F47.3090503@UGent.be> <7401f25c721941799a8dc9a288ad1ce8@xmail101.UGent.be> <5B604F95.3040705@UGent.be> Message-ID: On Tue, Jul 31, 2018 at 7:35 AM Victor Stinner wrote: > 2018-07-31 14:01 GMT+02:00 Jeroen Demeyer : > > Anyway, I know that this is probably not going to happen, but I just wanted > > to bring it up in case people would find it a great idea. But maybe not many > > CPython core developers actually know and use Cython? > > I know that Yury wants to use Cython for "C extensions" of CPython. FWIW, Brett, Barry, and I were discussing the (small) possibility of adding a minimal tool based on Cython to the repo. I've opened a thread about this on capi-sig. Replies there, please. -eric From solipsis at pitrou.net Tue Jul 31 12:03:13 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 31 Jul 2018 18:03:13 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <20180730110120.6d03e6d8@fsol> <20180731085816.7bf79326@fsol> <20180731135545.24c4d427@fsol> Message-ID: <20180731180313.2bf6d103@fsol> On Tue, 31 Jul 2018 15:34:05 +0200 Victor Stinner wrote: > Antoine: would you mind to subscribe to the capi-sig mailing list? As > expected, they are many interesting points discussed here, but I would > like to move all C API discussions to capi-sig. I only continue on > python-dev since you started here (and ignored my request to start > discussing my idea on capi-sig :-)). Well, I responded to your e-mail discussion thread. I see more messages in this thread here than on capi-sig. ;-) > For example, PyPy uses different memory allocators depending on the > scope and the lifetime of an object. I'm not sure that you can > implement such optimization if you are stuck with reference counting. But what does reference counting have to do with memory allocators exactly? > > OS vendors seem to be doing a fine job AFAICT. And if I want a recent > > Python I just download Miniconda/Anaconda. > > Is it used in production to deploy services? Or is it more used by > developers? I never used Anaconda. I don't know, but there's no hard reason why you couldn't use it to deploy services (though some people may prefer Docker or other technologies). > > I think you don't realize that the C API is *already* annoying. People > > started with it mostly because there wasn't a better alternative at the > > time. You don't need to make it more annoying than it already is ;-) > > > > Replacing existing C extensions with something else is entirely a > > developer time/effort problem, not an attractivity problem. And I'm > > not sure that porting a C extension to a new C API is more reasonable > > than porting to Cython entirely. > > Do you think that it's doable to port numpy to Cython? It's made of > 255K lines of C code. Numpy is a bit special as it exposes its own C API, so porting it entirely to Cython would be difficult (how do you expose a C macro in Cython?). Also, internally it has a lot of macro-generated code for specialized loop implementations (metaprogramming in C :-)). I suppose some bits could be (re)written in Cython. Actually, the numpy.random module is already a Cython module. > > It's just that I disagree that removing the C API will make CPython 2x > > faster. > > How can we make CPython 2x faster? Why everybody, except of PyPy, > failed to do that? Because PyPy spent years working full time on a JIT compiler. It's also written in (a dialect of) Python, which helps a lot with experimenting and building abstractions, compared to C or even C++. Regards Antoine. From vstinner at redhat.com Tue Jul 31 12:25:25 2018 From: vstinner at redhat.com (Victor Stinner) Date: Tue, 31 Jul 2018 18:25:25 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <20180731180313.2bf6d103@fsol> References: <20180730110120.6d03e6d8@fsol> <20180731085816.7bf79326@fsol> <20180731135545.24c4d427@fsol> <20180731180313.2bf6d103@fsol> Message-ID: I replied on capi-sig. 2018-07-31 18:03 GMT+02:00 Antoine Pitrou : > On Tue, 31 Jul 2018 15:34:05 +0200 > Victor Stinner wrote: >> Antoine: would you mind to subscribe to the capi-sig mailing list? As >> expected, they are many interesting points discussed here, but I would >> like to move all C API discussions to capi-sig. I only continue on >> python-dev since you started here (and ignored my request to start >> discussing my idea on capi-sig :-)). > > Well, I responded to your e-mail discussion thread. I see more > messages in this thread here than on capi-sig. ;-) > >> For example, PyPy uses different memory allocators depending on the >> scope and the lifetime of an object. I'm not sure that you can >> implement such optimization if you are stuck with reference counting. > > But what does reference counting have to do with memory allocators > exactly? > >> > OS vendors seem to be doing a fine job AFAICT. And if I want a recent >> > Python I just download Miniconda/Anaconda. >> >> Is it used in production to deploy services? Or is it more used by >> developers? I never used Anaconda. > > I don't know, but there's no hard reason why you couldn't use it to > deploy services (though some people may prefer Docker or other > technologies). > >> > I think you don't realize that the C API is *already* annoying. People >> > started with it mostly because there wasn't a better alternative at the >> > time. You don't need to make it more annoying than it already is ;-) >> > >> > Replacing existing C extensions with something else is entirely a >> > developer time/effort problem, not an attractivity problem. And I'm >> > not sure that porting a C extension to a new C API is more reasonable >> > than porting to Cython entirely. >> >> Do you think that it's doable to port numpy to Cython? It's made of >> 255K lines of C code. > > Numpy is a bit special as it exposes its own C API, so porting it > entirely to Cython would be difficult (how do you expose a C macro in > Cython?). Also, internally it has a lot of macro-generated code for > specialized loop implementations (metaprogramming in C :-)). > > I suppose some bits could be (re)written in Cython. Actually, the > numpy.random module is already a Cython module. > >> > It's just that I disagree that removing the C API will make CPython 2x >> > faster. >> >> How can we make CPython 2x faster? Why everybody, except of PyPy, >> failed to do that? > > Because PyPy spent years working full time on a JIT compiler. It's also > written in (a dialect of) Python, which helps a lot with experimenting > and building abstractions, compared to C or even C++. > > Regards > > Antoine. > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com From erik.m.bray at gmail.com Tue Jul 31 13:16:59 2018 From: erik.m.bray at gmail.com (Erik Bray) Date: Tue, 31 Jul 2018 19:16:59 +0200 Subject: [Python-Dev] Update on Cygwin support (was: Clarifying Cygwin support in CPython) In-Reply-To: References: Message-ID: On Mon, Jul 30, 2018 at 5:26 PM Nick Coghlan wrote: > > On 26 July 2018 at 02:13, Erik Bray wrote: > > I think a new approach that might be more practical for actually > > getting this platform re-supported, is to go ahead and add a CI build, > > and just skip all known failing test modules. This is what I've done > > in a new PR to add a Cygwin build on AppVeyor: > > > > https://github.com/python/cpython/pull/8463 > > > > This is far from ideal of course, and should not mean the platform is > > "supported". But now I and others can go through and fix the > > remaining test failures, re-enable those modules in the CI > > configuration, and actually obtain some meaningful results, which will > > hopefully encourage the core committers to accept fixes for the > > platform. > > I believe the PEP 538 & 540 locale handling tests are amongst those > that are still a bit sketchy (or outright broken?) on Cygwin, and I > think having an advisory CI bot would definitely help with that. > (Cygwin/MinGW are an interesting hybrid that really highlight the fact > that neither "POSIX implies not Windows" nor "Windows implies the > Win32 API" are entirely valid assumptions) Yes, I believe those tests are still a little broken, though the improvements you last made to them should be helpful in getting it passing. I haven't looked at it in a few months. Indeed, it makes for some interesting broken assumptions. Another example I've encountered recently is because Cygwin uses the posixpath module, all handling of Windows-style paths is broken. This is fine, because in general a developer should *not* be using Windows paths on Cygwin; POSIX paths only. However, the fact remains that Cygwin does (mostly) transparently support Windows paths at the system level, so some things work. But if a user runs a script that happens to be written in Python, but passes Windows paths to it, say, as command-line arguments, it may or may not work. If the path is passed directly to open(), no problem. But if it goes through os.path.abspath for example things blow up. I'm undecided as to whether this is something that developers writing applications that support Cygwin need to handle, or if this is something that could work better on the Python end as well. I lean toward the former, but I also wonder if there isn't more that could be done in the stdlib to improve this issue as well. In the meantime I wrote pycygwin [1] to help with these sorts of issues in my own software. > So your suggested approach seems like a plausible way forward to me. > > The main potentially viable alternative I see would be to set up the > *buildbot* first, and then devote the custom builder branch to the > task of Cygwin testing for a while: > https://devguide.python.org/buildbots/#custom-builders > > However, I think the overall UX of that would be worse than going down > the advisory CI path (especially since it wouldn't really help with > the aspect of parallel development introducing new Cygwin failures). Exactly. And at least for starters I might have to push to buildbot-custom, because without a few minimal fixes in place CPython currently does not build successfully at all on Cygwin, which makes the buildbot a little unhelpful. But Zach is already in touch with me about getting a buildbot worker set up anyways. I agree it's still good to have, and will be more and more useful as I get the requisite fixes merged... Thanks, E [1] http://pycygwin.readthedocs.io/en/latest/ From solipsis at pitrou.net Tue Jul 31 13:32:03 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Tue, 31 Jul 2018 19:32:03 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: References: <20180730110120.6d03e6d8@fsol> <20180731085816.7bf79326@fsol> <20180731135545.24c4d427@fsol> <20180731180313.2bf6d103@fsol> Message-ID: <20180731193203.1851fcff@fsol> Well, I tried to subscribe to capi-sig, but I didn't get a confirmation e-mail. Regards Antoine. On Tue, 31 Jul 2018 18:25:25 +0200 Victor Stinner wrote: > I replied on capi-sig. > > 2018-07-31 18:03 GMT+02:00 Antoine Pitrou : > > On Tue, 31 Jul 2018 15:34:05 +0200 > > Victor Stinner wrote: > >> Antoine: would you mind to subscribe to the capi-sig mailing list? As > >> expected, they are many interesting points discussed here, but I would > >> like to move all C API discussions to capi-sig. I only continue on > >> python-dev since you started here (and ignored my request to start > >> discussing my idea on capi-sig :-)). > > > > Well, I responded to your e-mail discussion thread. I see more > > messages in this thread here than on capi-sig. ;-) > > > >> For example, PyPy uses different memory allocators depending on the > >> scope and the lifetime of an object. I'm not sure that you can > >> implement such optimization if you are stuck with reference counting. > > > > But what does reference counting have to do with memory allocators > > exactly? > > > >> > OS vendors seem to be doing a fine job AFAICT. And if I want a recent > >> > Python I just download Miniconda/Anaconda. > >> > >> Is it used in production to deploy services? Or is it more used by > >> developers? I never used Anaconda. > > > > I don't know, but there's no hard reason why you couldn't use it to > > deploy services (though some people may prefer Docker or other > > technologies). > > > >> > I think you don't realize that the C API is *already* annoying. People > >> > started with it mostly because there wasn't a better alternative at the > >> > time. You don't need to make it more annoying than it already is ;-) > >> > > >> > Replacing existing C extensions with something else is entirely a > >> > developer time/effort problem, not an attractivity problem. And I'm > >> > not sure that porting a C extension to a new C API is more reasonable > >> > than porting to Cython entirely. > >> > >> Do you think that it's doable to port numpy to Cython? It's made of > >> 255K lines of C code. > > > > Numpy is a bit special as it exposes its own C API, so porting it > > entirely to Cython would be difficult (how do you expose a C macro in > > Cython?). Also, internally it has a lot of macro-generated code for > > specialized loop implementations (metaprogramming in C :-)). > > > > I suppose some bits could be (re)written in Cython. Actually, the > > numpy.random module is already a Cython module. > > > >> > It's just that I disagree that removing the C API will make CPython 2x > >> > faster. > >> > >> How can we make CPython 2x faster? Why everybody, except of PyPy, > >> failed to do that? > > > > Because PyPy spent years working full time on a JIT compiler. It's also > > written in (a dialect of) Python, which helps a lot with experimenting > > and building abstractions, compared to C or even C++. > > > > Regards > > > > Antoine. > > _______________________________________________ > > Python-Dev mailing list > > Python-Dev at python.org > > https://mail.python.org/mailman/listinfo/python-dev > > Unsubscribe: https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com From brett at python.org Tue Jul 31 13:48:50 2018 From: brett at python.org (Brett Cannon) Date: Tue, 31 Jul 2018 10:48:50 -0700 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <20180731193203.1851fcff@fsol> References: <20180730110120.6d03e6d8@fsol> <20180731085816.7bf79326@fsol> <20180731135545.24c4d427@fsol> <20180731180313.2bf6d103@fsol> <20180731193203.1851fcff@fsol> Message-ID: On Tue, 31 Jul 2018 at 10:32 Antoine Pitrou wrote: > > Well, I tried to subscribe to capi-sig, but I didn't get a > confirmation e-mail. > I subscribed yesterday without issue. I would email postmaster to try and find out what happened. -Brett > > Regards > > Antoine. > > > On Tue, 31 Jul 2018 18:25:25 +0200 > Victor Stinner wrote: > > I replied on capi-sig. > > > > 2018-07-31 18:03 GMT+02:00 Antoine Pitrou : > > > On Tue, 31 Jul 2018 15:34:05 +0200 > > > Victor Stinner wrote: > > >> Antoine: would you mind to subscribe to the capi-sig mailing list? As > > >> expected, they are many interesting points discussed here, but I would > > >> like to move all C API discussions to capi-sig. I only continue on > > >> python-dev since you started here (and ignored my request to start > > >> discussing my idea on capi-sig :-)). > > > > > > Well, I responded to your e-mail discussion thread. I see more > > > messages in this thread here than on capi-sig. ;-) > > > > > >> For example, PyPy uses different memory allocators depending on the > > >> scope and the lifetime of an object. I'm not sure that you can > > >> implement such optimization if you are stuck with reference > counting. > > > > > > But what does reference counting have to do with memory allocators > > > exactly? > > > > > >> > OS vendors seem to be doing a fine job AFAICT. And if I want a > recent > > >> > Python I just download Miniconda/Anaconda. > > >> > > >> Is it used in production to deploy services? Or is it more used by > > >> developers? I never used Anaconda. > > > > > > I don't know, but there's no hard reason why you couldn't use it to > > > deploy services (though some people may prefer Docker or other > > > technologies). > > > > > >> > I think you don't realize that the C API is *already* annoying. > People > > >> > started with it mostly because there wasn't a better alternative at > the > > >> > time. You don't need to make it more annoying than it already is > ;-) > > >> > > > >> > Replacing existing C extensions with something else is entirely a > > >> > developer time/effort problem, not an attractivity problem. And I'm > > >> > not sure that porting a C extension to a new C API is more > reasonable > > >> > than porting to Cython entirely. > > >> > > >> Do you think that it's doable to port numpy to Cython? It's made of > > >> 255K lines of C code. > > > > > > Numpy is a bit special as it exposes its own C API, so porting it > > > entirely to Cython would be difficult (how do you expose a C macro in > > > Cython?). Also, internally it has a lot of macro-generated code for > > > specialized loop implementations (metaprogramming in C :-)). > > > > > > I suppose some bits could be (re)written in Cython. Actually, the > > > numpy.random module is already a Cython module. > > > > > >> > It's just that I disagree that removing the C API will make CPython > 2x > > >> > faster. > > >> > > >> How can we make CPython 2x faster? Why everybody, except of PyPy, > > >> failed to do that? > > > > > > Because PyPy spent years working full time on a JIT compiler. It's > also > > > written in (a dialect of) Python, which helps a lot with experimenting > > > and building abstractions, compared to C or even C++. > > > > > > Regards > > > > > > Antoine. > > > _______________________________________________ > > > Python-Dev mailing list > > > Python-Dev at python.org > > > https://mail.python.org/mailman/listinfo/python-dev > > > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/vstinner%40redhat.com > > _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/brett%40python.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From stefan_ml at behnel.de Tue Jul 31 16:40:46 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 31 Jul 2018 22:40:46 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <20180731094528.118471f9@fsol> References: <20180730110120.6d03e6d8@fsol> <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> <5B600F47.3090503@UGent.be> <20180731094528.118471f9@fsol> Message-ID: Antoine Pitrou schrieb am 31.07.2018 um 09:45: > On Tue, 31 Jul 2018 09:27:03 +0200 > Jeroen Demeyer wrote: >> On 2018-07-31 08:58, Antoine Pitrou wrote: >>> I think Stefan is right that we >>> should push people towards Cython and alternatives, rather than direct >>> use of the C API (which people often fail to use correctly, in my >>> experience). >> >> I know this probably isn't the correct place to bring it up, but I'm >> sure that CPython itself could benefit from using Cython. For example, >> most of the C extensions in Modules/ could be written in Cython. > > We don't depend on any third-party Python modules. Adding a Cython > dependency for CPython development would be a tough sell. I don't really want to get into that discussion (it's more about processes than arguments), but let me note that the CPython development already has a couple of dependencies, such as github and its bots, or tools like argument clinic (admittedly included), make and a C compiler (not included), and a text editor. It's not like it's free of tools that help in writing and maintaining the code. That's pretty much the level at which I also see Cython. It's more complex than argument clinic, but it otherwise serves a similar need. > Also, a C extension can be built-in (linked statically into the > interpreter), which I think would be hard to do with Cython. Someone recently contributed a feature of hiding the pyinit function for the embedding case, so people do these things already. This could use the normal inittab mechanism, for example. What I think you might be referring to is that Cython modules require the CPython runtime to be initialised to a certain extent, so you couldn't implement "sys" in Cython, for example. But Jeroen is right, Cython should be a viable option for (most of?) the extension modules in the stdlib. Whether the CPython core devs would accept it in their workflow or not is a totally different question. Stefan From stefan_ml at behnel.de Tue Jul 31 16:48:31 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Tue, 31 Jul 2018 22:48:31 +0200 Subject: [Python-Dev] Let's change to C API! In-Reply-To: <5B604F95.3040705@UGent.be> References: <20180730110120.6d03e6d8@fsol> <74a848fa0eff42fc8ae5aa58e3fe71d0@xmail101.UGent.be> <5B600F47.3090503@UGent.be> <7401f25c721941799a8dc9a288ad1ce8@xmail101.UGent.be> <5B604F95.3040705@UGent.be> Message-ID: Jeroen Demeyer schrieb am 31.07.2018 um 14:01: > On 2018-07-31 12:56, Victor Stinner wrote: >> I would be nice to be able to use something to "generate" C >> extensions, maybe even from pure Python code. > > Cython has a "pure Python mode" which does exactly that. There are several > ways to include typing information, to ensure that a module remains > Python-compatible but can be compiled by Cython in an optimized way. FWIW, modules like difflib can easily be speed up by factors when compiling and optimising them with Cython, without giving up the Python syntax compatibility. I just gave a Cython talk at EuroPython last week where I used difflib as one of my examples. Stefan From vstinner at redhat.com Tue Jul 31 18:14:28 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 1 Aug 2018 00:14:28 +0200 Subject: [Python-Dev] New _Py_InitializeFromConfig() function (PEP 432) Message-ID: Hi, I finished my work on the _PyCoreConfig structure: it's a C structure in Include/pystate.h which has many fields used to configure Python initialization. In Python 3.6 and older, these parameters were scatted around the code, and it was hard to get an exhaustive list of it. This work is linked to the Nick Coghlan's PEP 432 "Restructuring the CPython startup sequence": https://www.python.org/dev/peps/pep-0432/ Right now, the new API is still private. Nick Coghlan splitted the initialization in two parts: "core" and "main". I'm not sure that this split is needed. We should see what to do, but it would be nice to make the _PyCoreConfig API public! IMHO it's way better than the old way to configuration Python initialization. -- It is now possible to only use _PyCoreConfig to initialize Python: it overrides old ways to configure Python like environment variables (ex: PYTHONPATH), global configuration variables (ex: Py_BytesWarningFlag) and C functions (ex: Py_SetProgramName()). I added tests to test_embed on the different ways to configure Python initialization: * environment variables (ex: PYTHONPATH) * global configuration variables (ex: Py_BytesWarningFlag) and C functions (ex: Py_SetProgramName()) * _PyCoreConfig I found and fixed many issues when writing these tests :-) Reading the current configuration, _PyCoreConfig_Read(), no longer changes the configuration. Now the code to read the configuration and the code to apply the configuration is properly separated. The work is not fully complete, there are a few remaining corner cases and some parameters (ex: Py_FrozenFlag) which cannot be set by _PyCoreConfig yet. My latest issue used to work on this API: https://bugs.python.org/issue34170 I had to refactor a lot of code to implement all of that. -- The problem is that Python 3.7 got the half-baked implementation, and it caused issues: * Calling Py_Main() after Py_Initialize() fails with a fatal error on Python 3.7.0 https://bugs.python.org/issue34008 * PYTHONOPTIMIZE environment variable is ignored by Py_Initialize() https://bugs.python.org/issue34247 I fixed the first issue, I'm now working on the second one to see how it can be fixed. Other option would be to backport the code from master to the 3.7 branch, since the code in master has a way better design. But it requires to backport a lot of changes. I'm not sure yet what is the best option. Victor From mike at selik.org Tue Jul 31 18:57:33 2018 From: mike at selik.org (Michael Selik) Date: Tue, 31 Jul 2018 15:57:33 -0700 Subject: [Python-Dev] Accessing mailing list archives In-Reply-To: References: Message-ID: Would it be possible to normalize by the number of mailing list members and also by "active" members? The latter would be tricky to define. On Mon, Jul 30, 2018 at 3:29 PM Victor Stinner wrote: > Hi Bob, > > I wrote a basic script to compute the number of emails per PEP. It > requires to download gzipped mbox files from the web page of archives per > month, then ungzip them: > > https://github.com/vstinner/misc/blob/master/python/parse_mailman_mbox_peps.py > > Results: > https://mail.python.org/pipermail/python-committers/2018-April/005310.html > > Victor > > Le lundi 30 juillet 2018, Bob Purvy a ?crit : > > hi all, > > I've been trying to figure out how to access the archives > programmatically. I'm sure this is easy once you know, but googling various > things hasn't worked. What I want to do is graph the number of messages > about PEP 572 by time. (or has someone already done that?) > > I installed GNU Mailman, and downloaded the gzip'ed archives for a > number of months and unzipped them, and I suspect that there's some way to > get them all into a single database, but it hasn't jumped out at me. If I > count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines > in one of those text files, I get slightly different numbers for each. > > Alternatively, they're maybe already in a database, and I just need API > access to do the querying? Can someone help me out? > > Bob _______________________________________________ > Python-Dev mailing list > Python-Dev at python.org > https://mail.python.org/mailman/listinfo/python-dev > Unsubscribe: > https://mail.python.org/mailman/options/python-dev/mike%40selik.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From vstinner at redhat.com Tue Jul 31 20:30:49 2018 From: vstinner at redhat.com (Victor Stinner) Date: Wed, 1 Aug 2018 02:30:49 +0200 Subject: [Python-Dev] Accessing mailing list archives In-Reply-To: References: Message-ID: Feel free to modify the script to make your own statistics ;-) Victor 2018-08-01 0:57 GMT+02:00 Michael Selik : > Would it be possible to normalize by the number of mailing list members and > also by "active" members? The latter would be tricky to define. > > On Mon, Jul 30, 2018 at 3:29 PM Victor Stinner wrote: >> >> Hi Bob, >> >> I wrote a basic script to compute the number of emails per PEP. It >> requires to download gzipped mbox files from the web page of archives per >> month, then ungzip them: >> >> https://github.com/vstinner/misc/blob/master/python/parse_mailman_mbox_peps.py >> >> Results: >> https://mail.python.org/pipermail/python-committers/2018-April/005310.html >> >> Victor >> >> Le lundi 30 juillet 2018, Bob Purvy a ?crit : >> > hi all, >> > I've been trying to figure out how to access the archives >> > programmatically. I'm sure this is easy once you know, but googling various >> > things hasn't worked. What I want to do is graph the number of messages >> > about PEP 572 by time. (or has someone already done that?) >> > I installed GNU Mailman, and downloaded the gzip'ed archives for a >> > number of months and unzipped them, and I suspect that there's some way to >> > get them all into a single database, but it hasn't jumped out at me. If I >> > count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines >> > in one of those text files, I get slightly different numbers for each. >> > Alternatively, they're maybe already in a database, and I just need API >> > access to do the querying? Can someone help me out? >> > Bob _______________________________________________ >> Python-Dev mailing list >> Python-Dev at python.org >> https://mail.python.org/mailman/listinfo/python-dev >> Unsubscribe: >> https://mail.python.org/mailman/options/python-dev/mike%40selik.org From cs at cskk.id.au Tue Jul 31 20:46:39 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Wed, 1 Aug 2018 10:46:39 +1000 Subject: [Python-Dev] Accessing mailing list archives In-Reply-To: References: Message-ID: <20180801004639.GA35267@cskk.homeip.net> On 30Jul2018 13:40, Bob Purvy wrote: >I've been trying to figure out how to access the archives programmatically. >I'm sure this is easy once you know, but googling various things hasn't >worked. What I want to do is graph the number of messages about PEP 572 by >time. (or has someone already done that?) > >I installed GNU Mailman, and downloaded the gzip'ed archives for a number >of months and unzipped them, and I suspect that there's some way to get >them all into a single database, but it hasn't jumped out at me. If I >count the "Message-ID" lines, the "Subject:" lines, and the "\nFrom " lines >in one of those text files, I get slightly different numbers for each. > >Alternatively, they're maybe *already* in a database, and I just need API >access to do the querying? Can someone help me out? Like Victor, I download mailing list archives. Between pulling them in and also subscribing, ideally I get a complete history in my "python" mail folder. Likewise for other lists. The mailman archives are UNIX mbox files, compressed, with a bit of header munging (to make address harvesting harder). You can concatenate them and uncompress and reverse the munging like this: cat *.gz | gunzip | fix-mail-dates --mbox | un-at- where fix-mail-dates is here: https://bitbucket.org/cameron_simpson/css/src/tip/bin/fix-mail-dates and un-at- is here: https://bitbucket.org/cameron_simpson/css/src/tip/bin/un-at- and the output is a nice UNIX mbox file. You can load that into most mail readers or parse it with Python's email modules (in the stdlib). It should be easy enough to scan such a thing and count header contents etc. Ignore the "From " line content, prefer the "From:" header. (Separate messages on "From " of course, just don't grab email addresses from it.) Cheers, Cameron Simpson From zhang.lei.fly at gmail.com Tue Jul 31 23:16:00 2018 From: zhang.lei.fly at gmail.com (Jeffrey Zhang) Date: Wed, 1 Aug 2018 11:16:00 +0800 Subject: [Python-Dev] Confused on git commit tree about Lib/datetime.py Message-ID: I found a interesting issue when checking the Lib/datetime.py implementation in python3 This patch is introduced by cf86e368ebd17e10f68306ebad314eea31daaa1e [0]. But if you check the github page[0], or using git tag --contains, you will find v2.7.x includes this commit too. $ git tag --contains cf86e368ebd17e10f68306ebad314eea31daaa1e 3.2 v2.7.10 v2.7.10rc1 v2.7.11 v2.7.11rc1 ... whereas, if you check the v2.7.x code base, nothing is found $ git log v2.7.4 -- Lib/datetime.py I guess it maybe a git tool bug, or the commit tree is messed up. Is there any guys could explain this situation? [0] https://github.com/python/cpython/commit/cf86e368ebd17e10f68306ebad314eea31daaa1e -- Regards, Jeffrey Zhang Blog: http://xcodest.me -------------- next part -------------- An HTML attachment was scrubbed... URL: