From ncoghlan at gmail.com Thu Mar 1 00:08:09 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 1 Mar 2018 15:08:09 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 1 March 2018 at 06:34, Chris Angelico wrote: > On Thu, Mar 1, 2018 at 7:28 AM, Paul Moore wrote: > > I've no idea how that would work under statement-local names either, > though. > > > > boom = lambda: boom() > > boom() > > > > is just an infinite recursion. I'm less sure that the as version is. > > Or the alternative form > > > > ((lambda: boom()) as boom)() > > > > I know you can tell me what the implementation does - but I can't > > reason it out from the spec. > > The only part that isn't yet properly specified is how this interacts > with closures, and that's because I really don't know what makes > sense. Honestly, *any* situation where you're closing over a SLNB is > going to have readability penalties, no matter what the spec says. > My suggestion is to just flat out prohibit it - if closures can capture the reference, then it isn't a statement local name binding any more, and it's reasonable to ask folks to promote it to a regular function local variable. This is another area where syntactic disambiguation on the reference side would help # prints 12 twice x = 12 if (1 as .x) == 1: def foo(): return x print(foo()) print(foo()) # prints 1 twice x = 12 if (1 as .x) == 1: x = .x def foo(): return x print(foo()) print(foo()) # Raises UnboundLocalError x = 12 if (1 as .x) == 1: def foo(): return .x print(foo()) print(foo()) That last example could potentially even raise a compile time error during the symbol table analysis pass, since the compiler would *know* there's no statement local by that name in the current lexical scope, and statement local references wouldn't have a runtime fallback to module globals or the builtins the way regular variable references do. Another perk of using the ".NAME" syntax is that we could extend it to allow statement local name bindings in for loops as well: for .i in range(10): print(.i) # This is fine print(.i) # This is an error (unless an outer statement also sets .i) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 1 00:28:57 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 1 Mar 2018 15:28:57 +1000 Subject: [Python-ideas] Upgrading python-ideas to MM3? Message-ID: Hi folks, A number of smaller python.org mailing lists have been successfully migrated to the PSF's Mailman 3 infrastructure, for example, the core-workflow mailing list: https://mail.python.org/mm3/archives/list/core-workflow at python.org/ The biggest practical gain from the upgrade is to allow folks that prefer a more forum-like experience to use the native web gateway rather than subscribing to the list. (Although it does come with some general account management improvements as well, since you get one user account with multiple list subscriptions, rather than multiple independent subscriptions that happen to share an email address). So far we've refrained from upgrading any of the "big" lists with lots of subscribers, but it seems to me that python-ideas could be a good test case on that front: * anything we discuss here necessarily isn't urgent, so if the migration encounters unexpected problems, it's OK if list activity is disrupted for a while * for newcomers to the core development process, python-ideas is typically going to be more relevant to them than the other big lists Cheers, Nick. P.S. As per https://github.com/python/core-workflow/issues/200#issuecomment-361818530, subscription migrations are based on email addresses, so the key point affecting individual users and list moderators is that you'll need to either go to https://mail.python.org/mm3/accounts/login/ and log in using a 3rd party service that already has your subscription email associated with it, or else go to https://mail.python.org/mm3/accounts/signup/ and create a new account directly with the appropriate email address. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Thu Mar 1 01:40:51 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Mar 2018 17:40:51 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On Thu, Mar 1, 2018 at 3:31 PM, Nick Coghlan wrote: > On 28 February 2018 at 08:27, Chris Angelico wrote: >> 2. The current implementation [1] implements statement-local names using >> a special (and mostly-invisible) name mangling. This works perfectly >> inside functions (including list comprehensions), but not at top >> level. Is this a serious limitation? Is it confusing? > > > It isn't clear to me from the current PEP what the intended lifecycle of the > bound names actually is, especially for compound statements. I think you're looking at an old version of the PEP, but that's kinda gonna happen on the first day of a hot topic :) But to remove all confusion, I've now added a section clarifying execution order, using several of your examples. > x = (expr as y) > assert x == y # Does this pass? Or raise NameError for 'y'? NameError. The SLNB is gone at end of line. > if (condition as c): > assert c # Does this pass? Or raise NameError for 'c'? > else: > assert not c # Does this pass? Or raise NameError for 'c'? > assert c or not c # Does this pass? Or raise NameError for 'c'? c is available in the indented blocks, and is gone once the entire 'if/else' block is done. > class C: > x = (True as y) > assert C.y # Does this pass? Or raise AttributeError for 'y'? That'll raise. (At least, according to the specification and intent. The reference implementation may be lagging a bit.) > I think it would also be worth explicitly considering a syntactic variant > that requires statement local references to be explicitly disambiguated from > regular variable names by way of a leading dot: > > result = [[(f(x) as .y), .y] for x in range(5)] > > [chomp] > > Since ".NAME" is illegal for both variable and attribute names, this makes > the fact statement locals are a distinct namespace visible to readers as > well as to the compiler, and also reduces the syntactic ambiguity in with > statements and exception handlers. I've mentioned this in the alternate syntaxes, but I don't like having to state a variable's scope in its name. Python doesn't force us to adorn all global names with a character, and the difference between function-local and statement-local is generally less important than the difference between global and function-local. But it IS a viable syntax, and I'm saying so in the PEP. Thanks for the feedback! Much appreciated. ChrisA From rosuav at gmail.com Thu Mar 1 01:47:59 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 1 Mar 2018 17:47:59 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: <54c90d32-ef5f-229d-28d8-2e3433eadf0b@btinternet.com> <5A96C673.4070804@stoneleaf.us> <5A970476.3000101@brenbarn.net> Message-ID: On Thu, Mar 1, 2018 at 3:54 PM, Nick Coghlan wrote: > On 1 March 2018 at 06:00, Chris Angelico wrote: >> >> On Thu, Mar 1, 2018 at 6:35 AM, Brendan Barnwell >> wrote: >> > On 2018-02-28 07:18, Chris Angelico wrote: >> >> >> >> Except that assignment is evaluated RHS before LHS as part of a single >> >> statement. When Python goes to look up the name "a" to store it (as >> >> the final step of the assignment), the SLNB is still active (it's >> >> still the same statement - note that this is NOT expression-local), so >> >> it uses the temporary. >> > >> > >> > Wait, so you're saying that if I do >> > >> > a = (2 as a) >> > >> > The "a = " assignment assigns to the SLNB, and so is then >> > discarded >> > after the statement finishes? >> > >> > That seems very bad to me. If there are SLNBs with this special >> > "as" syntax, I think the ONLY way to assign to an SLNB should be with >> > the >> > "as" syntax. You shouldn't be able to assign to an SLNB with regular >> > assignment syntax, even if you created an SNLB with the same name as the >> > LHS >> > within the RHS. >> >> That seems a reasonable requirement on the face of it, but what about >> these variants? >> >> a = (x as a) >> a[b] = (x as a) >> b[a] = (x as a) >> a[b].c = (x as a) >> b[a].c = (x as a) >> >> Which of these should use the SLNB, which should be errors, which >> should use the previously-visible binding of 'a'? > > > This is the kind of ambiguity of intent that goes away if statement locals > are made syntactically distinct in addition to being semantically distinct: > > .a = (2 as .a) # Syntax error (persistent bindings can't target statement > locals) > a = (2 as .a) # Binds both ".a" (ephemerally) and "a" (persistently) to > "2" > .a[b] = (x as .a) # Syntax error (persistent bindings can't target > statement locals) > b[.a] = (x as .a) # LHS references .a > .a[b].c = (x as .a) # Syntax error (persistent bindings can't target > statement locals) > b[.a].c = (x as .a) # LHS references .a > > We may still decide that even the syntactically distinct variant poses a net > loss to overall readability, but I do think it avoids many of the > confusability problems that arise when statement locals use the same > reference syntax as regular variable names. Okay. I think I have the solution, then. One of two options: 1) SLNBs are not permitted as assignment (incl augassign) targets. Doing so is a SyntaxError. 2) SLNBs are ignored when compiling assignment targets. Doing so will bind to the "real" name. Using an SLNB to subscript another object is perfectly acceptable, as that's simply referencing. The only case that might slip between the cracks is "a[b].c" which technically is looking up a[b] and only assigning to *that* object (for instance, if 'a' is a tuple and 'b' is zero, it's perfectly legal to write "a[b].c = 1" even though tuples are immutable). Other than that, the intention given in all your examples would be sustained. Which of the two is preferable? I'm thinking of going with option 2, but there are arguments on both sides. ChrisA From tritium-list at sdamon.com Thu Mar 1 02:19:21 2018 From: tritium-list at sdamon.com (Alex Walters) Date: Thu, 1 Mar 2018 02:19:21 -0500 Subject: [Python-ideas] Upgrading python-ideas to MM3? In-Reply-To: References: Message-ID: <17b001d3b12d$9ed4b0a0$dc7e11e0$@sdamon.com> Besides the login issues (or non-issues), assuming best case scenario, would the migration be seamless from my inbox?s point of view? Would I have to re-subscribe? From: Python-ideas [mailto:python-ideas-bounces+tritium-list=sdamon.com at python.org] On Behalf Of Nick Coghlan Sent: Thursday, March 1, 2018 12:29 AM To: python-ideas Subject: [Python-ideas] Upgrading python-ideas to MM3? Hi folks, A number of smaller python.org mailing lists have been successfully migrated to the PSF's Mailman 3 infrastructure, for example, the core-workflow mailing list: https://mail.python.org/mm3/archives/list/core-workflow at python.org/ The biggest practical gain from the upgrade is to allow folks that prefer a more forum-like experience to use the native web gateway rather than subscribing to the list. (Although it does come with some general account management improvements as well, since you get one user account with multiple list subscriptions, rather than multiple independent subscriptions that happen to share an email address). So far we've refrained from upgrading any of the "big" lists with lots of subscribers, but it seems to me that python-ideas could be a good test case on that front: * anything we discuss here necessarily isn't urgent, so if the migration encounters unexpected problems, it's OK if list activity is disrupted for a while * for newcomers to the core development process, python-ideas is typically going to be more relevant to them than the other big lists Cheers, Nick. P.S. As per https://github.com/python/core-workflow/issues/200#issuecomment-361818530, subscription migrations are based on email addresses, so the key point affecting individual users and list moderators is that you'll need to either go to https://mail.python.org/mm3/accounts/login/ and log in using a 3rd party service that already has your subscription email associated with it, or else go to https://mail.python.org/mm3/accounts/signup/ and create a new account directly with the appropriate email address. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Mar 1 02:33:56 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 1 Mar 2018 09:33:56 +0200 Subject: [Python-ideas] Upgrading python-ideas to MM3? In-Reply-To: References: Message-ID: 01.03.18 07:28, Nick Coghlan ????: > A number of smaller python.org mailing lists have > been successfully migrated to the PSF's Mailman 3 infrastructure, for > example, the core-workflow mailing list: > https://mail.python.org/mm3/archives/list/core-workflow at python.org/ > > The biggest practical gain from the upgrade is to allow folks that > prefer a more forum-like experience to use the native web gateway rather > than subscribing to the list. (Although it does come with some general > account management improvements as well, since you get one user account > with multiple list subscriptions, rather than multiple independent > subscriptions that happen to share an email address). What about permanent links? Will https://mail.python.org/pipermail/python-ideas/2018-March/049108.html be accessible after migration? From ncoghlan at gmail.com Thu Mar 1 02:43:29 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 1 Mar 2018 17:43:29 +1000 Subject: [Python-ideas] Upgrading python-ideas to MM3? In-Reply-To: <17b001d3b12d$9ed4b0a0$dc7e11e0$@sdamon.com> References: <17b001d3b12d$9ed4b0a0$dc7e11e0$@sdamon.com> Message-ID: On 1 March 2018 at 17:19, Alex Walters wrote: > Besides the login issues (or non-issues), assuming best case scenario, > would the migration be seamless from my inbox?s point of view? Would I > have to re-subscribe? > Assuming all goes well (which it did for both core-mentorship and core-workflow), then folks shouldn't need to do anything to maintain their existing subscription settings. The only folks that should notice a change are: * list moderators (since the admin login arrangements change) * anyone changing their list subscription settings through the web UI (since that changes to the MM3 admin) * anyone creating links to emails archived post-migration (since the old pipermail archives will stop getting updated, but post-migration emails will have their archive link in the Archived-At header) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 1 02:46:07 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 1 Mar 2018 17:46:07 +1000 Subject: [Python-ideas] Upgrading python-ideas to MM3? In-Reply-To: References: Message-ID: On 1 March 2018 at 17:33, Serhiy Storchaka wrote: > 01.03.18 07:28, Nick Coghlan ????: > >> A number of smaller python.org mailing lists have >> been successfully migrated to the PSF's Mailman 3 infrastructure, for >> example, the core-workflow mailing list: https://mail.python.org/mm3/ar >> chives/list/core-workflow at python.org/ >> >> The biggest practical gain from the upgrade is to allow folks that prefer >> a more forum-like experience to use the native web gateway rather than >> subscribing to the list. (Although it does come with some general account >> management improvements as well, since you get one user account with >> multiple list subscriptions, rather than multiple independent subscriptions >> that happen to share an email address). >> > > What about permanent links? Will https://mail.python.org/piperm > ail/python-ideas/2018-March/049108.html be accessible after migration? > Yeah, the existing archives will remain untouched (but will also stop getting updated). New and migrated messages get archived based on a content-addressable hash (e.g. https://mail.python.org/mm3/archives/list/core-workflow at python.org/message/2Y7R6YAF6NER6TPSFLD5U4BVW2P4ABUL/), with a thread view that will happily accommodate discussions that span a change of month. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Thu Mar 1 03:27:35 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Thu, 1 Mar 2018 10:27:35 +0200 Subject: [Python-ideas] Upgrading python-ideas to MM3? In-Reply-To: References: Message-ID: 01.03.18 09:46, Nick Coghlan ????: > Yeah, the existing archives will remain untouched (but will also stop > getting updated). New and migrated messages get archived based on a > content-addressable hash (e.g. > https://mail.python.org/mm3/archives/list/core-workflow at python.org/message/2Y7R6YAF6NER6TPSFLD5U4BVW2P4ABUL/), > with a thread view that will happily accommodate discussions that span a > change of month. Would it be possible to make a redirection from old addresses to the new shiny web interface? From p.f.moore at gmail.com Thu Mar 1 04:23:37 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Mar 2018 09:23:37 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: <3c8a2267-1605-8e47-7e62-f79b230e7aec@btinternet.com> Message-ID: On 28 February 2018 at 21:48, Robert Vanden Eynde wrote: > We are currently like a dozen of people talking about multiple sections of a > single subject. > > Isn't it easier to talk on a forum? Not for me, certainly. I could probably learn how to effectively work with a forum, but as of right now, if this discussion switched to a forum, I'd be unable to follow the discussion, and would likely not contribute. And whether I'd actually take the time to learn how to work with a forum is debatable (there are many different forms of forum software, and learning a new interface for each discussion group isn't effective for me. Conversely, with mailing lists, I just use the one interface, gmail). > Am I the only one who thinks mailing list isn't easy when lots of people > talking about multiple subjects? Possibly not, but a lot of participants have invested time in learning how to work effectively with mailing lists. IMO, a forum prioritises occasional use and newcomers, while penalising long-term and in-depth use. > Of course we would put the link in the mailing list so that everyone can > join. But the discussion would still be taking place in 2 locations, which would make it even harder to follow. Unless you plan on shutting down the list? > A forum (or just few "issues" thread on github) is where we could have > different thread in parallel, in my messages I end up with like 10 comments > not all related, in a forum we could talk about everything and it would > still be organized by subjects. Conversely, interested parties would find it harder by default to read all of the discussion on the subject of PEP 572. Rather than flagging *one* thread as important, we'd be relying on having the PEP number in the subject of multiple topics, and searching. > Also, it's more interactive than email on a global list, people can talk to > each other in parallel, if I want to answer about a mail that was 10 mail > ago, it gets quickly messy. No, you just reply to that mail. You may lose some of the intermediate comments by doing so, but that happens however you organise the discussion. You're choosing to branch out the discussion from an older point. > We could all discuss on a gist or some "Issues" thread on GitHub. Not everyone can access gists or github. My work network blocks gists as "personal storage", for example. Paul From p.f.moore at gmail.com Thu Mar 1 04:30:45 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 1 Mar 2018 09:30:45 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 1 March 2018 at 06:40, Chris Angelico wrote: > On Thu, Mar 1, 2018 at 3:31 PM, Nick Coghlan wrote: >> Since ".NAME" is illegal for both variable and attribute names, this makes >> the fact statement locals are a distinct namespace visible to readers as >> well as to the compiler, and also reduces the syntactic ambiguity in with >> statements and exception handlers. > > I've mentioned this in the alternate syntaxes, but I don't like having > to state a variable's scope in its name. Python doesn't force us to > adorn all global names with a character, and the difference between > function-local and statement-local is generally less important than > the difference between global and function-local. But it IS a viable > syntax, and I'm saying so in the PEP. Agreed. This feels far to much like Perl's "sigils" that attach to a name ($var is a scalar, @var is a list, etc). Strong -1 from me. Although I *do* agree that such decoration gets rid of a lot of the worst ambiguities in the proposal. But if we're being asked to choose the lesser of 2 evils, my response is "neither" :-) Paul From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Mar 1 04:41:45 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 1 Mar 2018 18:41:45 +0900 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: <3c8a2267-1605-8e47-7e62-f79b230e7aec@btinternet.com> Message-ID: <23191.51929.778976.623245@turnbull.sk.tsukuba.ac.jp> Reply-To set to *me*. I don't really think this idea has legs at this point. It's been discussed before (it even has a SIG), but that ran out of steam and the move to Mailman 3 (whose archiver provides a more forum-like interface) is more or less imminent anyway, so it's kind of moot. I'm happy to deal with it off-list, but for the moment it's OT IMO. >>>>> Robert Vanden Eynde writes: > We are currently like a dozen of people talking about multiple > sections of a single subject. > > Isn't it easier to talk on a forum? There's the Overload SIG for this discussion of media (currently inactive for many months). Transparency notice: Mailman dev here, bias acknowledged, some in the SIG were more sympathetic to forums, I don't claim to be representative of SIG opinion. > Am I the only one who thinks mailing list isn't easy when lots of > people talking about multiple subjects? No, you are hardly alone. However, in my experience good forum software is distinctly inferior to good mail software in high-traffic multi-threaded communications. There is very little that forum software can do that a mail client cannot.[1] The main advantages of forum software are better thread management (message ids are reliable and threads are built synchronously, but it's not much better) and server-side storage (but of course that's a SPoF). As the name of the SIG suggests, the problem is not so much that mail is such a bad medium; it's more that the traffic levels are higher than people are used to, so the bad experience of high traffic is conflated with a bad experience of using mail. (We do have one or two on the SIG who say that they do fine with high-traffic multi-threaded forums. I'm not sure how closely the use case matches the main python-* channels, excluding python-list.) The main advantage to using email is that management of multiple high-traffic streams requires smart, highly configurable clients. These have been available for mail since the late 80s. They've only gotten better since. Forums, on the other hand, are based on thin clients (ie web browsers), and so you get only the configuration the forum offers, and you have to tell the forum all about it. Of course we are hackers: all the main browsers are scriptable, the language is javascript, so we could write our own forum clients. But that's exactly what forum advocates want to avoid, at least if you call it "email". The problem for mail advocates is the flip side of that. That is, AFAIK there is no good mail software that runs in a browser (let alone a smartphone), and change of mail clients, like religions and programmer's editors, is advocated only at risk to limb and life. As mentioned above, this whole issue may become moot when the mailing lists are moved to Mailman 3 (we have some experimental Mailman 3 lists; ETA of a general migration is 6--18 months, at a slightly- informed guess), which has a forum-like interface to the archives called "HyperKitty". HyperKitty is not a rival to Discourse, at least not in the near future[2], but it's already pretty good. > A forum (or just few "issues" thread on github) is where we could > have different thread in parallel, in my messages I end up with > like 10 comments not all related, in a forum we could talk about > everything and it would still be organized by subjects. > > Also, it's more interactive than email on a global list, people can > talk to each other in parallel, if I want to answer about a mail > that was 10 mail ago, it gets quickly messy. I have no idea why any of those things would be unmanageable in email, unless you have disabled threading in your client, or your client doesn't allow you to kill (or collapse) threads of no interest to you. > We could all discuss on a gist or some "Issues" thread on GitHub. Those are non-starters. *You* can already do that, but very few of the committers will follow. We already have an issue tracker for very focussed discussion. This works because people actually working on an issue are frequently spending hours on it and related work, so a minute or so spent finding the appropriate issue is small overhead. The overhead would be substantially greater if you had to regularly scan all recent issues to see if any are of interest to you. Trackers are not very good at switching threads (issues), so even if you turn on push notification for new issues, there will be substantial overhead in following more than a tiny number of threads. The kinds of things that are discussed on the mailing lists generally need to be broadcast, at least at first, and (assuming a decent mail client) threads of no interest can easily be suppressed in email. That's what the committers do, mostly. (Some just live with the firehose.) Note, that's not to say a *forum* is a non-starter. I guess that good forum software provides similar features. Some senior developers have expressed interest in moving to a forum-based discussion. Just that most of the committers have broad interest in the discussion, so are going to get their news from whatever the main channels are: we need to have a fairly small number of them, and we do *not* want them split across multiple media. Finally, speaking of "split across media", any serious proposal for change of medium will need to deal with the (voluminous) mail archives. This may not be difficult ("just provide a link"), but then again it may not be that easy. I have given no thought to it yet, and I don't think the forum advocates have either. Bottom line: in the end at least one PEP (I would guess two or three) will be needed to make a major change to the workflow like this, unless it's done in the context of a hybrid Mailman + forum system like Mailman 3 including HyperKitty. Once again, I admit that I strongly prefer mailing lists. However, I also believe that it's not bias to suggest that it's volume of communication, not the medium, that is the problem here. At least in principle mail is fully capable of carrying the load. In practice, deficient mail clients (I'm looking at you, GMail and Outlook and anything on a handheld) are all over the place. In the end we'll need to balance the costs of moving (which includes the preferences of those who "just" like email better as well as setup and migration of archives) against the benefits (mostly the preferences of those who have learned to use forums very efficiently). Footnotes: [1] To my knowledge. I'm willing to be educated. [2] And it won't solve the asynchronicity of SMTP messaging. Mailman will still be primarily a mail-based system, at least at first. From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Mar 1 04:46:05 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 1 Mar 2018 18:46:05 +0900 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: <23191.52189.522837.931057@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > Memoization can only be done for pure functions. True, but when for impure f you naively construct a list [f(x), f(x)], substituting [(f(x) as wrong), wrong] is just [wrong, wrong] ;-), because the side effect doesn't get reevaluated. Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Mar 1 04:46:12 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 1 Mar 2018 18:46:12 +0900 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: <23191.52196.251542.400406@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > for .i in range(10): > print(.i) # This is fine > print(.i) # This is an error (unless an outer statement also sets .i) That's one heck of a lot of grit on Tim's screen! From solipsis at pitrou.net Thu Mar 1 05:08:10 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Mar 2018 11:08:10 +0100 Subject: [Python-ideas] Upgrading python-ideas to MM3? References: Message-ID: <20180301110810.2c8243e4@fsol> On Thu, 1 Mar 2018 15:28:57 +1000 Nick Coghlan wrote: > Hi folks, > > A number of smaller python.org mailing lists have been successfully > migrated to the PSF's Mailman 3 infrastructure, for example, the > core-workflow mailing list: > https://mail.python.org/mm3/archives/list/core-workflow at python.org/ That URL fails for me. You probably mean https://mail.python.org/mm3/archives/list/core-workflow at python.org/ As long as there's a gmane (NNTP) gateway I can live with the MM3 UI experience. Regards Antoine. From solipsis at pitrou.net Thu Mar 1 05:11:00 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 1 Mar 2018 11:11:00 +0100 Subject: [Python-ideas] Upgrading python-ideas to MM3? References: <20180301110810.2c8243e4@fsol> Message-ID: <20180301111100.5ab0725a@fsol> On Thu, 1 Mar 2018 11:08:10 +0100 Antoine Pitrou wrote: > On Thu, 1 Mar 2018 15:28:57 +1000 > Nick Coghlan wrote: > > Hi folks, > > > > A number of smaller python.org mailing lists have been successfully > > migrated to the PSF's Mailman 3 infrastructure, for example, the > > core-workflow mailing list: > > https://mail.python.org/mm3/archives/list/core-workflow at python.org/ > > That URL fails for me. > You probably mean > https://mail.python.org/mm3/archives/list/core-workflow at python.org/ Uh, ok, I see what happened. The URL was rewritten by the gmane gateway :-) Sorry for the noise. Cheers Antoine. From turnbull.stephen.fw at u.tsukuba.ac.jp Thu Mar 1 08:31:41 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Thu, 1 Mar 2018 22:31:41 +0900 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: <23192.189.872650.303545@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > for .i in range(10): > print(.i) # This is fine > print(.i) # This is an error (unless an outer statement also sets .i) That's one heck of a lot of grit on Tim's screen! From ncoghlan at gmail.com Thu Mar 1 08:36:39 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 1 Mar 2018 23:36:39 +1000 Subject: [Python-ideas] Upgrading python-ideas to MM3? In-Reply-To: <20180301111100.5ab0725a@fsol> References: <20180301110810.2c8243e4@fsol> <20180301111100.5ab0725a@fsol> Message-ID: On 1 March 2018 at 20:11, Antoine Pitrou wrote: > On Thu, 1 Mar 2018 11:08:10 +0100 > Antoine Pitrou wrote: > > On Thu, 1 Mar 2018 15:28:57 +1000 > > Nick Coghlan wrote: > > > Hi folks, > > > > > > A number of smaller python.org mailing lists have been successfully > > > migrated to the PSF's Mailman 3 infrastructure, for example, the > > > core-workflow mailing list: > > > https://mail.python.org/mm3/archives/list/core-workflow at python.org/ > > > > That URL fails for me. > > You probably mean > > https://mail.python.org/mm3/archives/list/core-workflow at python.org/ > > Uh, ok, I see what happened. The URL was rewritten by the gmane > gateway :-) Sorry for the noise. > Thanks for explaining what happened, as both your link and mine came through unaltered for me and I was *very* confused :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Thu Mar 1 09:53:59 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Thu, 1 Mar 2018 14:53:59 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: <54c90d32-ef5f-229d-28d8-2e3433eadf0b@btinternet.com> <5A96C673.4070804@stoneleaf.us> <5A970476.3000101@brenbarn.net> Message-ID: On 01/03/2018 06:47, Chris Angelico wrote: > >> This is the kind of ambiguity of intent that goes away if statement locals >> are made syntactically distinct in addition to being semantically distinct: >> >> .a = (2 as .a) # Syntax error (persistent bindings can't target statement >> locals) >> a = (2 as .a) # Binds both ".a" (ephemerally) and "a" (persistently) to >> "2" >> .a[b] = (x as .a) # Syntax error (persistent bindings can't target >> statement locals) >> b[.a] = (x as .a) # LHS references .a >> .a[b].c = (x as .a) # Syntax error (persistent bindings can't target >> statement locals) >> b[.a].c = (x as .a) # LHS references .a -1.? Too much grit!? And I think trying to put the dots in the right places would be a frequent source of mistakes (albeit mistakes that could usually be corrected quickly). > Okay. I think I have the solution, then. One of two options: > > 1) SLNBs are not permitted as assignment (incl augassign) targets. > Doing so is a SyntaxError. > 2) SLNBs are ignored when compiling assignment targets. Doing so will > bind to the "real" name. > > Using an SLNB to subscript another object is perfectly acceptable, as > that's simply referencing. The only case that might slip between the > cracks is "a[b].c" which technically is looking up a[b] and only > assigning to *that* object (for instance, if 'a' is a tuple and 'b' is > zero, it's perfectly legal to write "a[b].c = 1" even though tuples > are immutable). Other than that, the intention given in all your > examples would be sustained. > > Which of the two is preferable? I'm thinking of going with option 2, > but there are arguments on both sides. > > +1 to one of these two options. +1 to choosing 2).? I think it's easier to understand and explain "temporary variables do not apply to the LHS of an assignment statement". (Incidentally, when in an earlier post I suggested that Expression LNBs might be better than Statement LNBs, I didn't realise that a temporary variable created in the first line of a suite ("if", "while" etc.) remained in scope for the rest of that suite.? That seems (on balance) like a Good Thing, and a lot of the rationale for SLNBs.? But I didn't like a temporary variable applying to the LHS of as assignment.? So, with the above change to assignment statements, I am now happy about SLNBs.) Rob Cliffe From guido at python.org Thu Mar 1 20:25:47 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Mar 2018 17:25:47 -0800 Subject: [Python-ideas] Medium for discussion potential changes to python (was: PEP 572: Statement-Local Name Bindings) In-Reply-To: <20180228223946.ytgtnbcv32pgyfno@phdru.name> References: <172401d3b0de$b6ad10e0$240732a0$@sdamon.com> <20180228221210.adiz2s4lyityrv26@phdru.name> <20180228223946.ytgtnbcv32pgyfno@phdru.name> Message-ID: The best we can do here is move to MM3, as is being discussed in another thread. On Wed, Feb 28, 2018 at 2:39 PM, Oleg Broytman wrote: > On Wed, Feb 28, 2018 at 10:23:08PM +0000, Matt Arcidy > wrote: > > if Linux kernel can handle it, there is no argument for it being > factually > > superior or inferior. It is only preference. > > > > There is nothing stopping a forum link being created and posted to the > list > > as an alternative right now. > > > > The result of that experiment would be the answer. > > The problem with the approach is division inside community and > miscommunication between subgroup. > > One medium (currently it's the mailing list) is more preferable. Once > an idea is discussed using the preferred medium and code is created the > other groups will feel they have been singled out and their ideas were > ignored. > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Thu Mar 1 17:47:28 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Fri, 02 Mar 2018 11:47:28 +1300 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: <3c8a2267-1605-8e47-7e62-f79b230e7aec@btinternet.com> Message-ID: <5A988300.4030800@canterbury.ac.nz> > On 28 February 2018 at 21:48, Robert Vanden Eynde wrote: > >>Isn't it easier to talk on a forum? For me, the fact that all the mailing lists I subscribe to feed into one input queue is a feature, not a bug. It means I can easily keep abreast of developments in many areas at once, without having to laboriously and explicitly go and visit multiple forums and threads. Also, every web forum I've ever used has been slow and klunky to use compared to a mail/news client. -- Greg From guido at python.org Thu Mar 1 22:28:12 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 1 Mar 2018 19:28:12 -0800 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: <5A988300.4030800@canterbury.ac.nz> References: <3c8a2267-1605-8e47-7e62-f79b230e7aec@btinternet.com> <5A988300.4030800@canterbury.ac.nz> Message-ID: On Thu, Mar 1, 2018 at 2:47 PM, Greg Ewing wrote: > On 28 February 2018 at 21:48, Robert Vanden Eynde >> wrote: >> >> Isn't it easier to talk on a forum? >>> >> > For me, the fact that all the mailing lists I subscribe to > feed into one input queue is a feature, not a bug. It means > I can easily keep abreast of developments in many areas at once, > without having to laboriously and explicitly go and visit > multiple forums and threads. > > Also, every web forum I've ever used has been slow and klunky > to use compared to a mail/news client. > We can kill this topic right now. We'll be moving to MM3, which is a mailing list for those who like email, and can also be used as a web forum for those who like that. Personally I'm partial to both, so I think this is the best of both worlds. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Mar 2 00:08:08 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Mar 2018 15:08:08 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 1 March 2018 at 19:30, Paul Moore wrote: > On 1 March 2018 at 06:40, Chris Angelico wrote: > > On Thu, Mar 1, 2018 at 3:31 PM, Nick Coghlan wrote: > >> Since ".NAME" is illegal for both variable and attribute names, this > makes > >> the fact statement locals are a distinct namespace visible to readers as > >> well as to the compiler, and also reduces the syntactic ambiguity in > with > >> statements and exception handlers. > > > > I've mentioned this in the alternate syntaxes, but I don't like having > > to state a variable's scope in its name. Python doesn't force us to > > adorn all global names with a character, and the difference between > > function-local and statement-local is generally less important than > > the difference between global and function-local. But it IS a viable > > syntax, and I'm saying so in the PEP. > > Agreed. This feels far to much like Perl's "sigils" that attach to a > name ($var is a scalar, @var is a list, etc). Strong -1 from me. > While that's a fair criticism, one of the current challenges with Python's variable naming is that given a NAME reference, there are already several potential places for that name to be resolved: * the current local scope * an enclosing function scope * the module globals * the builtin namespace This means the compiler has to be conservative and assume that if a name isn't visible as a local namespace entry or as a closure reference, then it will still be available at runtime (somehow). Structural linters and static analysers like pylint or mypy can do better and say "We can't figure out how you're expecting this name reference to be resolved at runtime", but it's still a very complex system to try to reason about when reading a code snippet. Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already complicated system, and making it even harder to reason about correctly (especially if statement locals interact with nested scopes differently from the way other locals in the same scope do). Thus the intent behind the ".NAME" suggestion is to ask whether or not it's possible to allow for name bindings that are strictly local to a compilation unit (i.e. without allowing dynamic runtime access to outer scopes or from contained scopes), *without* incurring the cost of making ordinary NAME references even more complicated to understand. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Mar 2 01:39:24 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Thu, 01 Mar 2018 22:39:24 -0800 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: <5A98F19C.7010403@stoneleaf.us> On 03/01/2018 09:08 PM, Nick Coghlan wrote: > On 1 March 2018 at 19:30, Paul Moore > On 1 March 2018 at 06:40, Chris Angelico wrote: >>> On Thu, Mar 1, 2018 at 3:31 PM, Nick Coghlan wrote: >>>> Since ".NAME" is illegal for both variable and attribute names, this makes >>>> the fact statement locals are a distinct namespace visible to readers as >>>> well as to the compiler, and also reduces the syntactic ambiguity in with >>>> statements and exception handlers. >>> >>> I've mentioned this in the alternate syntaxes, but I don't like having >>> to state a variable's scope in its name. Python doesn't force us to >>> adorn all global names with a character, and the difference between >>> function-local and statement-local is generally less important than >>> the difference between global and function-local. But it IS a viable >>> syntax, and I'm saying so in the PEP. >> >> Agreed. This feels far to much like Perl's "sigils" that attach to a >> name ($var is a scalar, @var is a list, etc). Strong -1 from me. > > > While that's a fair criticism, one of the current challenges with Python's variable naming is that given a NAME > reference, there are already several potential places for that name to be resolved: > > * the current local scope > * an enclosing function scope > * the module globals > * the builtin namespace > > This means the compiler has to be conservative and assume that if a name isn't visible as a local namespace entry or as > a closure reference, then it will still be available at runtime (somehow). Structural linters and static analysers like > pylint or mypy can do better and say "We can't figure out how you're expecting this name reference to be resolved at > runtime", but it's still a very complex system to try to reason about when reading a code snippet. > > Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already > complicated system, and making it even harder to reason about correctly (especially if statement locals interact with > nested scopes differently from the way other locals in the same scope do). Seems like it would far easier and (IMHO) more useful to scale the proposal back from a statement scope to simple expression assignment, and the variable is whatever scope it would have been if assigned to outside the expression (default being local, but non-local or global if already declared as such). No grammatical grit on anyone's monitor, no confusion about which variable is being accessed, and no confusion about the lifetime of that variable (okay, no /extra/ confusion ;) . Maybe somebody could explain why a statement-local limited scope variable is better than an ordinary well-understood local-scope variable? Particularly why it's better enough to justify more line-noise in the syntax. I'm willing to be convinced (not happy to, just willing ;) . -- ~Ethan~ From rosuav at gmail.com Fri Mar 2 01:52:22 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Mar 2018 17:52:22 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: <5A98F19C.7010403@stoneleaf.us> References: <5A98F19C.7010403@stoneleaf.us> Message-ID: On Fri, Mar 2, 2018 at 5:39 PM, Ethan Furman wrote: > Seems like it would far easier and (IMHO) more useful to scale the proposal > back from a statement scope to simple expression assignment, and the > variable is whatever scope it would have been if assigned to outside the > expression (default being local, but non-local or global if already declared > as such). > > No grammatical grit on anyone's monitor, no confusion about which variable > is being accessed, and no confusion about the lifetime of that variable > (okay, no /extra/ confusion ;) . > > Maybe somebody could explain why a statement-local limited scope variable is > better than an ordinary well-understood local-scope variable? Particularly > why it's better enough to justify more line-noise in the syntax. I'm > willing to be convinced (not happy to, just willing ;) . Sounds like what you're proposing could be done with the exact syntax that I'm using, and just remove subscopes from the discussion. (It'd still need parenthesization, I believe, to prevent syntactic ambiguities.) As a competing proposal, it's plausible; basically, it gives Python a way to assign to any name at any time. I'm honestly not sure which variant would see more backlash :) That's worthy of a mention in the alternates, at any rate. ChrisA From p.f.moore at gmail.com Fri Mar 2 04:05:49 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Mar 2018 09:05:49 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 2 March 2018 at 05:08, Nick Coghlan wrote: > On 1 March 2018 at 19:30, Paul Moore wrote: >> >> Agreed. This feels far to much like Perl's "sigils" that attach to a >> name ($var is a scalar, @var is a list, etc). Strong -1 from me. > > While that's a fair criticism, one of the current challenges with Python's > variable naming is that given a NAME reference, there are already several > potential places for that name to be resolved: > > * the current local scope > * an enclosing function scope > * the module globals > * the builtin namespace While that is true, the current scenario is (conceptually, at least) a relatively traditional set of 4 nested namespaces, with name resolution working from innermost to outermost, and with the extent of the scopes being pretty clearly defined. So although the *compiler* may have to defer a chunk of that resolution to runtime, the human reader can work with a relatively conventional model and not hit problems in the majority of cases. The problem with statement local variables is that the extent over which the name is in scope is not as clear to the human reader (the rules the *compiler* follows may be precise, but they aren't obvious to the human reader - that's the root of the debate I'm having with Chris over "what the reference implementation does isn't a sufficient spec"). In particular, assignment statements are non-obvious, as shown by the examples that triggered your suggestion of a "." prefix. [...] > Adding statement local variables into that mix *without* some form of > syntactic marker would mean taking an already complicated system, and making > it even harder to reason about correctly (especially if statement locals > interact with nested scopes differently from the way other locals in the > same scope do). Well, an alternative to a syntactic marker would be an easy-to-determine extent. That's where proposals like PEP 3150 (the "given" clause) work better, because they provide a clearer indication of the extent of the new scope. IMO, lack of a well-defined extent is a flaw of this proposal, and syntactic markers are essentially a (ugly) workaround for that flaw. > Thus the intent behind the ".NAME" suggestion is to ask whether or not it's > possible to allow for name bindings that are strictly local to a compilation > unit (i.e. without allowing dynamic runtime access to outer scopes or from > contained scopes), *without* incurring the cost of making ordinary NAME > references even more complicated to understand. ... or the cost of imposing a more user-visible indication of the extent of the scope into the proposal. Paul From ncoghlan at gmail.com Fri Mar 2 05:47:06 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Mar 2018 20:47:06 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: <5A98F19C.7010403@stoneleaf.us> References: <5A98F19C.7010403@stoneleaf.us> Message-ID: On 2 March 2018 at 16:39, Ethan Furman wrote: > On 03/01/2018 09:08 PM, Nick Coghlan wrote: > >> Adding statement local variables into that mix *without* some form of >> syntactic marker would mean taking an already >> complicated system, and making it even harder to reason about correctly >> (especially if statement locals interact with >> nested scopes differently from the way other locals in the same scope do). >> > > Seems like it would far easier and (IMHO) more useful to scale the > proposal back from a statement scope to simple expression assignment, and > the variable is whatever scope it would have been if assigned to outside > the expression (default being local, but non-local or global if already > declared as such). > Because that would put us back in the exact same problematic situation we had when "[x*x for x in sequence]" leaked the iteration variable (only worse): no function locals would be safe, since arbitrary expressions could clobber them, not just name binding operations (assignment, import statements, for loops, with statements, exception handlers, class and function definitions). > No grammatical grit on anyone's monitor, no confusion about which variable > is being accessed, and no confusion about the lifetime of that variable > (okay, no /extra/ confusion ;) . > Unfortunately, it would mean a lot more "All I did was name a repeated subexpression and now my function is behaving weirdly". > Maybe somebody could explain why a statement-local limited scope variable > is better than an ordinary well-understood local-scope variable? > Particularly why it's better enough to justify more line-noise in the > syntax. I'm willing to be convinced (not happy to, just willing ;) . > It breaks the expectation that only a well defined subset of statement can make changes to local name bindings. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Fri Mar 2 06:15:35 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Mar 2018 21:15:35 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 2 March 2018 at 19:05, Paul Moore wrote: > The problem with statement local variables is that the extent over > which the name is in scope is not as clear to the human reader (the > rules the *compiler* follows may be precise, but they aren't obvious > to the human reader - that's the root of the debate I'm having with > Chris over "what the reference implementation does isn't a sufficient > spec"). In particular, assignment statements are non-obvious, as shown > by the examples that triggered your suggestion of a "." prefix. > Those examples didn't trigger the suggestion: the suggestion was borne from the fact that I don't think it should be possible to close over statement locals. If you can't close over statement locals, then it isn't acceptable to allow this scenario: x = 12 if (True as x): def f(): return x print(x, f()) # "True True"? Or "True 12"? print(x, f()) # "12 12", but it's not obvious why it isn't "True True" By contrast, if the two kinds of local namespace are visibly different, then the nested scope *can't* give the appearance of referencing the statement local: x = 12 if (True as .x): def f(): return x print(.x, x, f()) # Clearly "True, 12, 12", since x never gets rebound print(x, f()) # Clearly "12, 12", since x never gets rebound > > [...] > > > Adding statement local variables into that mix *without* some form of > > syntactic marker would mean taking an already complicated system, and > making > > it even harder to reason about correctly (especially if statement locals > > interact with nested scopes differently from the way other locals in the > > same scope do). > > Well, an alternative to a syntactic marker would be an > easy-to-determine extent. That's where proposals like PEP 3150 (the > "given" clause) work better, because they provide a clearer indication > of the extent of the new scope. IMO, lack of a well-defined extent is > a flaw of this proposal, and syntactic markers are essentially a > (ugly) workaround for that flaw. > PEP 3150 ended up needing syntactic markers as well, to handle the forward references to names set in the `given` clause while staying within the LL(1) parsing design constraint imposed on Python's grammar. Currently it proposes `?.name` as that marker, with `?` referring to the entire given namespace, but it could equally well use `.name` instead (and if you wanted a reference to a namespace instead, you'd need to define one inside the given clause). One of the key *problems* with PEP 3150 though is that it doesn't compose nicely with other compound statements, whereas PEP 572 does (by treating each statement as its own extent - PEP 3150 then just provides a way to add a suite to statements that don't already have one of their own). > > Thus the intent behind the ".NAME" suggestion is to ask whether or not > it's > > possible to allow for name bindings that are strictly local to a > compilation > > unit (i.e. without allowing dynamic runtime access to outer scopes or > from > > contained scopes), *without* incurring the cost of making ordinary NAME > > references even more complicated to understand. > > ... or the cost of imposing a more user-visible indication of the > extent of the scope into the proposal. > Right, but that extra notation *does* convey useful information to a reader that better enables local reasoning about a piece of code. Currently, if you're looking at an unfamiliar function and see a name you don't recognise, then you need to search the whole module for that name to see whether or not it's defined anywhere. Even if it's missing, you may still need to check for dynamic injection of module level names via globals(). Seeing ".name" would be different (both for the compiler and for the human reader): if such a reference can't be resolved explicitly within the scope of the current statement, then *it's a bug* (and the compiler would be able to flag it as such at compile time). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 2 06:43:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 2 Mar 2018 22:43:21 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings Message-ID: After dozens of posts and a wide variety of useful opinions and concerns being raised, here is the newest version of PEP 572 for your debating pleasure. Formatted version: https://www.python.org/dev/peps/pep-0572/ There are now several more examples, greater clarity in edge cases, and improved wording of the actual proposal and specifications. Also, the reference implementation has been significantly enhanced, for those who wish to try this themselves. ChrisA PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018, 02-Mar-2018 Abstract ======== Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against name collisions. Rationale ========= When a subexpression is used multiple times in a list comprehension, there are currently several ways to spell this, none of which is universally accepted as ideal. A statement-local name allows any subexpression to be temporarily captured and then used multiple times. Additionally, this syntax can in places be used to remove the need to write an infinite loop with a ``break`` in it. Capturing part of a ``while`` loop's condition can improve the clarity of the loop header while still making the actual value available within the loop body. Syntax and semantics ==================== In any context where arbitrary Python expressions can be used, a named expression can appear. This must be parenthesized for clarity, and is of the form ``(expr as NAME)`` where ``expr`` is any valid Python expression, and ``NAME`` is a simple name. The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value in all retrievals for the remainder of the current statement. Just as function-local names shadow global names for the scope of the function, statement-local names shadow other names for that statement. They can also shadow each other, though actually doing this should be strongly discouraged in style guides. Assignment to statement-local names is ONLY through this syntax. Regular assignment to the same name will remove the statement-local name and affect the name in the surrounding scope (function, class, or module). Statement-local names never appear in locals() or globals(), and cannot be closed over by nested functions. Execution order and its consequences ------------------------------------ Since the statement-local name binding lasts from its point of execution to the end of the current statement, this can potentially cause confusion when the actual order of execution does not match the programmer's expectations. Some examples:: # A simple statement ends at the newline or semicolon. a = (1 as y) print(y) # NameError # The assignment ignores the SLNB - this adds one to 'a' a = (a + 1 as a) # Compound statements usually enclose everything... if (re.match(...) as m): print(m.groups(0)) print(m) # NameError # ... except when function bodies are involved... if (input("> ") as cmd): def run_cmd(): print("Running command", cmd) # NameError # ... but function *headers* are executed immediately if (input("> ") as cmd): def run_cmd(cmd=cmd): # Capture the value in the default arg print("Running command", cmd) # Works Some of these examples should be considered *bad code* and rejected by code review and/or linters; they are not, however, illegal. Example usage ============= These list comprehensions are all approximately equivalent:: # Calling the function twice stuff = [[f(x), x/f(x)] for x in range(5)] # External helper function def pair(x, value): return [value, x/value] stuff = [pair(x, f(x)) for x in range(5)] # Inline helper function stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)] # Extra 'for' loop - see also Serhiy's optimization stuff = [[y, x/y] for x in range(5) for y in [f(x)]] # Iterating over a genexp stuff = [[y, x/y] for x, y in ((x, f(x)) for x in range(5))] # Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, x/y]) # Wrapping the loop in a generator function def g(): for x in range(5): y = f(x) yield [y, x/y] stuff = list(g) # Using a statement-local name stuff = [[(f(x) as y), x/y] for x in range(5)] If calling ``f(x)`` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra ``for`` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning. Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as ``lambda`` should be avoided when ``def`` is an option. As the name's scope extends to the full current statement, even a block statement, this can be used to good effect in the header of an ``if`` or ``while`` statement:: # Current Python, not caring about function return value while input("> ") != "quit": print("You entered a command.") # Current Python, capturing return value - four-line loop header while True: command = input("> "); if command == "quit": break print("You entered:", command) # Proposed alternative to the above while (input("> ") as command) != "quit": print("You entered:", command) # See, for instance, Lib/pydoc.py if (re.search(pat, text) as match): print("Found:", match.group(0)) while (sock.read() as data): print("Received data:", data) Particularly with the ``while`` loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value. Performance costs ================= The cost of SLNBs must be kept to a minimum, particularly when they are not used; the normal case MUST NOT be measurably penalized. SLNBs are expected to be uncommon, and using many of them in a single function should definitely be discouraged. Thus the current implementation uses a linked list of SLNB cells, with the absence of such a list being the normal case. This list is used for code compilation only; once a function's bytecode has been baked in, execution of that bytecode has no performance cost compared to regular assignment. Other Python implementations may choose to do things differently, but a zero run-time cost is strongly recommended, as is a minimal compile-time cost in the case where no SLNBs are used. Open questions ============== 1. What happens if the name has already been used? ``(x, (1 as x), x)`` Currently, prior usage functions as if the named expression did not exist (following the usual lookup rules); the new name binding will shadow the other name from the point where it is evaluated until the end of the statement. Is this acceptable? Should it raise a syntax error or warning? 2. Syntactic confusion in ``except`` statements. While technically unambiguous, it is potentially confusing to humans. In Python 3.7, parenthesizing ``except (Exception as e):`` is illegal, and there is no reason to capture the exception type (as opposed to the exception instance, as is done by the regular syntax). Should this be made outright illegal, to prevent confusion? Can it be left to linters? It may also (and independently) be of value to use a subscope for the normal except clause binding, such that ``except Exception as e:`` will no longer unbind a previous use of the name ``e``. 3. Similar confusion in ``with`` statements, with the difference that there is good reason to capture the result of an expression, and it is also very common for ``__enter__`` methods to return ``self``. In many cases, ``with expr as name:`` will do the same thing as ``with (expr as name):``, adding to the confusion. 4. Should closures be able to refer to statement-local names? Either way, there will be edge cases that make no sense. Assigning to a name will "push through" the SLNB and bind to the regular name; this means that a statement ``x = x`` will promote the SLNB to full name, and thus has an impact. Closing over statement-local names, however, introduces scope and lifetime confusions, as it then becomes possible to have two functions in almost the same context, closing over the same name, referring to two different cells. Alternative proposals ===================== Proposals of this nature have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above. 1. ``where``, ``let``, ``given``:: stuff = [(y, x/y) where y = f(x) for x in range(5)] stuff = [(y, x/y) let y = f(x) for x in range(5)] stuff = [(y, x/y) given y = f(x) for x in range(5)] This brings the subexpression to a location in between the 'for' loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, ``where`` reads the most cleanly, but also has the greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` methods, as does ``tkinter.dnd.Icon`` in the standard library). 2. ``with``:: stuff = [(y, x/y) with y = f(x) for x in range(5)] As above, but reusing the `with` keyword. Doesn't read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into "longhand" for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. 3. ``with... as``:: stuff = [(y, x/y) with f(x) as y for x in range(5)] As per option 2, but using ``as`` in place of the equals sign. Aligns syntactically with other uses of ``as`` for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of ``with`` inside a comprehension would be completely different from the meaning as a stand-alone statement. 4. ``EXPR as NAME`` without parentheses:: stuff = [[f(x) as y, x/y] for x in range(5)] Omitting the parentheses from this PEP's proposed syntax introduces many syntactic ambiguities. 5. Adorning statement-local names with a leading dot:: stuff = [[(f(x) as .y), x/.y] for x in range(5)] This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable's scope is encoded into its name, making refactoring harder. This syntax is quite viable, and could be promoted to become the current recommendation if its advantages are found to outweigh its cost. 6. Allowing ``(EXPR as NAME)`` to assign to any form of name. This is exactly the same as the promoted proposal, save that the name is bound in the same scope that it would otherwise have. Any expression can assign to any name, just as it would if the ``=`` operator had been used. Discrepancies in the current implementation =========================================== 1. SLNBs are implemented using a special (and mostly-invisible) name mangling. They may sometimes appear in globals() and/or locals() with their simple or mangled names (but buggily and unreliably). They should be suppressed as though they were guinea pigs. References ========== .. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/statement-local-variables) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From p.f.moore at gmail.com Fri Mar 2 06:50:57 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Mar 2018 11:50:57 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 2 March 2018 at 11:15, Nick Coghlan wrote: > On 2 March 2018 at 19:05, Paul Moore wrote: >> >> The problem with statement local variables is that the extent over >> which the name is in scope is not as clear to the human reader (the >> rules the *compiler* follows may be precise, but they aren't obvious >> to the human reader - that's the root of the debate I'm having with >> Chris over "what the reference implementation does isn't a sufficient >> spec"). In particular, assignment statements are non-obvious, as shown >> by the examples that triggered your suggestion of a "." prefix. > > Those examples didn't trigger the suggestion: the suggestion was borne from > the fact that I don't think it should be possible to close over statement > locals. Ah, OK. If closing over statement locals isn't allowed, then yes, they are a different type of name, and you may need to distinguish them. On the other hand, I'm not sure I agree with you that it shouldn't be possible to close over statement locals. I can see that there are a lot of *difficulties* with allowing it, but that's not the same. What's your logic for saying you shouldn't be able to close over a statement local name? What is fundamentally different about them that makes them unsuitable to work like all other names in Python? > PEP 3150 ended up needing syntactic markers as well, to handle the forward > references to names set in the `given` clause while staying within the LL(1) > parsing design constraint imposed on Python's grammar. Is this basically about forward references then? Certainly the "a = (1 as a)" examples are a problem because of forward references. And the problem here is that we can't use the normal solution of simply prohibiting forward references ("Name assigned before declaration" errors) because - well, I'm not quite sure why, actually. Surely if you're naming a subexpression that's repeated, you can always just choose to name the *first* occurrence and use that name for the rest? I guess the exception is statements that introduce or bind names (assignments, for, etc). I'd still be inclined to just say prohibit such cases (if we can't, then we're back into the territory of implementation difficulties driving the design). This all still feels to me like an attempt to rescue the proposal from the issues that arise from not treating statement-local names exactly like any other name. > Right, but that extra notation *does* convey useful information to a reader > that better enables local reasoning about a piece of code. Currently, if > you're looking at an unfamiliar function and see a name you don't recognise, > then you need to search the whole module for that name to see whether or not > it's defined anywhere. Even if it's missing, you may still need to check for > dynamic injection of module level names via globals(). Hang on, that's how all existing names in Python work (and in pretty much any language that doesn't require explicit declarations). Surely no-one is trying to suggest that this is a fundamental flaw? > Seeing ".name" would be different (both for the compiler and for the human > reader): if such a reference can't be resolved explicitly within the scope > of the current statement, then *it's a bug* (and the compiler would be able > to flag it as such at compile time). Sorry, but you could use exactly that argument to propose that function local variables should be prefixed with "$". I don't buy it. I guess I remain -1 on the proposal, and nothing that's getting said about how we can make it work is doing anything to persuade me otherwise (quite the opposite). Paul From p.f.moore at gmail.com Fri Mar 2 06:56:55 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Mar 2018 11:56:55 +0000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 2 March 2018 at 11:43, Chris Angelico wrote: > After dozens of posts and a wide variety of useful opinions and > concerns being raised, here is the newest version of PEP 572 for your > debating pleasure. This is a distinct improvement - thanks for incorporating the varied feedback so well. I still remain -1, though. Reasons scattered through the other thread but mostly relating to the fact that there are subtle differences between statement-local names and other names in Python (notably that you can't close over them). I don't think the benefits of the proposal are sufficient to justify introducing a new similar-but-subtly-different type of name. Paul From ncoghlan at gmail.com Fri Mar 2 07:48:48 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 2 Mar 2018 22:48:48 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 2 March 2018 at 21:50, Paul Moore wrote: > On 2 March 2018 at 11:15, Nick Coghlan wrote: > > On 2 March 2018 at 19:05, Paul Moore wrote: > >> > >> The problem with statement local variables is that the extent over > >> which the name is in scope is not as clear to the human reader (the > >> rules the *compiler* follows may be precise, but they aren't obvious > >> to the human reader - that's the root of the debate I'm having with > >> Chris over "what the reference implementation does isn't a sufficient > >> spec"). In particular, assignment statements are non-obvious, as shown > >> by the examples that triggered your suggestion of a "." prefix. > > > > Those examples didn't trigger the suggestion: the suggestion was borne > from > > the fact that I don't think it should be possible to close over statement > > locals. > > Ah, OK. If closing over statement locals isn't allowed, then yes, they > are a different type of name, and you may need to distinguish them. On > the other hand, I'm not sure I agree with you that it shouldn't be > possible to close over statement locals. I can see that there are a > lot of *difficulties* with allowing it, but that's not the same. > > What's your logic for saying you shouldn't be able to close over a > statement local name? What is fundamentally different about them that > makes them unsuitable to work like all other names in Python? > I have two reasons, one based on design intent, one based on practicality of implementation. Starting with the practical motivation first: based on my experience preventing the implementation variable from leaking in comprehensions, I don't think it's going to be practical to allow closing over statement locals while still scoping them to the statement that sets them in any meaningful way. The difficulty of implementing that correctly is why I ended up going for the implicit-nested-scope implementation for Python 3 comprehensions in the first place. While I do assume it would technically be possible to go that way (it's only software after all), it would require some *significant* changes to the way at least CPython's compiler handles symbol analysis, and likely to the way symtable analysis is handled in other compilers as well. By contrast, if you drop the "must support closures" requirement, then the required name mangling to only nested statements to re-use the same statement local names without interfering with each other or with matching function local names gets *much* simpler (especially with a syntactic marker to distinguish statement local references from normal variable names), since all you need to track is how many levels deep you are in statement nesting within the current compilation unit. The design intent related rationale stems from the fact that closed over references can live for an arbitrarily long time (as can regular function locals in a generator or coroutine), and I think statement locals should be as reliably ephemeral as we can reasonably make them if they're going to be different enough from function locals to be worth the hassle of adding them. So that means asking a bunch of questions and deliberately giving the *opposite* answer for statement locals than I would for function locals: * Visible in locals()? Yes for function locals, no for statement locals * Visible in frame.f_locals? Yes for function locals, no for statement locals * Prevents access to the same name in outer scopes? Yes for function locals, no for statement locals * Can be closed over? Yes for function locals, no for statement locals Answer "no" to all those questions is only reasonable if statement local references *look* different from regular variable names. But I also think we need to proposing answering "No" to all of them to make statement locals potentially interesting enough to be worth considering. (Debuggers would need to access these in order to display them, so they'd have to be available on the frame object together with metadata on the code object to correctly populate them at runtime, but that's a solvable problem) While this would require non-trivial compiler changes as well, they'd be much safer updates with a lower risk of unintended consequences, since they wouldn't need to touch the existing name resolution code - they'd be their own thing, operating in parallel with the established dynamic name lookup support. > > PEP 3150 ended up needing syntactic markers as well, to handle the > forward > > references to names set in the `given` clause while staying within the > LL(1) > > parsing design constraint imposed on Python's grammar. > > Is this basically about forward references then? Certainly the "a = (1 > as a)" examples are a problem because of forward references. And the > problem here is that we can't use the normal solution of simply > prohibiting forward references ("Name assigned before declaration" > errors) because - well, I'm not quite sure why, actually. Surely if > you're naming a subexpression that's repeated, you can always just > choose to name the *first* occurrence and use that name for the rest? > I guess the exception is statements that introduce or bind names > (assignments, for, etc). I'd still be inclined to just say prohibit > such cases (if we can't, then we're back into the territory of > implementation difficulties driving the design). > Neither PEP 527 nor 3150 *needs* the syntactic markers - the compiler can figure out what is going on because it does the symbol table analysis pass before the code generation pass, and hence can tag names appropriately based on what it finds. My concern is for people reading the code, where omitting a syntactic marker also forces *humans* to read things in two passes to make sure they understand them correctly. Consider this example: x = 10 data = [x*x for i in range(10)] This will give you a 10 item list, where each item is 100. But now consider a world with statement locals, where statement locals use the same reference syntax as regular locals: x = 10 data = [x*x for i in range(10) if (12 as x)] That's a deliberately pathological example (and you can already write something similarly misleading using the "for x in [12]" trick), but the fact remains that we allow out of order evaluation in expressions in a way that we don't permit for statements. With a syntactic marker though, there's typically going to be less ambiguity about where a name comes from: x = 10 data = [.x*.x for i in range(10) if (12 as .x)] It's not a panacea (since you may still be shadowing a name from an outer statement), and adapting it for PEP 3150 isn't without it's problems (specifically, you need to allow ".x = 12" in the given clause to make the names match up), but it should be less confusing than allowing a new subexpression to interfere with name resolution semantics folks have been relying on for years. This all still feels to me like an attempt to rescue the proposal from > the issues that arise from not treating statement-local names exactly > like any other name. > If statement locals behave just like function locals, then there's no reason to add them to the language in the first place - "(expr as name)" would just become a confusing second way to spell "name = expr". It's only potentially worthwhile if statement locals offer us something that function locals don't, and the clearest potential candidate for that differentiation is to provide a greater level of assurance regarding locality of use than regular name binding operations do. > Right, but that extra notation *does* convey useful information to a > reader > > that better enables local reasoning about a piece of code. Currently, if > > you're looking at an unfamiliar function and see a name you don't > recognise, > > then you need to search the whole module for that name to see whether or > not > > it's defined anywhere. Even if it's missing, you may still need to check > for > > dynamic injection of module level names via globals(). > > Hang on, that's how all existing names in Python work (and in pretty > much any language that doesn't require explicit declarations). Surely > no-one is trying to suggest that this is a fundamental flaw? > It's good when that's what you want (which is most of the time once you've made the decision to use Python in the first place). It's not good when you're just wanting to name a subexpression to avoid evaluating it twice, and want to minimise the risk of introducing unintended side effects when doing so (hence the decision to hide comprehension iteration variables in Py3). > > Seeing ".name" would be different (both for the compiler and for the > human > > reader): if such a reference can't be resolved explicitly within the > scope > > of the current statement, then *it's a bug* (and the compiler would be > able > > to flag it as such at compile time). > > Sorry, but you could use exactly that argument to propose that > function local variables should be prefixed with "$". I don't buy it. > Function locals have been part of the language from the beginning though, rather than only being added after folks have already had years or decades to develop their intuitions about how variable name resolution works in Python. I'll also note that we *do* offer declarations to override what the compiler would infer by default based on assignment statements: global and nonlocal. > I guess I remain -1 on the proposal, and nothing that's getting said > about how we can make it work is doing anything to persuade me > otherwise (quite the opposite). > Yep, that's fair, as there are *lots* of viable alternatives to inline naming of subexpressions (each with various trade-offs). The question at hand is whether we can come up with semantics and syntax that folks actually like well enough to put to python-dev for a yes/no decision, with the pros and cons consolidated in one place. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Mar 2 08:26:38 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Mar 2018 13:26:38 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 2 March 2018 at 12:48, Nick Coghlan wrote: > The design intent related rationale stems from the fact that closed over > references can live for an arbitrarily long time (as can regular function > locals in a generator or coroutine), and I think statement locals should be > as reliably ephemeral as we can reasonably make them if they're going to be > different enough from function locals to be worth the hassle of adding them. > > So that means asking a bunch of questions and deliberately giving the > *opposite* answer for statement locals than I would for function locals: > > * Visible in locals()? Yes for function locals, no for statement locals > * Visible in frame.f_locals? Yes for function locals, no for statement > locals > * Prevents access to the same name in outer scopes? Yes for function locals, > no for statement locals > * Can be closed over? Yes for function locals, no for statement locals > > Answer "no" to all those questions is only reasonable if statement local > references *look* different from regular variable names. But I also think we > need to proposing answering "No" to all of them to make statement locals > potentially interesting enough to be worth considering. OK, that argument makes sense to me. At least in the sense that it makes a good case for how statement-local names should work *if* we adopt them. I reserve the right to change my mind, as I haven't thought this through fully, but at least superficially I'm with you this far. I will note that I don't see this as a compelling reason for *having* them, just a good analysis of how they might behave if we do. > Neither PEP 527 nor 3150 *needs* the syntactic markers - the compiler can > figure out what is going on because it does the symbol table analysis pass > before the code generation pass, and hence can tag names appropriately based > on what it finds. > > My concern is for people reading the code, where omitting a syntactic marker > also forces *humans* to read things in two passes to make sure they > understand them correctly. Consider this example: > > x = 10 > data = [x*x for i in range(10)] > > This will give you a 10 item list, where each item is 100. > > But now consider a world with statement locals, where statement locals use > the same reference syntax as regular locals: > > x = 10 > data = [x*x for i in range(10) if (12 as x)] > > That's a deliberately pathological example (and you can already write > something similarly misleading using the "for x in [12]" trick), but the > fact remains that we allow out of order evaluation in expressions in a way > that we don't permit for statements. OK. That's my concern as well. But my conclusion is different - I view this as a pretty strong argument that the complexity cost is too high to justify the feature, rather than as a difficulty we need to mitigate in order to make the feature usable. > With a syntactic marker though, there's typically going to be less ambiguity > about where a name comes from: > > x = 10 > data = [.x*.x for i in range(10) if (12 as .x)] > > It's not a panacea (since you may still be shadowing a name from an outer > statement), and adapting it for PEP 3150 isn't without it's problems > (specifically, you need to allow ".x = 12" in the given clause to make the > names match up), but it should be less confusing than allowing a new > subexpression to interfere with name resolution semantics folks have been > relying on for years. On my screen right now, I can barely see the dots. That's in email with a proportional font, so not "real coding", but it's surprising (and somewhat depressing :-() to think that probably the majority of code I read these days is in emails. So I don't think that this is a good enough fix to warrant making Tim's screen look gritty. > If statement locals behave just like function locals, then there's no reason > to add them to the language in the first place - "(expr as name)" would just > become a confusing second way to spell "name = expr". Well, the whole reason this debate keeps coming up is because people keep finding places they want to type "name = expr" but they can't, because their context isn't a statement. We're basically trying to design a replacement for "name = expr" that can be used in an expression. And the status quo is "refactor your expression into multiple statements". Doing so isn't always ideal (and particularly grates on people coming from languages where assignments are expressions, like C/C++) but it is always possible. Adding "mustn't just be another way to spell name = expr" to the requirements is an extra requirement - and arguably one that contradicts the original requirement which was "find a way to allow name = expr in expressions". Designing a broader feature which solves the original problem is a good goal, but we need to take care not to introduce scope creep or over generalisation. In this case, IMO, we're not doing that (yet!) but we are hitting design problems which are orthogonal to the original requirement. > It's only potentially worthwhile if statement locals offer us something that > function locals don't, and the clearest potential candidate for that > differentiation is to provide a greater level of assurance regarding > locality of use than regular name binding operations do. I disagree. It's *still* potentially worthwhile if it just offers a clean resolution for people who want to name complex sub-parts of a comprehension. But the bar for "clean" in that case is higher, because the benefit is pretty limited. I don't think it reaches that higher bar, but I also am not sure that the benefits of the wider proposal are sufficient to warrant the amount of lowering of the bar that is needed. (That metaphor got way too strained. I hope it still made sense). >> I guess I remain -1 on the proposal, and nothing that's getting said >> about how we can make it work is doing anything to persuade me >> otherwise (quite the opposite). > > Yep, that's fair, as there are *lots* of viable alternatives to inline > naming of subexpressions (each with various trade-offs). The question at > hand is whether we can come up with semantics and syntax that folks actually > like well enough to put to python-dev for a yes/no decision, with the pros > and cons consolidated in one place. Precisely. And the usual proviso that "status quo wins" applies strongly here, where (as noted in Chris' PEP) there are already *many* existing solutions. Paul From ncoghlan at gmail.com Fri Mar 2 09:23:16 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Mar 2018 00:23:16 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 2 March 2018 at 23:26, Paul Moore wrote: > Adding "mustn't just be another way to spell name = expr" to the > requirements is an extra requirement - and arguably one that > contradicts the original requirement which was "find a way to allow > name = expr in expressions". > I consider allowing the semantic equivalent of "name = expr" as part of arbitrary expressions to be an anti-requirement, not a requirement, for three reasons: 1. It means that arbitrary statements may accidentally clobber local variables that we care about. We used to have this for comprehension iteration variables, and it was so widely deplored that we broke compatibility with the variable overwriting behaviour in Python 3.0. 2. It means that naming a subexpression may significantly increase the longevity of that reference (especially in generators and coroutines). 3. It means that naming a subexpression will define a new class or module attribute when used in a class or module scope Folks do sometimes *ask* for these semantics, as it's the most obvious behaviour to ask for when it comes to naming subexpressions, but it would be a really bad idea to say "yes" to those requests. Chris's PEP addresses all those potential problems: 1. Statement locals explicitly avoid overwriting regular function locals (although they may shadow them in the current PEP - the '.name' syntax avoids even that) 2. Statement local bindings are cleared after each statement, so they only significantly extend the reference lifespan when used in compound statement headers or directly in the same statement as a yield or await. Even in comprehensions they'll typically be cleared on the next loop iteration. 3. Statement locals aren't added to locals() or globals(), so they never show up as module or class attributes either The eventual conclusion may still be "The proposal adds too much additional complexity without an adequate corresponding gain in expressiveness", but that complexity does have a solid rationale behind it. (Although it may be worth explicitly answering "Why not use the exact same semantics as 'name = expr'? in the PEP itself, since it's a reasonable question to ask) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Fri Mar 2 09:25:13 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Fri, 2 Mar 2018 15:25:13 +0100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: Hi all, I would like to observe that there is already a way to bind names in an expression: a = (lambda b: [b, b])(b=f()) Currently this is not so performant, but there is nothing stopping the compiler from special-casing such an IIFE (Immediate Invoked Function Expression), to use the Javascript terminology. It would then be fast in, say, Python 3.8, but still work in earlier versions. Stephan 2018-03-02 14:26 GMT+01:00 Paul Moore : > On 2 March 2018 at 12:48, Nick Coghlan wrote: > > The design intent related rationale stems from the fact that closed over > > references can live for an arbitrarily long time (as can regular function > > locals in a generator or coroutine), and I think statement locals should > be > > as reliably ephemeral as we can reasonably make them if they're going to > be > > different enough from function locals to be worth the hassle of adding > them. > > > > So that means asking a bunch of questions and deliberately giving the > > *opposite* answer for statement locals than I would for function locals: > > > > * Visible in locals()? Yes for function locals, no for statement locals > > * Visible in frame.f_locals? Yes for function locals, no for statement > > locals > > * Prevents access to the same name in outer scopes? Yes for function > locals, > > no for statement locals > > * Can be closed over? Yes for function locals, no for statement locals > > > > Answer "no" to all those questions is only reasonable if statement local > > references *look* different from regular variable names. But I also > think we > > need to proposing answering "No" to all of them to make statement locals > > potentially interesting enough to be worth considering. > > OK, that argument makes sense to me. At least in the sense that it > makes a good case for how statement-local names should work *if* we > adopt them. I reserve the right to change my mind, as I haven't > thought this through fully, but at least superficially I'm with you > this far. > > I will note that I don't see this as a compelling reason for *having* > them, just a good analysis of how they might behave if we do. > > > Neither PEP 527 nor 3150 *needs* the syntactic markers - the compiler can > > figure out what is going on because it does the symbol table analysis > pass > > before the code generation pass, and hence can tag names appropriately > based > > on what it finds. > > > > My concern is for people reading the code, where omitting a syntactic > marker > > also forces *humans* to read things in two passes to make sure they > > understand them correctly. Consider this example: > > > > x = 10 > > data = [x*x for i in range(10)] > > > > This will give you a 10 item list, where each item is 100. > > > > But now consider a world with statement locals, where statement locals > use > > the same reference syntax as regular locals: > > > > x = 10 > > data = [x*x for i in range(10) if (12 as x)] > > > > That's a deliberately pathological example (and you can already write > > something similarly misleading using the "for x in [12]" trick), but the > > fact remains that we allow out of order evaluation in expressions in a > way > > that we don't permit for statements. > > OK. That's my concern as well. But my conclusion is different - I view > this as a pretty strong argument that the complexity cost is too high > to justify the feature, rather than as a difficulty we need to > mitigate in order to make the feature usable. > > > With a syntactic marker though, there's typically going to be less > ambiguity > > about where a name comes from: > > > > x = 10 > > data = [.x*.x for i in range(10) if (12 as .x)] > > > > It's not a panacea (since you may still be shadowing a name from an outer > > statement), and adapting it for PEP 3150 isn't without it's problems > > (specifically, you need to allow ".x = 12" in the given clause to make > the > > names match up), but it should be less confusing than allowing a new > > subexpression to interfere with name resolution semantics folks have been > > relying on for years. > > On my screen right now, I can barely see the dots. That's in email > with a proportional font, so not "real coding", but it's surprising > (and somewhat depressing :-() to think that probably the majority of > code I read these days is in emails. > > So I don't think that this is a good enough fix to warrant making > Tim's screen look gritty. > > > If statement locals behave just like function locals, then there's no > reason > > to add them to the language in the first place - "(expr as name)" would > just > > become a confusing second way to spell "name = expr". > > Well, the whole reason this debate keeps coming up is because people > keep finding places they want to type "name = expr" but they can't, > because their context isn't a statement. We're basically trying to > design a replacement for "name = expr" that can be used in an > expression. And the status quo is "refactor your expression into > multiple statements". Doing so isn't always ideal (and particularly > grates on people coming from languages where assignments are > expressions, like C/C++) but it is always possible. > > Adding "mustn't just be another way to spell name = expr" to the > requirements is an extra requirement - and arguably one that > contradicts the original requirement which was "find a way to allow > name = expr in expressions". > > Designing a broader feature which solves the original problem is a > good goal, but we need to take care not to introduce scope creep or > over generalisation. In this case, IMO, we're not doing that (yet!) > but we are hitting design problems which are orthogonal to the > original requirement. > > > It's only potentially worthwhile if statement locals offer us something > that > > function locals don't, and the clearest potential candidate for that > > differentiation is to provide a greater level of assurance regarding > > locality of use than regular name binding operations do. > > I disagree. It's *still* potentially worthwhile if it just offers a > clean resolution for people who want to name complex sub-parts of a > comprehension. But the bar for "clean" in that case is higher, because > the benefit is pretty limited. I don't think it reaches that higher > bar, but I also am not sure that the benefits of the wider proposal > are sufficient to warrant the amount of lowering of the bar that is > needed. (That metaphor got way too strained. I hope it still made > sense). > > > >> I guess I remain -1 on the proposal, and nothing that's getting said > >> about how we can make it work is doing anything to persuade me > >> otherwise (quite the opposite). > > > > Yep, that's fair, as there are *lots* of viable alternatives to inline > > naming of subexpressions (each with various trade-offs). The question at > > hand is whether we can come up with semantics and syntax that folks > actually > > like well enough to put to python-dev for a yes/no decision, with the > pros > > and cons consolidated in one place. > > Precisely. > > And the usual proviso that "status quo wins" applies strongly here, > where (as noted in Chris' PEP) there are already *many* existing > solutions. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 2 09:51:55 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 01:51:55 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On Fri, Mar 2, 2018 at 10:56 PM, Paul Moore wrote: > On 2 March 2018 at 11:43, Chris Angelico wrote: >> After dozens of posts and a wide variety of useful opinions and >> concerns being raised, here is the newest version of PEP 572 for your >> debating pleasure. > > This is a distinct improvement - thanks for incorporating the varied > feedback so well. > > I still remain -1, though. Reasons scattered through the other thread > but mostly relating to the fact that there are subtle differences > between statement-local names and other names in Python (notably that > you can't close over them). I don't think the benefits of the proposal > are sufficient to justify introducing a new > similar-but-subtly-different type of name. That's fine :) I'm under no illusions that everyone will adore this proposal (hey, I'm no more than +0.5 on it myself). The greatest goal of this PEP is to record the arguments for and against each of the viable alternatives, as a long-term document. ChrisA From p.f.moore at gmail.com Fri Mar 2 09:52:04 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 2 Mar 2018 14:52:04 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On 2 March 2018 at 14:23, Nick Coghlan wrote: > On 2 March 2018 at 23:26, Paul Moore wrote: >> >> Adding "mustn't just be another way to spell name = expr" to the >> requirements is an extra requirement - and arguably one that >> contradicts the original requirement which was "find a way to allow >> name = expr in expressions". > > > I consider allowing the semantic equivalent of "name = expr" as part of > arbitrary expressions to be an anti-requirement, not a requirement, for > three reasons: > > 1. It means that arbitrary statements may accidentally clobber local > variables that we care about. We used to have this for comprehension > iteration variables, and it was so widely deplored that we broke > compatibility with the variable overwriting behaviour in Python 3.0. > > 2. It means that naming a subexpression may significantly increase the > longevity of that reference (especially in generators and coroutines). > > 3. It means that naming a subexpression will define a new class or module > attribute when used in a class or module scope > > Folks do sometimes *ask* for these semantics, as it's the most obvious > behaviour to ask for when it comes to naming subexpressions, but it would be > a really bad idea to say "yes" to those requests. > > Chris's PEP addresses all those potential problems: > > 1. Statement locals explicitly avoid overwriting regular function locals > (although they may shadow them in the current PEP - the '.name' syntax > avoids even that) > 2. Statement local bindings are cleared after each statement, so they only > significantly extend the reference lifespan when used in compound statement > headers or directly in the same statement as a yield or await. Even in > comprehensions they'll typically be cleared on the next loop iteration. > 3. Statement locals aren't added to locals() or globals(), so they never > show up as module or class attributes either > > The eventual conclusion may still be "The proposal adds too much additional > complexity without an adequate corresponding gain in expressiveness", but > that complexity does have a solid rationale behind it. (Although it may be > worth explicitly answering "Why not use the exact same semantics as 'name = > expr'? in the PEP itself, since it's a reasonable question to ask) OK, understood. Yes, I think it would be worth adding that question to the PEP. I also think it would be worth being more explicit in the body of the PEP over how[1] the binding of statement-local names is fundamentally different from assignment, as that certainly wasn't obvious to me, and may not be to others (particularly people reading it with the point of view "This satisfies my request to allow assignment in expressions"). I don't really have much more to add in that case. I don't personally think the added complexity is justified by the benefits, but I've already said that and it's been noted - so thanks for helping me understand the details of the proposal better, but it hasn't changed my view much in the end. Paul [1] The "why" is covered by the question you suggested. From rhodri at kynesim.co.uk Fri Mar 2 09:53:06 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Fri, 2 Mar 2018 14:53:06 +0000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> On 02/03/18 11:43, Chris Angelico wrote: > After dozens of posts and a wide variety of useful opinions and > concerns being raised, here is the newest version of PEP 572 for your > debating pleasure. I haven't said this yet, so thanks Chris for putting this all together. Even if the result is a rejected PEP, at least we have everything in one place. [snip] > # Compound statements usually enclose everything... > if (re.match(...) as m): > print(m.groups(0)) > print(m) # NameError This (and the equivalent in while loops) is the big win in the PEP, in my opinion. The number of ugly loops I've had to write in Python because I can't write "while (something_to_do() as event):"... +1 on this. > # Using a statement-local name > stuff = [[(f(x) as y), x/y] for x in range(5)] As Paul said, the asymmetry of this bothers me a lot. It doesn't read naturally to me. -1 on this. > 1. What happens if the name has already been used? ``(x, (1 as x), x)`` > Currently, prior usage functions as if the named expression did not > exist (following the usual lookup rules); the new name binding will > shadow the other name from the point where it is evaluated until the > end of the statement. Is this acceptable? Should it raise a syntax > error or warning? I wouldn't worry too much about this case. Anyone gratuitously reusing names like that deserves all that will be coming to them. > Alternative proposals > ===================== > 3. ``with... as``:: > > stuff = [(y, x/y) with f(x) as y for x in range(5)] > > As per option 2, but using ``as`` in place of the equals sign. Aligns > syntactically with other uses of ``as`` for name binding, but a simple > transformation to for-loop longhand would create drastically different > semantics; the meaning of ``with`` inside a comprehension would be > completely different from the meaning as a stand-alone statement. Honestly I prefer this syntax for comprehensions. It doesn't read perfectly but it's good enough (though I am a mathematician by original training, so set notation works for me anyway), and the syntax is clear and limited. I'm not sure the case for fully general statement-local variables has been made. So, counter-proposal(s): 1. Allow "(f() as a)" in the conditions of "if" and "while" statements, after some arguing as to whether "a" is a special snowflake or just a normal local variable. 2. Add a "with" clause to comprehensions to make comprehension-local variables (presumably the same class of thing as the iteration variables). -- Rhodri James *-* Kynesim Ltd From robertve92 at gmail.com Fri Mar 2 10:03:31 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Fri, 2 Mar 2018 16:03:31 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: +1 on extracting the big win for "if" and "while" (the regex case is wonderul). It would be see as an "extended if/while" rather than a general statement assignation. +1 on list comprehensions, even if I prefer the [(y, x/y) with y = f(x) for x in range(5)] or [(y, x/y) for x in range(5) with y = f(x)] syntax (more close to "for y in [ f(x) ]". Le 2 mars 2018 15:54, "Rhodri James" a ?crit : On 02/03/18 11:43, Chris Angelico wrote: > After dozens of posts and a wide variety of useful opinions and > concerns being raised, here is the newest version of PEP 572 for your > debating pleasure. > I haven't said this yet, so thanks Chris for putting this all together. Even if the result is a rejected PEP, at least we have everything in one place. [snip] # Compound statements usually enclose everything... > if (re.match(...) as m): > print(m.groups(0)) > print(m) # NameError > This (and the equivalent in while loops) is the big win in the PEP, in my opinion. The number of ugly loops I've had to write in Python because I can't write "while (something_to_do() as event):"... +1 on this. # Using a statement-local name > stuff = [[(f(x) as y), x/y] for x in range(5)] > As Paul said, the asymmetry of this bothers me a lot. It doesn't read naturally to me. -1 on this. 1. What happens if the name has already been used? ``(x, (1 as x), x)`` > Currently, prior usage functions as if the named expression did not > exist (following the usual lookup rules); the new name binding will > shadow the other name from the point where it is evaluated until the > end of the statement. Is this acceptable? Should it raise a syntax > error or warning? > I wouldn't worry too much about this case. Anyone gratuitously reusing names like that deserves all that will be coming to them. Alternative proposals > ===================== > > 3. ``with... as``:: > > stuff = [(y, x/y) with f(x) as y for x in range(5)] > > As per option 2, but using ``as`` in place of the equals sign. Aligns > syntactically with other uses of ``as`` for name binding, but a simple > transformation to for-loop longhand would create drastically different > semantics; the meaning of ``with`` inside a comprehension would be > completely different from the meaning as a stand-alone statement. > Honestly I prefer this syntax for comprehensions. It doesn't read perfectly but it's good enough (though I am a mathematician by original training, so set notation works for me anyway), and the syntax is clear and limited. I'm not sure the case for fully general statement-local variables has been made. So, counter-proposal(s): 1. Allow "(f() as a)" in the conditions of "if" and "while" statements, after some arguing as to whether "a" is a special snowflake or just a normal local variable. 2. Add a "with" clause to comprehensions to make comprehension-local variables (presumably the same class of thing as the iteration variables). -- Rhodri James *-* Kynesim Ltd _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Fri Mar 2 10:18:39 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Fri, 2 Mar 2018 16:18:39 +0100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: <3c8a2267-1605-8e47-7e62-f79b230e7aec@btinternet.com> Message-ID: Sorry, answer to an old post but I just realized I didn't use the correct email address... (See below to see which message I'm answering). That's what I said on github, there are two different use cases : *ListComp* Vs *AnyExpressions* : 1) List comprehension : [y+2 for x in range(5) for y in [ x+1 ]] ? [y+2 for x in range(5) with y = x+1] 2) Any expression : print(y + 2 with x = 2) So, IF the syntax is allowed in *both* cases, then yes, the "any expression case" would introduced a "There is not one obvious way to do it". Now introducing *scoping*, the "any expression case" can become handy because it would create local variables in parenthesis : x = 2 print([y, x/y] with y = x + 2) print(y) # NameError It would structure expressions and would be in the same idea as PEP 3150 but with a less drastic syntax and more simple use case. Maybe both cases are completely different and your we would have 3 proposals : 1) "EXPR as NAME" ? to easily reuse an expression used multiple times in an expression. 2) "EXPR with NAME = EXPR" (any expression) ? same purpose, different syntax 3) "[EXPR for x in ITERABLE with NAME = EXPR] ? to allow reuse of an expression in a list comprehension. Maybe the only use cases are in list comprehensions and would be an improvement to the for y in [ f(x) ] not obvious syntax (that was my original thought weeks ago and in June of last year). Your syntax gives a proof of concept implementation The AnyExpression case had a proof of concept in thektulu (branch where-expr) Should we create "yet another pep" for each of this use cases or is it part of a big same idea ? Cheers, Robert Le 28 f?vr. 2018 22:53, "Chris Angelico" a ?crit : On Thu, Mar 1, 2018 at 8:38 AM, Robert Vanden Eynde wrote: > Le 28 f?vr. 2018 11:43, "Chris Angelico" a ?crit : >> If you aren't using the variable multiple times, there's no point >> giving it a name. Unless I'm missing something here? > > Yes, variables are not there "just because we reuse them", but also to > include temporary variables to better understand the code. > Same for functions, you could inline functions when used only once, but you > introduce them for clarity no ? Sure, but if you're creating temporaries for clarity, you should probably make them regular variables, not statement-local ones. If you're going to use something only once and you don't want to break it out into a separate statement, just use a comment. > > ``` > a = v ** 2 / R # the acceleration in a circular motion > f = m * a # law of Newton > ``` > > could be written as > > ``` > f = m * (v ** 2 / R) # compute the force, trivial > ``` > > But having temporary variables help a lot to understand the code, otherwise > why would we create temporary variables ? > I can give you an example where you do a process and each time the variable > is used only one. Neither of your examples needs SLNBs. >> Scoping is a fundamental part of both my proposal and the others I've >> seen here. (BTW, that would be a NameError, not a SyntaxError; it's >> perfectly legal to ask for the name 'y', it just hasn't been given any >> value.) By my definition, the variable is locked to the statement that >> created it, even if that's a compound statement. By the definition of >> a "(expr given var = expr)" proposal, it would be locked to that >> single expression. > > Confer the discussion on scoping on github > (https://github.com/python/peps/commit/2b4ca20963a24cf5faac054226857e a9705471e5) > : > > """ > In the current implementation it looks like it is like a regular assignment > (function local then). > > Therefore in the expression usage, the usefulness would be debatable (just > assign before). > > But in a list comprehension after the for (as I mentioned in my mail), aka. > when used as a replacement for for y in [ x + 1 ] this would make sense. > > But I think that it would be much better to have a local scope, in the > parenthesis. So that print(y+2 where y = x + 1) wouldn't leak y. And when > there are no parenthesis like in a = y+2 where y = x+1, it would imply one, > giving the same effect as a = (y+2 where y = x+1). Moreover, it would > naturally shadow variables in the outermost scope. So the question is: what is the benefit of the local name 'y'? In any non-trivial example, it's not going to fit in a single line, so you have to either wrap it as a single expression (one that's been made larger by the "where" clause at the end), or break it out into a real variable as a separate assignment. ChrisA _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 2 11:04:17 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 03:04:17 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: On Sat, Mar 3, 2018 at 1:53 AM, Rhodri James wrote: > On 02/03/18 11:43, Chris Angelico wrote: >> >> After dozens of posts and a wide variety of useful opinions and >> concerns being raised, here is the newest version of PEP 572 for your >> debating pleasure. > > > I haven't said this yet, so thanks Chris for putting this all together. Even > if the result is a rejected PEP, at least we have everything in one place. No problem. And I agree, a rejected PEP is still a successful result here. (Am I going to get a reputation for captaining dead PEPs?) >> # Compound statements usually enclose everything... >> if (re.match(...) as m): >> print(m.groups(0)) >> print(m) # NameError > > > This (and the equivalent in while loops) is the big win in the PEP, in my > opinion. The number of ugly loops I've had to write in Python because I > can't write "while (something_to_do() as event):"... +1 on this. > >> # Using a statement-local name >> stuff = [[(f(x) as y), x/y] for x in range(5)] > > > As Paul said, the asymmetry of this bothers me a lot. It doesn't read > naturally to me. -1 on this. Interesting. I fully expected to get a lot more backlash for the if/while usage, but a number of people are saying that that's the only (or the biggest) part of this proposal that they like. ChrisA From mertz at gnosis.cx Fri Mar 2 12:34:51 2018 From: mertz at gnosis.cx (David Mertz) Date: Fri, 2 Mar 2018 09:34:51 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: I remain -1 on the PEP, but thank you very much Chris for the great work writing it and the extra clarifications in it. The part I like most about the proposal is the use in blocks, like: if (re.match(...) as m): print(m.groups(0)) if (input("> ") as cmd): def run_cmd(cmd=cmd): # Capture the value in the default arg print("Running command", cmd) # Works But what I like has nothing much to do with scope limitation. It is only about the idea of introducing a variable into a block. We can do that with a context manager. No, there's no cleanup of the namespace in a straightforward approach, but using e.g. `_x` as a convention for names meant to be transient is already established practice and requires no special syntax. So right now, I can do these: class bind(object): def __init__(self, *args): self.args = args def __enter__(self): return self.args[0] if len(self.args)==1 else self.args def __exit__(self, *args): pass >>> with bind(sqrt(2)) as _a: ... print(_a) 1.4142135623730951 >>> with bind(sqrt(2), log(2)) as (a, b): ... print(a, b, a+b) 1.4142135623730951 0.6931471805599453 2.1073607429330403 This would cover 98% of the cases that I would want with the proposed statement-local name bindings. I suppose I could write something much more obscure that poked into the call stack and actually deleted names from scopes in the context manager. But really I rarely care about the temporary use of a few names, especially if the same few temporary names are used repeatedly for these things. On Fri, Mar 2, 2018 at 8:04 AM, Chris Angelico wrote: > On Sat, Mar 3, 2018 at 1:53 AM, Rhodri James wrote: > > On 02/03/18 11:43, Chris Angelico wrote: > >> > >> After dozens of posts and a wide variety of useful opinions and > >> concerns being raised, here is the newest version of PEP 572 for your > >> debating pleasure. > > > > > > I haven't said this yet, so thanks Chris for putting this all together. > Even > > if the result is a rejected PEP, at least we have everything in one > place. > > No problem. And I agree, a rejected PEP is still a successful result here. > > (Am I going to get a reputation for captaining dead PEPs?) > > >> # Compound statements usually enclose everything... > >> if (re.match(...) as m): > >> print(m.groups(0)) > >> print(m) # NameError > > > > > > This (and the equivalent in while loops) is the big win in the PEP, in my > > opinion. The number of ugly loops I've had to write in Python because I > > can't write "while (something_to_do() as event):"... +1 on this. > > > >> # Using a statement-local name > >> stuff = [[(f(x) as y), x/y] for x in range(5)] > > > > > > As Paul said, the asymmetry of this bothers me a lot. It doesn't read > > naturally to me. -1 on this. > > Interesting. I fully expected to get a lot more backlash for the > if/while usage, but a number of people are saying that that's the only > (or the biggest) part of this proposal that they like. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Mar 2 12:51:19 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 02 Mar 2018 09:51:19 -0800 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: <5A98F19C.7010403@stoneleaf.us> Message-ID: <5A998F17.4000606@stoneleaf.us> On 03/02/2018 02:47 AM, Nick Coghlan wrote: > On 2 March 2018 at 16:39, Ethan Furman wrote: >> On 03/01/2018 09:08 PM, Nick Coghlan wrote: >>> Adding statement local variables into that mix *without* some form of syntactic marker would mean taking an already >>> complicated system, and making it even harder to reason about correctly (especially if statement locals interact >>> with nested scopes differently from the way other locals in the same scope do). >> >> Seems like it would far easier and (IMHO) more useful to scale the proposal back from a statement scope to simple >> expression assignment, and the variable is whatever scope it would have been if assigned to outside the expression >> (default being local, but non-local or global if already declared as such). > > Because that would put us back in the exact same problematic situation we had when "[x*x for x in sequence]" leaked the > iteration variable (only worse): no function locals would be safe, since arbitrary expressions could clobber them, not > just name binding operations (assignment, import statements, for loops, with statements, exception handlers, class and > function definitions). Ah, right. Since the PEP primarily covers comprehensions, but then went on to discuss multi-line statements, I had forgotten the comprehension part. The answer is easy: assignment expressions in comprehensions stay inside comprehensions, just like other inner comprehension variables already do (function sub-scope, after all). Thank you for pointing that out. >> Maybe somebody could explain why a statement-local limited scope variable is better than an ordinary well-understood >> local-scope variable? Particularly why it's better enough to justify more line-noise in the syntax. I'm willing to >> be convinced (not happy to, just willing ;) . > > It breaks the expectation that only a well defined subset of statement can make changes to local name bindings. We already use the keyword "as" to assign names, so there is nothing extra there -- the only change is that we can now use it in one more place. Are those valid counter-arguments, or is there still something I am missing? -- ~Ethan~ From carl at bordum.dk Fri Mar 2 13:00:33 2018 From: carl at bordum.dk (Carl Bordum Hansen) Date: Fri, 2 Mar 2018 19:00:33 +0100 Subject: [Python-ideas] breakpoint(): print(*args, **kwargs) before entering pdb Message-ID: <14d2dc8a-f7fd-011e-5525-c295a9a7ee49@bordum.dk> I played around with a newer Python build, but when using the new `breakpoint` builtin, I missed my weapon of choice: dirty print-debugging. I suggest we combine forces and make the default `sys.breakpointhook` forward *args and **kwargs to print before entering pdb. - Carl Bordum Hansen From ethan at stoneleaf.us Fri Mar 2 13:27:05 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 02 Mar 2018 10:27:05 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: <5A999779.4050104@stoneleaf.us> On 03/02/2018 03:43 AM, Chris Angelico wrote: > PEP: 572 > Title: Syntax for Statement-Local Name Bindings > Author: Chris Angelico Looks nice, thanks for the updates! > Alternative proposals > ===================== > 6. Allowing ``(EXPR as NAME)`` to assign to any form of name. > > This is exactly the same as the promoted proposal, save that the name is > bound in the same scope that it would otherwise have. Any expression can > assign to any name, just as it would if the ``=`` operator had been used. Just to clarify: "bound in the same scope" means exactly that -- if the (EXPR as NAME) is in a comprehension, then it is local /to that comprehension/ (emphasis because I didn't adequately convey that it my initial response to Nick). Still -1 to the PEP as written. (It would be positive for alternative 6. ;) -- ~Ethan~ From jelle.zijlstra at gmail.com Fri Mar 2 13:27:18 2018 From: jelle.zijlstra at gmail.com (Jelle Zijlstra) Date: Fri, 2 Mar 2018 10:27:18 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: 2018-03-02 7:03 GMT-08:00 Robert Vanden Eynde : > +1 on extracting the big win for "if" and "while" (the regex case is > wonderul). It would be see as an "extended if/while" rather than a general > statement assignation. > > I wonder if we could have a more limited change to the language that would allow only the as/while use cases. Specifically, that means we could do: while do_something() as x: print(x) if re.match(r'.*', text) as match: print(match.groups()) Parentheses would no longer be necessary for syntactic ambiguity, and there is no real need for new scoping rules?we could just create new locals. This would alleviate some of the problems with the current proposal, such as complicated scoping rules and ugly syntax in comprehensions. > +1 on list comprehensions, even if I prefer the > [(y, x/y) with y = f(x) for x in range(5)] or [(y, x/y) for x in range(5) > with y = f(x)] syntax (more close to "for y in [ f(x) ]". > > Le 2 mars 2018 15:54, "Rhodri James" a ?crit : > > On 02/03/18 11:43, Chris Angelico wrote: > >> After dozens of posts and a wide variety of useful opinions and >> concerns being raised, here is the newest version of PEP 572 for your >> debating pleasure. >> > > I haven't said this yet, so thanks Chris for putting this all together. > Even if the result is a rejected PEP, at least we have everything in one > place. > > [snip] > > > # Compound statements usually enclose everything... >> if (re.match(...) as m): >> print(m.groups(0)) >> print(m) # NameError >> > > This (and the equivalent in while loops) is the big win in the PEP, in my > opinion. The number of ugly loops I've had to write in Python because I > can't write "while (something_to_do() as event):"... +1 on this. > > > # Using a statement-local name >> stuff = [[(f(x) as y), x/y] for x in range(5)] >> > > As Paul said, the asymmetry of this bothers me a lot. It doesn't read > naturally to me. -1 on this. > > > 1. What happens if the name has already been used? ``(x, (1 as x), x)`` >> Currently, prior usage functions as if the named expression did not >> exist (following the usual lookup rules); the new name binding will >> shadow the other name from the point where it is evaluated until the >> end of the statement. Is this acceptable? Should it raise a syntax >> error or warning? >> > > I wouldn't worry too much about this case. Anyone gratuitously reusing > names like that deserves all that will be coming to them. > > Alternative proposals >> ===================== >> >> 3. ``with... as``:: >> >> stuff = [(y, x/y) with f(x) as y for x in range(5)] >> >> As per option 2, but using ``as`` in place of the equals sign. Aligns >> syntactically with other uses of ``as`` for name binding, but a simple >> transformation to for-loop longhand would create drastically different >> semantics; the meaning of ``with`` inside a comprehension would be >> completely different from the meaning as a stand-alone statement. >> > > Honestly I prefer this syntax for comprehensions. It doesn't read > perfectly but it's good enough (though I am a mathematician by original > training, so set notation works for me anyway), and the syntax is clear and > limited. > > I'm not sure the case for fully general statement-local variables has been > made. > > So, counter-proposal(s): > > 1. Allow "(f() as a)" in the conditions of "if" and "while" statements, > after some arguing as to whether "a" is a special snowflake or just a > normal local variable. > > 2. Add a "with" clause to comprehensions to make comprehension-local > variables (presumably the same class of thing as the iteration variables). > > -- > Rhodri James *-* Kynesim Ltd > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Fri Mar 2 13:30:38 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Fri, 02 Mar 2018 10:30:38 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: <5A99984E.9060305@brenbarn.net> On 2018-03-02 03:43, Chris Angelico wrote: > After dozens of posts and a wide variety of useful opinions and > concerns being raised, here is the newest version of PEP 572 for your > debating pleasure. After following the discussion here, I think I've come to the conclusion that something like "EXPR with NAME as VALUE" is better, and that's because the "out-of-order" execution there is actually a benefit. The example that gave me the most pause in this thread was Paul Moore's involving the quadratic formula (slightly emended here :-): x = [(-b + sqrt((b**2 - 4*a*c as D))/(2*a), (-b + sqrt(D)/(2*a)] To me this is a classic example of the kind of thing I would want to do with a statement-local binding, so whatever syntax is chosen won't be useful to me if it doesn't nicely work in situations like this. But I don't find the above example nice, for more or less the reason Paul cited: the need to assign the expression "inline" (at the point where it's first defined) creates an asymmetry that masks the fact that it's the same expression being re-used elsewhere --- and it's exactly that re-use that I would want to HIGHLIGHT with a statement-local binding. The alternative with the name binding at the end is much better: x = [(-b + sqrt(D))/(2*a), (-b - sqrt(D))/(2*a) with b**2 - 4*a*c as D] To my mind, it's a real desideratum for a statement-local binding that it should pull the bound expression OUT of the enclosing context. The overall statement gains in readability only if the reader can easily see a shared element across multiple parts. If I can't tell at a glance that the two roots both involve the same value D, there's little point in having a statement-local binding for it. (It still may gain in computational cost, since the expression won't have to be evaluated multiple times, but I see that as a much less important benefit than readability.) Also note that the version above comes perilously close to the existing solution with a regular local variable: D = b**2 - 4*a*c x = [(-b + sqrt(D))/(2*a), (-b - sqrt(D))/(2*a)] The only difference is that now D is "leaked" to following code. Nick Coghlan has argued that there's no point in an inline-assignment construct if it's not somehow local, since a big part of its purpose is to simplify reasoning by avoiding any stomping on "real" local variables. But if that's the case, maybe what we want is actually another thing that's been discussed many times on this list, namely something like a with-block that can define "super-local" variables that disappear at the end of the block: with b**2 - 4*a*c as D: x = [(-b + sqrt(D))/(2*a), (-b - sqrt(D))/(2*a)] This the same as my earlier version with "with", except the with clause comes at the beginning. There's no need to go into extreme detail here on these proposals as they're not really what's proposed by this PEP. But my point is that, from my perspective, they have something crucial that the current proposal lacks: they explicitly separate the *definition* of the shared expression from its *use* within the statement. Having to do the name-binding inline at the place where the re-used expression happens to occur makes the overall construct LESS readable for me, not more, so I'm -1 on the current proposal. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From rosuav at gmail.com Fri Mar 2 13:33:04 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 05:33:04 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: On Sat, Mar 3, 2018 at 5:27 AM, Jelle Zijlstra wrote: > > > 2018-03-02 7:03 GMT-08:00 Robert Vanden Eynde : >> >> +1 on extracting the big win for "if" and "while" (the regex case is >> wonderul). It would be see as an "extended if/while" rather than a general >> statement assignation. >> > > I wonder if we could have a more limited change to the language that would > allow only the as/while use cases. Specifically, that means we could do: > > while do_something() as x: > print(x) > > if re.match(r'.*', text) as match: > print(match.groups()) > > Parentheses would no longer be necessary for syntactic ambiguity, and there > is no real need for new scoping rules?we could just create new locals. > > This would alleviate some of the problems with the current proposal, such as > complicated scoping rules and ugly syntax in comprehensions. The trouble with this is that it's extremely restricted. It works for the regex case, because re.match() has been specifically written to function as either "return a match object" or "return true/false to indicate a match", by guaranteeing to return a truthy object or None, and nothing else. It would NOT work for anything where the bool() of the desired object doesn't exactly match the loop's condition. For instance, if do_something() returns None when it's done, but might return an empty string during operation, the correct condition is "while do_something() is not None" - and you can't capture the whole condition any more. To be able to write "while (do_something() as x) is not None:", you have to have the more general syntax that's used in this PEP. ChrisA From ethan at stoneleaf.us Fri Mar 2 13:36:45 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 02 Mar 2018 10:36:45 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: <5A9999BD.60907@stoneleaf.us> On 03/02/2018 08:04 AM, Chris Angelico wrote: > On Sat, Mar 3, 2018 at 1:53 AM, Rhodri James wrote: >> On 02/03/18 11:43, Chris Angelico wrote: >>> # Compound statements usually enclose everything... >>> if (re.match(...) as m): >>> print(m.groups(0)) >>> print(m) # NameError >> >> This (and the equivalent in while loops) is the big win in the PEP, in my >> opinion. The number of ugly loops I've had to write in Python because I >> can't write "while (something_to_do() as event):"... +1 on this. >> >>> # Using a statement-local name >>> stuff = [[(f(x) as y), x/y] for x in range(5)] >> >> As Paul said, the asymmetry of this bothers me a lot. It doesn't read >> naturally to me. -1 on this. > > Interesting. I fully expected to get a lot more backlash for the > if/while usage, but a number of people are saying that that's the only > (or the biggest) part of this proposal that they like. This is the part of the PEP that I really like as well -- far more useful to me than the list-comp examples. I could even be +0.5 if the "if/while/for" compound statements were kept and the comprehensions dropped -- but not with leading dots -- I would rather do $ than . . Kind of like decorators, they shouldn't be used all that often. -- ~Ethan~ From rosuav at gmail.com Fri Mar 2 13:42:33 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 05:42:33 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5A99984E.9060305@brenbarn.net> References: <5A99984E.9060305@brenbarn.net> Message-ID: On Sat, Mar 3, 2018 at 5:30 AM, Brendan Barnwell wrote: > But if that's the case, maybe what we want is actually another thing > that's been discussed many times on this list, namely something like a > with-block that can define "super-local" variables that disappear at the end > of the block: > > with b**2 - 4*a*c as D: > x = [(-b + sqrt(D))/(2*a), (-b - sqrt(D))/(2*a)] > > This the same as my earlier version with "with", except the with > clause comes at the beginning. > > There's no need to go into extreme detail here on these proposals as > they're not really what's proposed by this PEP. Maybe they should be. There are two halves to this proposal: 1) A syntax for capturing an expression with a name binding 2) A scoping system for statement-local names. I'm already seeing some support for the first half without the second ("just make it a regular assignment"). Do we need two completely separate PEPs to handle these two proposals? They could link to each other saying "this could work well with that", but they have independent value. Statement-local names without a syntax for name binding would be useful in basically two contexts - the "with" syntax given here, and "except Exception as e:", which currently uses a bit of a weird hack of unbinding the existing name. But the 'with' statement would then have a messy dual form. It can be used for name bindings, and it can also be used for resource management. The easiest way to handle this is: def object.__enter__(self): return self def object.__exit__(self, *a): pass But I don't know that people want that. ChrisA From ethan at stoneleaf.us Fri Mar 2 13:44:52 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 02 Mar 2018 10:44:52 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: <5A999BA4.3010708@stoneleaf.us> On 03/02/2018 09:34 AM, David Mertz wrote: > So right now, I can do these: > > class bind(object): > def __init__(self, *args): > self.args = args > def __enter__(self): > return self.args[0] if len(self.args)==1 else self.args > def __exit__(self, *args): > pass > > >>> with bind(sqrt(2)) as _a: > ... print(_a) > 1.4142135623730951 > > >>> with bind(sqrt(2), log(2)) as (a, b): > ... print(a, b, a+b) > 1.4142135623730951 0.6931471805599453 2.1073607429330403 > > > This would cover 98% of the cases that I would want with the proposed statement-local name bindings. Cool! But what's the advantage over simple assignment? -- ~Ethan~ From mertz at gnosis.cx Fri Mar 2 14:40:24 2018 From: mertz at gnosis.cx (David Mertz) Date: Fri, 2 Mar 2018 11:40:24 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5A999BA4.3010708@stoneleaf.us> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A999BA4.3010708@stoneleaf.us> Message-ID: On Fri, Mar 2, 2018 at 10:44 AM, Ethan Furman wrote: > On 03/02/2018 09:34 AM, David Mertz wrote: > >> >>> with bind(sqrt(2), log(2)) as (a, b): >> ... print(a, b, a+b) >> 1.4142135623730951 0.6931471805599453 2.1073607429330403 >> >> This would cover 98% of the cases that I would want with the proposed >> statement-local name bindings. >> > > Cool! But what's the advantage over simple assignment? > The only difference from simple assignment is just visual and to be more self documenting. Basically, it just says (to me at least): "I intend to use these name within this block, but don't care about them elsewhere." It's sort of an informal scope without actual scoping rules. But of course, this is just existing Python, and anyone who wants to or doesn't is free to use or not use that style. In truth, I've thought about doing it from time to time, but never actually bothered in production code, just as a toy. -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From zachary.ware+pydev at gmail.com Fri Mar 2 14:45:00 2018 From: zachary.ware+pydev at gmail.com (Zachary Ware) Date: Fri, 2 Mar 2018 13:45:00 -0600 Subject: [Python-ideas] breakpoint(): print(*args, **kwargs) before entering pdb In-Reply-To: <14d2dc8a-f7fd-011e-5525-c295a9a7ee49@bordum.dk> References: <14d2dc8a-f7fd-011e-5525-c295a9a7ee49@bordum.dk> Message-ID: On Fri, Mar 2, 2018 at 12:00 PM, Carl Bordum Hansen wrote: > I played around with a newer Python build, but when using the new > `breakpoint` builtin, I missed my weapon of choice: dirty print-debugging. > > I suggest we combine forces and make the default `sys.breakpointhook` > forward *args and **kwargs to print before entering pdb. You can do this for yourself by adding the following to sitecustomize.py or similar: import sys def printingbreakpointhook(*args, **kwargs): print(args, kwargs) return sys.__breakpointhook__() sys.breakpointhook = printingbreakpointhook -- Zach From rhodri at kynesim.co.uk Fri Mar 2 14:21:41 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Fri, 2 Mar 2018 19:21:41 +0000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> On 02/03/18 18:27, Jelle Zijlstra wrote: > 2018-03-02 7:03 GMT-08:00 Robert Vanden Eynde : Guys, please don't email to me *and* the mailing list. Getting two copies of your deathless prose makes me less likely to pay attention to you, not more. -- Rhodri James *-* Kynesim Ltd From robertve92 at gmail.com Fri Mar 2 14:55:42 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Fri, 2 Mar 2018 20:55:42 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: @Rhodri, this is what Everybody does because you hit the "reply to all" button, but, we don't receive two copies on the mail, I don't know why (but that's a good thing). Le 2 mars 2018 20:48, "Rhodri James" a ?crit : On 02/03/18 18:27, Jelle Zijlstra wrote: > 2018-03-02 7:03 GMT-08:00 Robert Vanden Eynde : > Guys, please don't email to me *and* the mailing list. Getting two copies of your deathless prose makes me less likely to pay attention to you, not more. -- Rhodri James *-* Kynesim Ltd -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 2 15:01:58 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 07:01:58 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: On Sat, Mar 3, 2018 at 6:55 AM, Robert Vanden Eynde wrote: > @Rhodri, this is what Everybody does because you hit the "reply to all" > button, but, we don't receive two copies on the mail, I don't know why (but > that's a good thing). > > Le 2 mars 2018 20:48, "Rhodri James" a ?crit : > > On 02/03/18 18:27, Jelle Zijlstra wrote: >> >> 2018-03-02 7:03 GMT-08:00 Robert Vanden Eynde : > > > Guys, please don't email to me *and* the mailing list. Getting two copies > of your deathless prose makes me less likely to pay attention to you, not > more. > If you're using a poor-quality mail client, you may have to fix things manually. (Or get a better mail client - it's up to you.) If "Reply-All" sends to both the sender and the list, either look for a "Reply-List" button, or manually delete the sender's address and post only to the list. And please, don't top-post. Again, if your mail client encourages top posting, either override it, or get a better one. ChrisA From jcgoble3 at gmail.com Fri Mar 2 15:02:50 2018 From: jcgoble3 at gmail.com (Jonathan Goble) Date: Fri, 02 Mar 2018 20:02:50 +0000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: On Fri, Mar 2, 2018 at 2:56 PM Robert Vanden Eynde wrote: > @Rhodri, this is what Everybody does because you hit the "reply to all" > button, but, we don't receive two copies on the mail, I don't know why (but > that's a good thing). > > Le 2 mars 2018 20:48, "Rhodri James" a ?crit : > > On 02/03/18 18:27, Jelle Zijlstra wrote: > >> 2018-03-02 7:03 GMT-08:00 Robert Vanden Eynde : >> > > Guys, please don't email to me *and* the mailing list. Getting two copies > of your deathless prose makes me less likely to pay attention to you, not > more. > > > -- > Rhodri James *-* Kynesim Ltd > > *de-lurks* There is an option in your personal settings for the list to choose whether to receive duplicate copies of messages sent to both you and the list. It is usually wise to set that to "no". To check this option, go to https://mail.python.org/mailman/options/python-ideas, enter your email address and password (if you don't recall your password, just enter your email address and click "Remind" at the bottom of the page), and click "Log in". From there, you can change a variety of options, including the one about receiving duplicate copies of messages aimed at both you and the list. -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfahlgren at gmail.com Fri Mar 2 15:04:34 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Fri, 2 Mar 2018 12:04:34 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: On Fri, Mar 2, 2018 at 10:27 AM, Jelle Zijlstra wrote: > > I wonder if we could have a more limited change to the language that would > allow only the as/while use cases. Specifically, that means we could do: > > while do_something() as x: > print(x) > ?The "while" case is the only part of the PEP that has any traction with me. It doesn't add any keywords, scope can be identical to "with" and it cleans up a code pattern that is very common.? Every one of these comprehension examples has me scratching my head, thinking back a couple hundred (thousand? :) ) posts to Barry's quip, "Sometimes a for loop is just better" (or something along those lines). -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 2 15:20:34 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 07:20:34 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: On Sat, Mar 3, 2018 at 7:04 AM, Eric Fahlgren wrote: > > On Fri, Mar 2, 2018 at 10:27 AM, Jelle Zijlstra > wrote: >> >> >> I wonder if we could have a more limited change to the language that would >> allow only the as/while use cases. Specifically, that means we could do: >> >> while do_something() as x: >> print(x) > > > The "while" case is the only part of the PEP that has any traction with me. > It doesn't add any keywords, scope can be identical to "with" and it cleans > up a code pattern that is very common. How often do you have a loop like this where you actually want to capture the exact condition? I can think of two: regular expressions (match object or None), and socket read (returns empty string on EOF). This simplified form is ONLY of value in that sort of situation; as soon as you want to add a condition around it, this stops working (you can't say "while do_something() is not _sentinel as x:" because all you'll get is True). And if you are looking for one specific return value as your termination signal, you can write "for x in iter(do_something, None):". > Every one of these comprehension examples has me scratching my head, > thinking back a couple hundred (thousand? :) ) posts to Barry's quip, > "Sometimes a for loop is just better" (or something along those lines). True, but there's a code smell to an unrolled 'for' loop that could have been a comprehension had it not been for one trivial point of pedantry. So there are advantages and disadvantages to each. ChrisA From brenbarn at brenbarn.net Fri Mar 2 15:31:49 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Fri, 02 Mar 2018 12:31:49 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: <5A99B4B5.2090500@brenbarn.net> On 2018-03-02 12:20, Chris Angelico wrote: > On Sat, Mar 3, 2018 at 7:04 AM, Eric Fahlgren wrote: >> >> On Fri, Mar 2, 2018 at 10:27 AM, Jelle Zijlstra >> wrote: >>> >>> >>> I wonder if we could have a more limited change to the language that would >>> allow only the as/while use cases. Specifically, that means we could do: >>> >>> while do_something() as x: >>> print(x) >> >> >> The "while" case is the only part of the PEP that has any traction with me. >> It doesn't add any keywords, scope can be identical to "with" and it cleans >> up a code pattern that is very common. > > How often do you have a loop like this where you actually want to > capture the exact condition? I can think of two: regular expressions > (match object or None), and socket read (returns empty string on EOF). > This simplified form is ONLY of value in that sort of situation; as > soon as you want to add a condition around it, this stops working (you > can't say "while do_something() is not _sentinel as x:" because all > you'll get is True). And if you are looking for one specific return > value as your termination signal, you can write "for x in > iter(do_something, None):". But you could have "while (do_something() as x) is not _sentinel". Not sure how proponents and opponents would react to that. Limiting the SLNB to the beginning of block-level statements seems perverse in a way, but also might cut down on gratuitous overuse mixed into all kinds of weird positions in statements. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From robertve92 at gmail.com Fri Mar 2 15:47:52 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Fri, 2 Mar 2018 21:47:52 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: Le 2 mars 2018 21:02, "Chris Angelico" a ?crit : On Sat, Mar 3, 2018 at 6:55 AM, Robert Vanden Eynde wrote: > @Rhodri, this is what Everybody does because you hit the "reply to all" > button, but, we don't receive two copies on the mail, I don't know why (but > that's a good thing). > > Le 2 mars 2018 20:48, "Rhodri James" a ?crit : > > On 02/03/18 18:27, Jelle Zijlstra wrote: >> >> 2018-03-02 7:03 GMT-08:00 Robert Vanden Eynde : > > > Guys, please don't email to me *and* the mailing list. Getting two copies > of your deathless prose makes me less likely to pay attention to you, not > more. > If you're using a poor-quality mail client, you may have to fix things manually. (Or get a better mail client - it's up to you.) If "Reply-All" sends to both the sender and the list, either look for a "Reply-List" button, or manually delete the sender's address and post only to the list. And please, don't top-post. Again, if your mail client encourages top posting, either override it, or get a better one. ChrisA _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ @Chris @Rohdri (@Jonathan below) For morons like me who didn't know what "top-posting" was, I went on Wikipedia (https://en.m.wikipedia.org/wiki/Posting_style). I think newcomers to mailing list aren't supposed to know all those "de facto standard" (they never used mailing list before ? And I'm 25 years old, not 12), so please don't tell "our mail client is bad, get a new one", I'm using gmail on Android and gmail on Google.com, that's like, very standard. But if you have other mail clients to suggest me, to configure my user interface, feel free to help me, but they will always be people not knowing these good mailing practice (is using html mail a bad practice too ? It used to be). That's why moving to MM3 so that newcomers can use tool a bit more familiar like a forum and will not make the same "mistakes" (top posting, reply only to sender...) over and over again is a good thing. As someone said in their mail comparing web-based to mail, mailing needs configuration. @Jonathan, thanks for the settings, I didn't know about it, it solves this common problem ! -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Fri Mar 2 16:04:54 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 02 Mar 2018 13:04:54 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: <5A99BC76.4020008@stoneleaf.us> On 03/02/2018 12:47 PM, Robert Vanden Eynde wrote: > @Chris @Rohdri (@Jonathan below) > > For morons like me who didn't know what "top-posting" was, I went on Wikipedia > (https://en.m.wikipedia.org/wiki/Posting_style ). You looked it up and researched it, so definitely NOT a moron. ;) > I think newcomers to mailing list aren't supposed to know all those "de facto standard" (they never used mailing list > before ? And I'm 25 years old, not 12), so please don't tell "our mail client is bad, get a new one", I'm using gmail on > Android and gmail on Google.com, that's like, very standard. Standard, maybe. Good for basic email discussion, nevermind mailing lists? Nope (My humble opinion, of course.) > But if you have other mail clients to suggest me, to configure my user interface, feel free to help me, but they will > always be people not knowing these good mailing practice (is using html mail a bad practice too ? It used to be). The Python mailing lists are plain-text, so html-mail is not great. > That's why moving to MM3 so that newcomers can use tool a bit more familiar like a forum and will not make the same > "mistakes" (top posting, reply only to sender...) over and over again is a good thing. As someone said in their mail > comparing web-based to mail, mailing needs configuration. Most things worth using require configuration/customization (cars, kitchens, and definitely software). Thanks for persevering! -- ~Ethan~ From rosuav at gmail.com Fri Mar 2 16:07:43 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 08:07:43 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5A99B4B5.2090500@brenbarn.net> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99B4B5.2090500@brenbarn.net> Message-ID: On Sat, Mar 3, 2018 at 7:31 AM, Brendan Barnwell wrote: > On 2018-03-02 12:20, Chris Angelico wrote: >> >> On Sat, Mar 3, 2018 at 7:04 AM, Eric Fahlgren >> wrote: >>> >>> >>> On Fri, Mar 2, 2018 at 10:27 AM, Jelle Zijlstra >>> >>> wrote: >>>> >>>> >>>> >>>> I wonder if we could have a more limited change to the language that >>>> would >>>> allow only the as/while use cases. Specifically, that means we could do: >>>> >>>> while do_something() as x: >>>> print(x) >>> >>> >>> >>> The "while" case is the only part of the PEP that has any traction with >>> me. >>> It doesn't add any keywords, scope can be identical to "with" and it >>> cleans >>> up a code pattern that is very common. >> >> >> How often do you have a loop like this where you actually want to >> capture the exact condition? I can think of two: regular expressions >> (match object or None), and socket read (returns empty string on EOF). >> This simplified form is ONLY of value in that sort of situation; as >> soon as you want to add a condition around it, this stops working (you >> can't say "while do_something() is not _sentinel as x:" because all >> you'll get is True). And if you are looking for one specific return >> value as your termination signal, you can write "for x in >> iter(do_something, None):". > > > But you could have "while (do_something() as x) is not _sentinel". > Not sure how proponents and opponents would react to that. Limiting the > SLNB to the beginning of block-level statements seems perverse in a way, but > also might cut down on gratuitous overuse mixed into all kinds of weird > positions in statements. > If we're going to have something that's syntactically valid in an expression, I don't see a lot of value in requiring that it be inside a while or if header. That just leads to confusion and mess down the road. ChrisA From robertve92 at gmail.com Fri Mar 2 16:09:00 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Fri, 2 Mar 2018 22:09:00 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5A99BC76.4020008@stoneleaf.us> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> <5A99BC76.4020008@stoneleaf.us> Message-ID: Le 2 mars 2018 22:03, "Ethan Furman" a ?crit : On 03/02/2018 12:47 PM, Robert Vanden Eynde wrote: @Chris @Rohdri (@Jonathan below) > > For morons like me who didn't know what "top-posting" was, I went on > Wikipedia > (https://en.m.wikipedia.org/wiki/Posting_style < > https://en.m.wikipedia.org/wiki/Posting_style>). > You looked it up and researched it, so definitely NOT a moron. ;) I think newcomers to mailing list aren't supposed to know all those "de > facto standard" (they never used mailing list > before ? And I'm 25 years old, not 12), so please don't tell "our mail > client is bad, get a new one", I'm using gmail on > Android and gmail on Google.com, that's like, very standard. > Standard, maybe. Good for basic email discussion, nevermind mailing lists? Nope (My humble opinion, of course.) But if you have other mail clients to suggest me, to configure my user > interface, feel free to help me, but they will > always be people not knowing these good mailing practice (is using html > mail a bad practice too ? It used to be). > The Python mailing lists are plain-text, so html-mail is not great. That's why moving to MM3 so that newcomers can use tool a bit more familiar > like a forum and will not make the same > "mistakes" (top posting, reply only to sender...) over and over again is a > good thing. As someone said in their mail > comparing web-based to mail, mailing needs configuration. > Most things worth using require configuration/customization (cars, kitchens, and definitely software). Thanks for persevering! -- ~Ethan~ _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ Configuration is awesome but having good default helps you not having to tell everyone "configure that, the default is bad" (the "no send duplicate" settings maybe changed default or you misclicked on it). I'm just saying, be comprehensive to newcomer making mistakes, or give them a good man to read when they sign in :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 2 16:13:43 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 08:13:43 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: On Sat, Mar 3, 2018 at 7:47 AM, Robert Vanden Eynde wrote: >> And please, don't top-post. Again, if your mail client encourages top >> posting, either override it, or get a better one. > > @Chris @Rohdri (@Jonathan below) > > For morons like me who didn't know what "top-posting" was, I went on > Wikipedia (https://en.m.wikipedia.org/wiki/Posting_style). As Ethan says, not a moron. (Though the way he put it, he sounded like Wheatley... and if you don't know what I'm talking about, check out Portal and Portal 2, they're games well worth playing.) One of the beauties of text is that it's easy to research; if someone drops a term like "top-posting", you key that into your search engine, and voila, extra information. This is another advantage of separate standards and common conventions. Everything works with everything else, because it is simple and because nobody has to do everything themselves. How do you use git to manage your CPython source tree? It's exactly the same as using git for anything else, so you don't need Python-specific information, just git-specific information. How do you use a mailing list to discuss proposed language changes in Python? Again, it's generic stuff about mailing lists, so you don't need anything from Python. This is why we work with open standards. Email and mailing lists are an important part of this. ChrisA From ericfahlgren at gmail.com Fri Mar 2 16:18:42 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Fri, 2 Mar 2018 13:18:42 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: On Fri, Mar 2, 2018 at 12:20 PM, Chris Angelico wrote: > How often do you have a loop like this where you actually want to > capture the exact condition? I can think of two: regular expressions > (match object or None), and socket read (returns empty string on EOF). > This simplified form is ONLY of value in that sort of situation; as > soon as you want to add a condition around it, this stops working (you > can't say "while do_something() is not _sentinel as x:" because all > you'll get is True). And if you are looking for one specific return > value as your termination signal, you can write "for x in > iter(do_something, None):". > ?For me, it's all the time. Our geometry modeling database is hierarchical, so you see things like this all over kernel, often with a lot more code than just that one line calculating the cumulative scale factor: >>> scale = self.scale >>> parent = self.parent >>> while parent: >>> scale? *= parent.scale >>> parent = parent.parent # The DRY part that I don't like... which would turn into >>> scale = self.scale >>> parent = self >>> while parent.parent as parent: >>> scale? *= parent.scale -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Fri Mar 2 16:21:03 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Fri, 2 Mar 2018 22:21:03 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: Le 2 mars 2018 22:13, "Chris Angelico" a ?crit : On Sat, Mar 3, 2018 at 7:47 AM, Robert Vanden Eynde wrote: >> And please, don't top-post. Again, if your mail client encourages top >> posting, either override it, or get a better one. > > @Chris @Rohdri (@Jonathan below) > > For morons like me who didn't know what "top-posting" was, I went on > Wikipedia (https://en.m.wikipedia.org/wiki/Posting_style). As Ethan says, not a moron. (Though the way he put it, he sounded like Wheatley... and if you don't know what I'm talking about, check out Portal and Portal 2, they're games well worth playing.) One of the beauties of text is that it's easy to research; if someone drops a term like "top-posting", you key that into your search engine, and voila, extra information. This is another advantage of separate standards and common conventions. Everything works with everything else, because it is simple and because nobody has to do everything themselves. How do you use git to manage your CPython source tree? It's exactly the same as using git for anything else, so you don't need Python-specific information, just git-specific information. How do you use a mailing list to discuss proposed language changes in Python? Again, it's generic stuff about mailing lists, so you don't need anything from Python. This is why we work with open standards. Email and mailing lists are an important part of this. ChrisA _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ As said on the Wikipedia page I shared, """ For a long time the traditional style was to post the answer below as much of the quoted original as was necessary to understand the reply (bottom or inline). Many years later, when email became widespread in business communication, it became a widespread practice to reply above the entire original and leave it (supposedly untouched) below the reply. While each online community differs on which styles are appropriate or acceptable, within some communities the use of the ?wrong? method risks being seen as a breach of netiquette , and can provoke vehement response from community regulars. """ De-facto I wouldn't know which convention (of course now I clearly prefer "interleaved posting" for obvious reason... Now that I know how to do it) so a Manual or HowTo when signing in would be an idea :) Of course, I clearly prefer interleaved posting for obvious reasons... now that I know how to produce it using my daily mail client. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Fri Mar 2 16:24:08 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Fri, 2 Mar 2018 22:24:08 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: Le 2 mars 2018 22:21, "Robert Vanden Eynde" a ?crit : Le 2 mars 2018 22:13, "Chris Angelico" a ?crit : On Sat, Mar 3, 2018 at 7:47 AM, Robert Vanden Eynde wrote: >> And please, don't top-post. Again, if your mail client encourages top >> posting, either override it, or get a better one. > > @Chris @Rohdri (@Jonathan below) > > For morons like me who didn't know what "top-posting" was, I went on > Wikipedia (https://en.m.wikipedia.org/wiki/Posting_style). As Ethan says, not a moron. (Though the way he put it, he sounded like Wheatley... and if you don't know what I'm talking about, check out Portal and Portal 2, they're games well worth playing.) One of the beauties of text is that it's easy to research; if someone drops a term like "top-posting", you key that into your search engine, and voila, extra information. This is another advantage of separate standards and common conventions. Everything works with everything else, because it is simple and because nobody has to do everything themselves. How do you use git to manage your CPython source tree? It's exactly the same as using git for anything else, so you don't need Python-specific information, just git-specific information. How do you use a mailing list to discuss proposed language changes in Python? Again, it's generic stuff about mailing lists, so you don't need anything from Python. This is why we work with open standards. Email and mailing lists are an important part of this. ChrisA _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ As said on the Wikipedia page I shared, """ For a long time the traditional style was to post the answer below as much of the quoted original as was necessary to understand the reply (bottom or inline). Many years later, when email became widespread in business communication, it became a widespread practice to reply above the entire original and leave it (supposedly untouched) below the reply. While each online community differs on which styles are appropriate or acceptable, within some communities the use of the ?wrong? method risks being seen as a breach of netiquette , and can provoke vehement response from community regulars. """ De-facto I wouldn't know which convention (of course now I clearly prefer "interleaved posting" for obvious reason... Now that I know how to do it) so a Manual or HowTo when signing in would be an idea :) Of course, I clearly prefer interleaved posting for obvious reasons... now that I know how to produce it using my daily mail client. Damn it, I forgot to erase the first paragraph, I'm used to forum app where you can quickly edit a sent message when nobody already read it. By the way, yes, text is wonderful, searchable and archivable, that's one of the reasons to love programming and markup language :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Fri Mar 2 20:36:39 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2018 14:36:39 +1300 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: <5A99FC27.3000809@canterbury.ac.nz> PEP 572 as it stands seems to have many problems: * Asymmetry between the first and subsequent uses of the bound value * Embedded as-clauses are subtle and hard to spot * Ever more convoluted semantics are being invented to address percieved pitfalls Alternative proposal -------------------- Getting back to the original motivating use case of intermediate values in comprehensions, to my mind there is really only one clear and obvious way to write such things: [(f(y), g(y)) for x in things where y = h(x)] Possible objections to this: * Requires a new keyword, which may break existing code. - Yes, but "where" is an unlikely choice of name, being neither a noun, verb or adjective, so it probably wouldn't break very *much* code. In return, we get something that resonates with language mathematicians have been using for centuries, and a word that can be usefully googled. * Out-of-order evaluation - Comprehensions and if-expressions already have that. Extension to other use cases ---------------------------- There are some other situations in which it could be useful to have a similar construct. 1. Name bindings local to an expression: roots = ([(-b-r)/(2*a), (-b+r)/(2*a)] where r = sqrt(b*b-4*a*c)) 2. Names bound as part of an expression and also available in a following suite: if m where m = pattern.match(text): do_something_with(m) There's a problem, though -- if "where" in an expression creates bindings local to that expression, they're not going to be visible in the rest of a comprehension or compound statement. So I'm thinking that there would actually be three distinct usages of "where": A. As a clause in a comprehension, where it creates bindings local to the comprehension and is treated on the same footing syntactically as "for" and "if" clauses. B. In an expression, surrounded by parentheses for disambiguation. Bindings are visible only within the parentheses. C. Following the expression of an "if" or "while" statement. Bindings are visible within the preceding expression and the following suite. Note that case B avoids the issue of whether expression-local bindings affect the LHS of an assignment, because in a[x] = (x * 2 where x = 17) the "x" bound by the where-clause is clearly restricted to the part in parentheses. This would be a syntax error: a[x] = x * 2 where x = 17 # illegal -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 2 20:48:30 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2018 14:48:30 +1300 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> Message-ID: <5A99FEEE.4000109@canterbury.ac.nz> Chris Angelico wrote: > It would NOT work for anything where the bool() of > the desired object doesn't exactly match the loop's condition. while condition(x) where x = something(): ... -- Greg From greg.ewing at canterbury.ac.nz Fri Mar 2 19:09:04 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sat, 03 Mar 2018 13:09:04 +1300 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: Message-ID: <5A99E7A0.50105@canterbury.ac.nz> Nick Coghlan wrote: > I don't think it should be possible to close over > statement locals. Why not? -- Greg From robertve92 at gmail.com Fri Mar 2 22:09:59 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Sat, 3 Mar 2018 04:09:59 +0100 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <5A99FC27.3000809@canterbury.ac.nz> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: The syntax you propose is already in the Alternate syntax and there is an implementation at https://github.com/thektulu/cpython/tree/where-expr Already discussed, but no conclusion, for me I see two different proposals, the "[y for x in range(5) with y = x+1]" in comprehensions list, and the case in any expression "print (y with y = x+1)", as discussed the second case corresponds to a simple assignment so is less useful, but if the scope is local to the expression, that'd be useful : print(y with y = x+1); print(y) # NameError But I like it :) Le 3 mars 2018 02:36, "Greg Ewing" a ?crit : > PEP 572 as it stands seems to have many problems: > > * Asymmetry between the first and subsequent > uses of the bound value > * Embedded as-clauses are subtle and hard to spot > * Ever more convoluted semantics are being invented > to address percieved pitfalls > > Alternative proposal > -------------------- > > Getting back to the original motivating use case of > intermediate values in comprehensions, to my mind there > is really only one clear and obvious way to write such > things: > > [(f(y), g(y)) for x in things where y = h(x)] > > Possible objections to this: > > * Requires a new keyword, which may break existing code. > > - Yes, but "where" is an unlikely choice of name, being > neither a noun, verb or adjective, so it probably wouldn't > break very *much* code. In return, we get something that > resonates with language mathematicians have been using > for centuries, and a word that can be usefully googled. > > * Out-of-order evaluation > > - Comprehensions and if-expressions already have that. > > Extension to other use cases > ---------------------------- > > There are some other situations in which it could be > useful to have a similar construct. > > 1. Name bindings local to an expression: > > roots = ([(-b-r)/(2*a), (-b+r)/(2*a)] where r = sqrt(b*b-4*a*c)) > > 2. Names bound as part of an expression and also available > in a following suite: > > if m where m = pattern.match(text): > do_something_with(m) > > There's a problem, though -- if "where" in an expression > creates bindings local to that expression, they're not > going to be visible in the rest of a comprehension or > compound statement. > > So I'm thinking that there would actually be three distinct > usages of "where": > > A. As a clause in a comprehension, where it creates bindings > local to the comprehension and is treated on the same footing > syntactically as "for" and "if" clauses. > > B. In an expression, surrounded by parentheses for > disambiguation. Bindings are visible only within the > parentheses. > > C. Following the expression of an "if" or "while" statement. > Bindings are visible within the preceding expression and the > following suite. > > Note that case B avoids the issue of whether expression-local > bindings affect the LHS of an assignment, because in > > a[x] = (x * 2 where x = 17) > > the "x" bound by the where-clause is clearly restricted to > the part in parentheses. This would be a syntax error: > > a[x] = x * 2 where x = 17 # illegal > > -- > Greg > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 2 22:30:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 14:30:24 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5A99FEEE.4000109@canterbury.ac.nz> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FEEE.4000109@canterbury.ac.nz> Message-ID: On Sat, Mar 3, 2018 at 12:48 PM, Greg Ewing wrote: > Chris Angelico wrote: >> >> It would NOT work for anything where the bool() of >> the desired object doesn't exactly match the loop's condition. > > > while condition(x) where x = something(): > ... > And we're back to needing a new keyword. Though this is reasonably plausible, so I could add it to the PEP (as another rejected alternative) if there's support for it. ChrisA From ncoghlan at gmail.com Sat Mar 3 02:04:08 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Mar 2018 17:04:08 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: <5A99E7A0.50105@canterbury.ac.nz> References: <5A99E7A0.50105@canterbury.ac.nz> Message-ID: On 3 March 2018 at 10:09, Greg Ewing wrote: > Nick Coghlan wrote: > >> I don't think it should be possible to close over statement locals. >> > > Why not? I gave a more detailed answer to that in https://mail.python.org/pipermail/python-ideas/2018-March/049138.html, but the short version is: * prohibiting closing over them opens up a lot of opportunities to simplify the implementation * prohibiting closing over them increase their semantic distance from regular function locals, making it easier for people to answer the question "Should I use a function local or a statement local?" in the future Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 3 02:11:33 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Mar 2018 17:11:33 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: <5A998F17.4000606@stoneleaf.us> References: <5A98F19C.7010403@stoneleaf.us> <5A998F17.4000606@stoneleaf.us> Message-ID: On 3 March 2018 at 03:51, Ethan Furman wrote: > On 03/02/2018 02:47 AM, Nick Coghlan wrote: > >> On 2 March 2018 at 16:39, Ethan Furman wrote: >> >>> On 03/01/2018 09:08 PM, Nick Coghlan wrote: >>> >> > Adding statement local variables into that mix *without* some form of >>>> syntactic marker would mean taking an already >>>> complicated system, and making it even harder to reason about correctly >>>> (especially if statement locals interact >>>> with nested scopes differently from the way other locals in the same >>>> scope do). >>>> >>> >>> Seems like it would far easier and (IMHO) more useful to scale the >>> proposal back from a statement scope to simple >>> expression assignment, and the variable is whatever scope it would have >>> been if assigned to outside the expression >>> (default being local, but non-local or global if already declared as >>> such). >>> >> >> Because that would put us back in the exact same problematic situation we >> had when "[x*x for x in sequence]" leaked the >> iteration variable (only worse): no function locals would be safe, since >> arbitrary expressions could clobber them, not >> just name binding operations (assignment, import statements, for loops, >> with statements, exception handlers, class and >> function definitions). >> > > Ah, right. Since the PEP primarily covers comprehensions, but then went > on to discuss multi-line statements, I had forgotten the comprehension > part. The answer is easy: assignment expressions in comprehensions stay > inside comprehensions, just like other inner comprehension variables > already do (function sub-scope, after all). Thank you for pointing that > out. > That wasn't the point I was try to make: my attempted point was that I see allowing an expression like "print((f() as x), x^2, x^3)" to overwrite the regular function local "x" as being just as unacceptable as "data = [x^2 for x in sequence]" overwriting it, and we already decided that the latter was sufficiently undesirable that we were willing to break backwards compatibility in order to change it. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 3 02:40:03 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Mar 2018 17:40:03 +1000 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <5A99FC27.3000809@canterbury.ac.nz> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: On 3 March 2018 at 11:36, Greg Ewing wrote: > PEP 572 as it stands seems to have many problems: > > * Asymmetry between the first and subsequent > uses of the bound value > * Embedded as-clauses are subtle and hard to spot > * Ever more convoluted semantics are being invented > to address percieved pitfalls > > Alternative proposal > -------------------- > > Getting back to the original motivating use case of > intermediate values in comprehensions, to my mind there > is really only one clear and obvious way to write such > things: > > [(f(y), g(y)) for x in things where y = h(x)] > I know Guido is on record as not wanting to allow both "for name in sequence" and "for name = expr" due to that being a very subtle distinction between iteration and simple assignment (especially given that Julia uses them as alternate spellings for the same thing), but I'm wondering if it may be worth considering such a distinction in *with statements*, such that we allowed "with name = expr" in addition to "with cm as name" (where "name = expr" is just an ordinary assignment statement that doesn't trigger the context management protocol). The related enhancement to comprehensions would then be to allow with clauses to be interleaved between the for loops and the if statements, such that you could write things like: pairs = [(f(y), g(y)) for x in things with y = h(x)] contents = [f.read() for fname in filenames with open(fname) as f] while still preserving the property where comprehensions can be correctly interpreted just by converting each of the clauses to the corresponding statement form. Even without the "with name = expr" change, allowing with clauses in comprehensions would let you do (by way of a suitably defined "bind" CM): pairs = [(f(y), g(y)) for x in things with bind(h(x)) as y] Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 3 02:45:16 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 3 Mar 2018 17:45:16 +1000 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <5A99FC27.3000809@canterbury.ac.nz> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: On 3 March 2018 at 11:36, Greg Ewing wrote: > 1. Name bindings local to an expression: > > roots = ([(-b-r)/(2*a), (-b+r)/(2*a)] where r = sqrt(b*b-4*a*c)) > > B. In an expression, surrounded by parentheses for > disambiguation. Bindings are visible only within the > parentheses. > I'll note that a significant benefit of this approach over the PEP 572 approach is that it would be amenable to a comprehension style scope-management solution: these expressions could create an implicitly nested function and immediately call it, just as 3.x comprehensions do. Adopting such an approach would *dramatically* lower the impact that hiding the bindings from the surrounding scope would have on the overall name resolution semantics. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Mar 3 03:23:44 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 03 Mar 2018 00:23:44 -0800 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FEEE.4000109@canterbury.ac.nz> Message-ID: <5A9A5B90.6080001@stoneleaf.us> On 03/02/2018 07:30 PM, Chris Angelico wrote: > On Sat, Mar 3, 2018 at 12:48 PM, Greg Ewing wrote: >> Chris Angelico wrote: >>> >>> It would NOT work for anything where the bool() of >>> the desired object doesn't exactly match the loop's condition. >> >> >> while condition(x) where x = something(): >> ... >> > > And we're back to needing a new keyword. Though this is reasonably > plausible, so I could add it to the PEP (as another rejected > alternative) if there's support for it. So what happens to rejected alternatives when the PEP itself is rejected? ;) -- ~Ethan~ From rosuav at gmail.com Sat Mar 3 04:01:30 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 3 Mar 2018 20:01:30 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5A9A5B90.6080001@stoneleaf.us> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FEEE.4000109@canterbury.ac.nz> <5A9A5B90.6080001@stoneleaf.us> Message-ID: On Sat, Mar 3, 2018 at 7:23 PM, Ethan Furman wrote: > On 03/02/2018 07:30 PM, Chris Angelico wrote: >> >> On Sat, Mar 3, 2018 at 12:48 PM, Greg Ewing wrote: >>> >>> Chris Angelico wrote: >>>> >>>> >>>> It would NOT work for anything where the bool() of >>>> the desired object doesn't exactly match the loop's condition. >>> >>> >>> >>> while condition(x) where x = something(): >>> ... >>> >> >> And we're back to needing a new keyword. Though this is reasonably >> plausible, so I could add it to the PEP (as another rejected >> alternative) if there's support for it. > > > So what happens to rejected alternatives when the PEP itself is rejected? > ;) > Algebraically, they must be accepted. Right? ChrisA From p.f.moore at gmail.com Sat Mar 3 05:32:28 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 3 Mar 2018 10:32:28 +0000 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: On 3 March 2018 at 07:45, Nick Coghlan wrote: > On 3 March 2018 at 11:36, Greg Ewing wrote: >> >> 1. Name bindings local to an expression: >> >> roots = ([(-b-r)/(2*a), (-b+r)/(2*a)] where r = sqrt(b*b-4*a*c)) >> >> B. In an expression, surrounded by parentheses for >> disambiguation. Bindings are visible only within the >> parentheses. > > > I'll note that a significant benefit of this approach over the PEP 572 > approach is that it would be amenable to a comprehension style > scope-management solution: these expressions could create an implicitly > nested function and immediately call it, just as 3.x comprehensions do. > Adopting such an approach would *dramatically* lower the impact that hiding > the bindings from the surrounding scope would have on the overall name > resolution semantics. While I'm *still* not sure any of this provides enough benefit to be worth doing, I will say that this proposal feels far less complicated than PEP 572. I don't particularly like the extension to if and while statements (and it's been mentioned that there are remarkably few cases where the value you want to capture is the actual condition rather than a value the condition tests) but otherwise I'm OK with it (call me -0 on the proposal without if/while, and -0.5 on the if/while part). Paul From phd at phdru.name Sat Mar 3 06:09:22 2018 From: phd at phdru.name (Oleg Broytman) Date: Sat, 3 Mar 2018 12:09:22 +0100 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <5A99FC27.3000809@canterbury.ac.nz> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: <20180303110922.25t2wacqc4r5niyt@phdru.name> On Sat, Mar 03, 2018 at 02:36:39PM +1300, Greg Ewing wrote: > [(f(y), g(y)) for x in things where y = h(x)] > > Possible objections to this: > > * Requires a new keyword, which may break existing code. > > - Yes, but "where" is an unlikely choice of name, being > neither a noun, verb or adjective, so it probably wouldn't > break very *much* code. ``where`` is a very popular name in code related to SQL. Take for example ORMs: https://github.com/sqlobject/sqlobject/search?l=Python&q=where&type=&utf8=%E2%9C%93 https://github.com/zzzeek/sqlalchemy/search?l=Python&q=where&type=&utf8=%E2%9C%93 > -- > Greg Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From fakedme+py at gmail.com Sat Mar 3 06:23:26 2018 From: fakedme+py at gmail.com (Soni L.) Date: Sat, 3 Mar 2018 08:23:26 -0300 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <20180303110922.25t2wacqc4r5niyt@phdru.name> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <20180303110922.25t2wacqc4r5niyt@phdru.name> Message-ID: <56fd1a38-4dfb-c830-f79b-bbfa564cdf93@gmail.com> On 2018-03-03 08:09 AM, Oleg Broytman wrote: > On Sat, Mar 03, 2018 at 02:36:39PM +1300, Greg Ewing wrote: >> [(f(y), g(y)) for x in things where y = h(x)] >> >> Possible objections to this: >> >> * Requires a new keyword, which may break existing code. >> >> - Yes, but "where" is an unlikely choice of name, being >> neither a noun, verb or adjective, so it probably wouldn't >> break very *much* code. > ``where`` is a very popular name in code related to SQL. Take for > example ORMs: > > https://github.com/sqlobject/sqlobject/search?l=Python&q=where&type=&utf8=%E2%9C%93 > > https://github.com/zzzeek/sqlalchemy/search?l=Python&q=where&type=&utf8=%E2%9C%93 [(lambda y: (f(y), g(y)))(h(x)) for x in things] ? > >> -- >> Greg > Oleg. From robertve92 at gmail.com Sat Mar 3 10:59:40 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Sat, 3 Mar 2018 16:59:40 +0100 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: Le 3 mars 2018 08:45, "Nick Coghlan" a ?crit : On 3 March 2018 at 11:36, Greg Ewing wrote: > 1. Name bindings local to an expression: > > roots = ([(-b-r)/(2*a), (-b+r)/(2*a)] where r = sqrt(b*b-4*a*c)) > > B. In an expression, surrounded by parentheses for > disambiguation. Bindings are visible only within the > parentheses. > I'll note that a significant benefit of this approach over the PEP 572 approach is that it would be amenable to a comprehension style scope-management solution: these expressions could create an implicitly nested function and immediately call it, just as 3.x comprehensions do. Adopting such an approach would *dramatically* lower the impact that hiding the bindings from the surrounding scope would have on the overall name resolution semantics. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ As I said, allowing this syntax (whether with a new keyword like 'where' or reusing something like 'with' to write print(y with y = x+1)) in any expression (not only after a "for" in a comprehension list) is useful only if a local scope is created (otherwise, it would be the same as simple assignement but in reverse ordre). One could see : print(y with y = x+1) As a shortcut for : print(next(y for y in [ x+1 ])) The same as this : [y for x in range(5) with y = x+1] being a shortcut for : [y for x in range(5) for y in [ x+1 ]] As said, allowing both use cases would lead to two different ways to write : [y for x in range(5) with y = x+1] vs [y with y = x+1 for x in range(5)] But that is not really an issue, it's logically different (is the y a conclusion of the iteration or a hidden lambda in the expression?). -------------- next part -------------- An HTML attachment was scrubbed... URL: From nathan12343 at gmail.com Sat Mar 3 11:12:06 2018 From: nathan12343 at gmail.com (Nathan Goldbaum) Date: Sat, 03 Mar 2018 16:12:06 +0000 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <20180303110922.25t2wacqc4r5niyt@phdru.name> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <20180303110922.25t2wacqc4r5niyt@phdru.name> Message-ID: On Sat, Mar 3, 2018 at 5:12 AM Oleg Broytman wrote: > On Sat, Mar 03, 2018 at 02:36:39PM +1300, Greg Ewing < > greg.ewing at canterbury.ac.nz> wrote: > > [(f(y), g(y)) for x in things where y = h(x)] > > > > Possible objections to this: > > > > * Requires a new keyword, which may break existing code. > > > > - Yes, but "where" is an unlikely choice of name, being > > neither a noun, verb or adjective, so it probably wouldn't > > break very *much* code. > > ``where`` is a very popular name in code related to SQL. Take for > example ORMs: > > > https://github.com/sqlobject/sqlobject/search?l=Python&q=where&type=&utf8=%E2%9C%93 > > > https://github.com/zzzeek/sqlalchemy/search?l=Python&q=where&type=&utf8=%E2%9C%93 > Where is also very common in numpy. So if someone did: from numpy import * data = where(condition) They might have issues > > -- > > Greg > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From jpic at yourlabs.org Sat Mar 3 12:12:06 2018 From: jpic at yourlabs.org (Jamesie Pic) Date: Sat, 3 Mar 2018 18:12:06 +0100 Subject: [Python-ideas] Class autoload Message-ID: Hello everybody, I thought perhaps we could allow the usage of a "new" keyword to instanciate an object, ie: obj = new yourmodule.YourClass() In this case, it would behave the same as from yourmodule import YourClass; obj = YourClass(), except that it wouldn't need to be imported. This would also eliminate the need to manage an import list at the beginning of a script in most case. I'm really not proud of this idea but PHP has had autoload for years and when i open scripts with hundred lines of imports it makes me think Python could do something about this. Thanks in advance for your feedback Best regards -- ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From eric at trueblade.com Sat Mar 3 12:38:20 2018 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 3 Mar 2018 12:38:20 -0500 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: <93a472ca-21eb-0b0a-7554-c1fca8416247@trueblade.com> On 3/3/2018 12:12 PM, Jamesie Pic wrote: > > Hello everybody, > > I thought perhaps we could allow the usage of a "new" keyword to > instanciate an object, ie: > > ?? obj = new yourmodule.YourClass() I'd just do: import yourmodule obj = yourmodule.YourClass() Or as one line, if that's your thing: import yourmodule; obj = yourmodule.YourClass() Imports don't need to be at the top of the file. If you want to delay loading modules, that's fine. It's pretty cheap for an already loaded module. In my opinion it's not worth a new keyword and something else to learn. Eric > > In this case, it would behave the same as from yourmodule import > YourClass; obj = YourClass(), except that it wouldn't need to be > imported. This would also eliminate the need to manage an import list at > the beginning of a script in most case. > > I'm really not proud of this idea but PHP has had autoload for years and > when i open scripts with hundred lines of imports it makes me think > Python could do something about this. > > Thanks in advance for your feedback > > Best regards > > -- > ? > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From rosuav at gmail.com Sat Mar 3 12:42:14 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 4 Mar 2018 04:42:14 +1100 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: On Sun, Mar 4, 2018 at 4:12 AM, Jamesie Pic wrote: > > Hello everybody, > > I thought perhaps we could allow the usage of a "new" keyword to instanciate > an object, ie: > > obj = new yourmodule.YourClass() > > In this case, it would behave the same as from yourmodule import YourClass; > obj = YourClass(), except that it wouldn't need to be imported. This would > also eliminate the need to manage an import list at the beginning of a > script in most case. > This actually has nothing to do with classes. You can currently write this: import yourmodule obj = yourmodule.YourClass() without any sort of 'new' keyword. So presumably what you're asking for is a way to avoid typing the 'import' statement. That's something that's come up every once in a while. Usually for the benefit of throwaway scripts and the interactive interpreter, because in serious applications, a single 'import' line is a small price to pay for the clarity. You may want to dig through the archives to find the arguments for and against this sort of automated import. > I'm really not proud of this idea but PHP has had autoload for years and > when i open scripts with hundred lines of imports it makes me think Python > could do something about this. A hundred lines of imports? Surely an exaggeration... or possibly you have a whole lot of "from module import name" lines that could become a single "import module" line. Also, "PHP does this" is a terrible justification for a feature... :) ChrisA From rob.cliffe at btinternet.com Sat Mar 3 13:44:20 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 3 Mar 2018 18:44:20 +0000 Subject: [Python-ideas] Class autoload In-Reply-To: <93a472ca-21eb-0b0a-7554-c1fca8416247@trueblade.com> References: <93a472ca-21eb-0b0a-7554-c1fca8416247@trueblade.com> Message-ID: On 03/03/2018 17:38, Eric V. Smith wrote: > > I'd just do: > ??? import yourmodule > ??? obj = yourmodule.YourClass() > > Or as one line, if that's your thing: > ??? import yourmodule; obj = yourmodule.YourClass() Which is ??? More transparent: it's evident what the imported module is used for. ??? More maintainable: you only need to add or remove one line or two adjacent lines per instance, instead of two widely separated lines. (as long as you're happy to defer the imports, of course). Regards Rob Cliffe From jpic at yourlabs.org Sat Mar 3 13:46:21 2018 From: jpic at yourlabs.org (Jamesie Pic) Date: Sat, 3 Mar 2018 19:46:21 +0100 Subject: [Python-ideas] Class autoload In-Reply-To: References: <93a472ca-21eb-0b0a-7554-c1fca8416247@trueblade.com> Message-ID: While i understand it would be harder to make it memory efficient, but this is python not go, and also this sort feature could be easily optional, also, it might even help against circular import issues, whoever hasn't imported a module from inside a function in their life may throw the first rock at my face (kidding) Yes i know it's sad php has this feature and python does not and again i'm not proud to say this but it's true. -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Mar 3 14:33:36 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 3 Mar 2018 14:33:36 -0500 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: On 3/3/2018 12:12 PM, Jamesie Pic wrote: > > Hello everybody, > > I thought perhaps we could allow the usage of a "new" keyword to > instanciate an object, ie: > > ?? obj = new yourmodule.YourClass() > > In this case, it would behave the same as from yourmodule import > YourClass; obj = YourClass(), except that it wouldn't need to be > imported. This would also eliminate the need to manage an import list at > the beginning of a script in most case. If you really want this (untested): def autoload(mod, cls, *args, **kwargs): from mod import cls return cls(*args, **kwargs) obj = autoload(yourmodule, YourClass) -- Terry Jan Reedy From ethan at stoneleaf.us Sat Mar 3 14:59:35 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 03 Mar 2018 11:59:35 -0800 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings In-Reply-To: References: <5A98F19C.7010403@stoneleaf.us> <5A998F17.4000606@stoneleaf.us> Message-ID: <5A9AFEA7.4000307@stoneleaf.us> On 03/02/2018 11:11 PM, Nick Coghlan wrote: > On 3 March 2018 at 03:51, Ethan Furman wrote: >> Ah, right. Since the PEP primarily covers comprehensions, but then went on to discuss multi-line statements, I had >> forgotten the comprehension part. The answer is easy: assignment expressions in comprehensions stay inside >> comprehensions, just like other inner comprehension variables already do (function sub-scope, after all). Thank you >> for pointing that out. > > That wasn't the point I was try to make: my attempted point was that I see allowing an expression like "print((f() as > x), x^2, x^3)" to overwrite the regular function local "x" as being just as unacceptable as "data = [x^2 for x in > sequence]" overwriting it, and we already decided that the latter was sufficiently undesirable that we were willing to > break backwards compatibility in order to change it. I think I explained myself poorly. I'm agreeing with you, and pointing out that the (var as expr) syntax /inside/ a comprehension would stay inside a comprehension, i.e. not leak out to locals(), just like your "x" above. -- ~Ethan~ From tjreedy at udel.edu Sat Mar 3 16:24:42 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 3 Mar 2018 16:24:42 -0500 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: On 3/3/2018 2:33 PM, Terry Reedy wrote: > On 3/3/2018 12:12 PM, Jamesie Pic wrote: >> >> Hello everybody, >> >> I thought perhaps we could allow the usage of a "new" keyword to >> instanciate an object, ie: >> >> ??? obj = new yourmodule.YourClass() >> >> In this case, it would behave the same as from yourmodule import >> YourClass; obj = YourClass(), except that it wouldn't need to be >> imported. This would also eliminate the need to manage an import list >> at the beginning of a script in most case. > > If you really want this (untested): > > def autoload(mod, cls, *args, **kwargs): > ??? from mod import cls The make the above work with variables: import importlib # put at top of file mod = importlib.import_module(mod) cls = mod.getattr(cls) > ??? return cls(*args, **kwargs) > > obj = autoload(yourmodule, YourClass) > -- Terry Jan Reedy From greg.ewing at canterbury.ac.nz Sat Mar 3 18:25:26 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Mar 2018 12:25:26 +1300 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: <5A9B2EE6.1090409@canterbury.ac.nz> Robert Vanden Eynde wrote: > One could see : > > print(y with y = x+1) > > As a shortcut for : > > print(next(y for y in [ x+1 ])) Or more straightforwardly, print((lambda y: y)(x + 1)) This is how the semantics of let-type constructs is often defined in lambda-calculus-inspired languages such as Scheme and Haskell, and it suggests a viable implementation strategy. -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 3 18:58:14 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Mar 2018 12:58:14 +1300 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <20180303110922.25t2wacqc4r5niyt@phdru.name> Message-ID: <5A9B3696.4010000@canterbury.ac.nz> Nathan Goldbaum wrote: > Where is also very common in numpy. So if someone did: > > from numpy import * > > data = where(condition) > > They might have issues Bummer, I didn't know numpy used it. That puts rather a big dampener on the idea of making it a keyword. :-( Remaining options include: * Make it a keyword only in certain contexts. That's been done before, but only as a temporary measure. Making it a permanent feature seems a bit hackish, and could cause problems for syntax highlighters. * Pick another word that isn't used often. My next choice would be "given", with maybe "letting" as a distant third. (Not just "let", because it doesn't read right after the expression.) * Re-use another keyword. Here "with" seems to be the best choice, but that would entail giving it two wildly different meanings. Guido probably isn't going to like that, since he has already expressed disapproval of doing it with "for". Some people thought Wirth was mad when he made all the keywords in Modula upper-case, but maybe he knew a thing or two. Going back further, Algol required all keywords to be marked specially in some way (that was left up to the implementation). If I ever design another language, I think I'm going to require the source to be HTML, and insist that all keywords be bold. -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 3 19:13:50 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Mar 2018 13:13:50 +1300 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: <5A9B3A3E.1090000@canterbury.ac.nz> Jamesie Pic wrote: > > obj = new yourmodule.YourClass() > > This would also eliminate the need to manage an import list at > the beginning of a script in most case. I like the fact that I can usually tell what modules a module depends on by looking at the top for import statements. If people were encouraged to scatter implicit imports throughout the code, this would be much harder. Also, this seems a rather subtle and non-obvious distinction to make between two constructs that look like they should mean exactly the same thing. -- Greg From steve at pearwood.info Sat Mar 3 19:16:52 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Mar 2018 11:16:52 +1100 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: <20180304001652.GD18868@ando.pearwood.info> On Sat, Mar 03, 2018 at 02:33:36PM -0500, Terry Reedy wrote: > def autoload(mod, cls, *args, **kwargs): > from mod import cls > return cls(*args, **kwargs) > > obj = autoload(yourmodule, YourClass) That won't work unless yourmodule and YourClass have already been imported, since you'll get NameError. You need to pass the names as strings, something like: # also untested def autoload(mod, cls, *args, **kwargs): module = __import__(mod) # or use importlib K = getattr(module, cls) return K(*args, **kwargs) obj = autoload('yourmodule', 'YourClass') -- Steve From njs at pobox.com Sat Mar 3 19:23:31 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 3 Mar 2018 16:23:31 -0800 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: On Sat, Mar 3, 2018 at 9:12 AM, Jamesie Pic wrote: > > Hello everybody, > > I thought perhaps we could allow the usage of a "new" keyword to instanciate > an object, ie: > > obj = new yourmodule.YourClass() > > In this case, it would behave the same as from yourmodule import YourClass; > obj = YourClass(), except that it wouldn't need to be imported. This would > also eliminate the need to manage an import list at the beginning of a > script in most case. The 'py' library has something like this for stdlib libraries. You could imagine extending it to handle arbitrary auto-imports, e.g. import auto_import as ai obj = ai.yourmodule.YourClass() The 'py' version never really caught on, but if you really like the idea there's nothing stopping you from implementing and using something similar today. -n -- Nathaniel J. Smith -- https://vorpus.org From steve at pearwood.info Sat Mar 3 19:25:13 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 4 Mar 2018 11:25:13 +1100 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: <20180304002513.GE18868@ando.pearwood.info> On Sat, Mar 03, 2018 at 06:12:06PM +0100, Jamesie Pic wrote: > Hello everybody, > > I thought perhaps we could allow the usage of a "new" keyword to > instanciate an object, ie: > > obj = new yourmodule.YourClass() > > In this case, it would behave the same as from yourmodule import YourClass; > obj = YourClass(), except that it wouldn't need to be imported. Of course it would still need to be imported, it just wouldn't be imported *once* into the current module. So: a = new yourmodule.YourClass() b = new yourmodule.YourClass() c = new yourmodule.YourClass() d = new yourmodule.YourClass() would have to go through the process of importing yourmodule *four* times. Admittedly only the first time would be really expensive (and you can't escape that: this applies to `import` too), but the others won't be free. It will also break pickling. (I think.) There's also the serious cost of adding a new keyword, breaking everyone's code that uses "new" for something else. There has to be a good reason to do that, better than just "PHP offers this" or "I don't like managing imports". > I'm really not proud of this idea but PHP has had autoload for years and > when i open scripts with hundred lines of imports it makes me think Python > could do something about this. Somehow I don't feel that "hundreds of lines of imports" in a single script is realistic. Do you have an example of a publicly viewable script with hundreds of lines of imports? -- Steve From greg.ewing at canterbury.ac.nz Sat Mar 3 19:38:40 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Mar 2018 13:38:40 +1300 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <5A9B2EE6.1090409@canterbury.ac.nz> Message-ID: <5A9B4010.1070505@canterbury.ac.nz> Robert Vanden Eynde wrote: > But I think that the implementation of print(y with y = x + 1) would > be more close to next(y for y in [ x+1 ]) WHy on earth should it be? Expanding that gives you a generator containing a loop that only executes once that is immediately run to yield exactly one value. Why bother with all that overhead? -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 3 19:51:24 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Mar 2018 13:51:24 +1300 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: <5A9B430C.2030200@canterbury.ac.nz> > 2018-03-03 8:40 GMT+01:00 Nick Coghlan : >> pairs = [(f(y), g(y)) for x in things with bind(h(x)) as y] I don't mucn like "with bind(h(x)) as y" because it's kind of like an abstraction inversion -- you're building something complicated on top of something complicated in order to get something simple, instead of just having the simple thing to begin with. If nothing else, it has a huge runtime cost for the benefit it gives. -- Greg From greg.ewing at canterbury.ac.nz Sat Mar 3 19:58:55 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Sun, 04 Mar 2018 13:58:55 +1300 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <20180303110922.25t2wacqc4r5niyt@phdru.name> <5A9B3696.4010000@canterbury.ac.nz> Message-ID: <5A9B44CF.4020502@canterbury.ac.nz> Robert Vanden Eynde wrote: > The "make it a keyword in a context" might lead to things like > "print(where where where = where)", not sure you'd want that :) What the heck, if it was good enough for PL/1... > But that's just a choice of syntax, Choice of syntax is important, though. It's all very well to come up with an "insert syntax here" proposal that has some big obvious benefit, but if you can't actually come up with a pleasing syntax for it, then it won't fly. So it makes sense to consider potential syntaxes along with other aspects of a proposal. -- Greg From cs at cskk.id.au Sat Mar 3 19:41:53 2018 From: cs at cskk.id.au (Cameron Simpson) Date: Sun, 4 Mar 2018 11:41:53 +1100 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: <20180304004153.GA45760@cskk.homeip.net> On 03Mar2018 19:46, Jamesie Pic wrote: >While i understand it would be harder to make it memory efficient, but this >is python not go, and also this sort feature could be easily optional, >also, it might even help against circular import issues, whoever hasn't >imported a module from inside a function in their life may throw the first >rock at my face (kidding) > >Yes i know it's sad php has this feature and python does not and again i'm >not proud to say this but it's true. In a recent incarnation I spent a lot of time writing PHP. I've nothing good to say about it. However, to the "feature" point above: Its import facility is broken to the point iof being actively misleading. We weren't doing autoimports because it is good to be able to look at the top of a module and see what it uses. But even PHP's "use" statement is broken. It doesn't do anything until you access a name from it. _Then_ it goes to see if such a module exists. I blew a whole morning on this misbehaviour that didn't show unless a particular code path got to run. My personal belief is that the PHP people thought "this stuff runs on the fly per web page load, it needs to be fast fast fast!" And therefore their "use" statement just loads a string mapping of where to look for a name. The looking doesn't happen until the name gets used, if it gets used. So you can "use" a totally misspelt module and not realise it until very late in the game. We had a particularly skilled PHP guy on the team, and he and I still blew a couple of hours trying to figure out why one of my "use" statements didn't work, and that one _was_ due to some kind of autoimport magic I now forget. Personally, I'm -1 for an autoimport because it doesn't actually buy much, and -2 for any autoimport based on PHP semantics, which generally lead to "fail late" instead of "fail early". Cheers, Cameron Simpson From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Mar 4 06:09:25 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 4 Mar 2018 20:09:25 +0900 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FEEE.4000109@canterbury.ac.nz> <5A9A5B90.6080001@stoneleaf.us> Message-ID: <23195.54245.700524.989371@turnbull.sk.tsukuba.ac.jp> Chris Angelico writes: > > So what happens to rejected alternatives when the PEP itself is rejected? > > ;) > > > > Algebraically, they must be accepted. Right? That's not how de Moivre's laws work. ;-) From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Mar 4 06:09:46 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 4 Mar 2018 20:09:46 +0900 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <56fd1a38-4dfb-c830-f79b-bbfa564cdf93@gmail.com> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <20180303110922.25t2wacqc4r5niyt@phdru.name> <56fd1a38-4dfb-c830-f79b-bbfa564cdf93@gmail.com> Message-ID: <23195.54266.972236.479639@turnbull.sk.tsukuba.ac.jp> Soni L. writes: > [(lambda y: (f(y), g(y)))(h(x)) for x in things] ? MicroLisp? From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Mar 4 06:10:47 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 4 Mar 2018 20:10:47 +0900 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> Message-ID: <23195.54327.737261.258103@turnbull.sk.tsukuba.ac.jp> Nick Coghlan writes: > pairs = [(f(y), g(y)) for x in things with y = h(x)] > contents = [f.read() for fname in filenames with open(fname) as f] This is horrible. I think Julia is just weird: in normal English we do distinguish between equality and membership. "x in y" is a very different statement from "x = y". I think even Guido would come around to the view if it were implemented (assuming not "over his dead body"). But the semantics of "x = y" and "y as x" in English are both pretty much the copula. It's hard enough to stay aware that there be dragons in a context manager; if "with" could denote a simple (local) binding, it would require conscious effort. > Even without the "with name = expr" change, allowing with clauses in > comprehensions would let you do (by way of a suitably defined "bind" CM): > > pairs = [(f(y), g(y)) for x in things with bind(h(x)) as y] This is *much* better. But suppose you wanted to have *several* bindings. Would this syntax allow "destructuring" of tuples (as the for clause will do): pairs = [(f(x) + g(y), f(x) - g(y)) for w, z in pairs_of_things with bind((h(w), k(z)) as (x, y)] ? This is a question about all the proposals for local binding, I think. Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Sun Mar 4 06:12:44 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 4 Mar 2018 20:12:44 +0900 Subject: [Python-ideas] Class autoload In-Reply-To: References: <93a472ca-21eb-0b0a-7554-c1fca8416247@trueblade.com> Message-ID: <23195.54444.11404.503511@turnbull.sk.tsukuba.ac.jp> Jamesie Pic writes: > obj = new yourmodule.YourClass() I don't understand what this is good for. Keeping up with PHP is not something that is a goal for Python. Borrowing useful features is definitely an idea, but you need to explain why it's useful. I also don't understand why you call this "autoload", and claim it avoids importing the module. The "autoload" I'm familiar with from Lisp is actually lazy loading: the object's name is marked as loadable and from where, but the module is loaded only if the name is dereferenced. In most implementations, the whole module is imported, as it is in Python. Experience says that modules are generally coherent, with lots of internal cross-references: you'll need the whole thing eventually anyway. I would guess this is true in PHP as well, as it is a very dynamic language too AIUI. Various barriers you'll need to clear: (1) Niggle: "new" already means something in Python that is different from initialization. (class.__new__ vs. class.__init__) You *could* ignore that and use "new" for this feature anyway, but you probably want to choose a different name. (2) Adding keywords is *very* hard to do in Python, for several non-technical reasons. This is not going to clear the bar. (3) Without *very* major changes to Python, you're not going to be able to import just that class (unless it's the entire content of the module). You have to import the whole module into memory anyway. > While i understand it would be harder to make it memory efficient, I'm not sure what you mean by "memory efficient". If you mean multiple copies of the module in memory, that won't happen anyway: Python knows what it has imported already, and when using the "from module import anothername as alias" syntax, the second time you import "from module", it reuses the existing copy in memory even though "module" isn't known in that namespace. This "new" syntax would be treated the same way. If you mean only importing enough of the module to make YourClass work, I don't think that will ever happen. That would require a kind of global knowledge of the application that is difficult, and maybe theoretically impossible, in a language as dynamic as Python. > but this is python not go, I have no idea what that is supposed to mean. > and also this sort feature could be easily optional, It's not optional for your successors in your job, though. They have to read your code. We care about readers of code (in general, not just your successor) more than we care about writers of code. Why make things more complicated for them? There has to be a real win in expressiveness or power here. I don't see it yet. We know what happens when you concentrate on making everything as terse as possible: Perl. Python will never be a good imitation of Perl; there's no real point in trying since we already have Perl if we want it. > also, it might even help against circular import issues You're just blowing smoke here, aren't you? I see no reason why this would help with circular import issues due to the semantics of import in Python: you get the whole module whether you need it or not. Without the global application analysis mentioned previously, "new" can't know that it's OK to omit any of the imports. > Yes i know it's sad php has this feature and python does not As several other posters say, it's not clear this is a feature at all from Python's point of view. I don't think it is. I've never used PHP so I have no idea what it's good for, but I suppose the same holds for it as for Perl: Python doesn't try to be PHP, so any feature that PHP has but Python doesn't needs to be justified on its merits *in Python*, not in PHP. > and again i'm not proud to say this but it's true. It's a fact. There's nothing to be ashamed of. Diversity is good. Regards, From ncoghlan at gmail.com Sun Mar 4 06:38:34 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 4 Mar 2018 21:38:34 +1000 Subject: [Python-ideas] Class autoload In-Reply-To: References: Message-ID: On 4 March 2018 at 03:42, Chris Angelico wrote: > On Sun, Mar 4, 2018 at 4:12 AM, Jamesie Pic wrote: > > > > Hello everybody, > > > > I thought perhaps we could allow the usage of a "new" keyword to > instanciate > > an object, ie: > > > > obj = new yourmodule.YourClass() > > > > In this case, it would behave the same as from yourmodule import > YourClass; > > obj = YourClass(), except that it wouldn't need to be imported. This > would > > also eliminate the need to manage an import list at the beginning of a > > script in most case. > > > > This actually has nothing to do with classes. You can currently write this: > > import yourmodule > obj = yourmodule.YourClass() > > without any sort of 'new' keyword. So presumably what you're asking > for is a way to avoid typing the 'import' statement. > > That's something that's come up every once in a while. Usually for the > benefit of throwaway scripts and the interactive interpreter, because > in serious applications, a single 'import' line is a small price to > pay for the clarity. You may want to dig through the archives to find > the arguments for and against this sort of automated import. > > > I'm really not proud of this idea but PHP has had autoload for years and > > when i open scripts with hundred lines of imports it makes me think > Python > > could do something about this. > > A hundred lines of imports? Surely an exaggeration... or possibly you > have a whole lot of "from module import name" lines that could become > a single "import module" line. Excessive numbers of top level imports in a single file are also frequently a sign that that module has too many responsibilities and could stand to be split up (either as multiple parallel APIs, or by splitting out a lower level helper library) for ease of comprehension. Those split up API implementations can then be aggregated back together into a single public API (e.g. in a package `__init__.py` file) As Chris has noted, function level lazy imports are also already supported, and we've (begrudgingly) made the import system work harder to successfully resolve circular imports in recent releases (they're also much harder to dead lock in Python 3 than they were in Python 2, since we also switched over to per-module import locks). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From adelfino at gmail.com Sun Mar 4 12:59:40 2018 From: adelfino at gmail.com (=?UTF-8?Q?Andr=C3=A9s_Delfino?=) Date: Sun, 4 Mar 2018 14:59:40 -0300 Subject: [Python-ideas] Split, slice, join and return "syntax" for str Message-ID: Hi! I was thinking: perhaps it would be nice to be able to quicky split a string, do some slicing, and then obtaining the joined string back. Say we have the string: "docs.python.org", and we want to change "docs" to "wiki". Of course, there are a ton of simpler ways to solve this particular need, but perhaps str could have something like this: spam = "docs.python.org" eggs = "wiki." + spam['.'][1:] print(eggs) #wiki.python.org A quick implementation to get the idea and try it: class Mystr(str): def __getitem__(self, item): if isinstance(item, str): return Mystr_helper(self, item) else: return super().__getitem__(item) class Mystr_helper: def __init__(self, obj, sep): self.obj = obj self.sep = sep def __getitem__(self, item): return self.sep.join(self.obj.split(self.sep)[item]) What are your thoughts? Greetings from Argentina. -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Sun Mar 4 13:18:44 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Sun, 4 Mar 2018 19:18:44 +0100 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings Message-ID: > Robert Vanden Eynde wrote: >> >> But I think that the implementation of print(y with y = x + 1) would >> be more close to next(y for y in [ x+1 ]) > > > WHy on earth should it be? Expanding that gives you a generator > containing a loop that only executes once that is immediately run > to yield exactly one value. Why bother with all that overhead? > You're probably right. From robertve92 at gmail.com Sun Mar 4 13:20:49 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Sun, 4 Mar 2018 19:20:49 +0100 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings Message-ID: > What the heck, if it was good enough for PL/1... It would still be parsable indeed. A keyword available in a context would then be something new in the language. > Choice of syntax is important, though. It's all very well > to come up with an "insert syntax here" proposal that has > some big obvious benefit, but if you can't actually come > up with a pleasing syntax for it, then it won't fly. So > it makes sense to consider potential syntaxes along with > other aspects of a proposal. Of course the syntax is very important, that's why we make a list of pro's and con's about each syntax. However, people also want to know if the feature is useful for a lot of users. So, should we do that list somewhere ? :) From robertve92 at gmail.com Sun Mar 4 13:18:00 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Sun, 4 Mar 2018 19:18:00 +0100 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings Message-ID: > I know Guido is on record as not wanting to allow both "for name in > sequence" and "for name = expr" due to that being a very subtle distinction > between iteration and simple assignment (especially given that Julia uses > them as alternate spellings for the same thing), but I'm wondering if it may > be worth considering such a distinction in *with statements*, such that we > allowed "with name = expr" in addition to "with cm as name" (where "name = > expr" is just an ordinary assignment statement that doesn't trigger the > context management protocol). > > The related enhancement to comprehensions would then be to allow with > clauses to be interleaved between the for loops and the if statements, such > that you could write things like: > > pairs = [(f(y), g(y)) for x in things with y = h(x)] > contents = [f.read() for fname in filenames with open(fname) as f] > > while still preserving the property where comprehensions can be correctly > interpreted just by converting each of the clauses to the corresponding > statement form. > > Even without the "with name = expr" change, allowing with clauses in > comprehensions would let you do (by way of a suitably defined "bind" CM): > > pairs = [(f(y), g(y)) for x in things with bind(h(x)) as y] Including standard "with open(...) as f" context management in list comprehensions would also be a good idea, I think it's in the same idea as including : allowing both statements (imperative programming) and list comprehension (more "functional"). The same story goes for exceptions : for x in range(5): try: yield f(x) except Exception: yield 0 isn't "refactorable" as a list comprehension like : [try f(i) except Exception: 0 for x in range(5)] But I think that would cover another subject :) But if the "with =" syntax is accepted for simple assignment, it could indeed introduce a confusion compared to the "with as" in context management, and having a complete different keyword (or punctuation) is maybe be better. As a symmetry, we could also introduce the "with =" syntax in normal statement :P that'd do a normal assignement, although that's be probably stupid because normal assignements works well. Robert From clint.hepner at gmail.com Sun Mar 4 13:44:20 2018 From: clint.hepner at gmail.com (Clint Hepner) Date: Sun, 4 Mar 2018 13:44:20 -0500 Subject: [Python-ideas] Split, slice, join and return "syntax" for str In-Reply-To: References: Message-ID: > On 2018 Mar 4 , at 12:59 p, Andr?s Delfino wrote: > > Hi! > > I was thinking: perhaps it would be nice to be able to quicky split a string, do some slicing, and then obtaining the joined string back. > > Say we have the string: "docs.python.org", and we want to change "docs" to "wiki". Of course, there are a ton of simpler ways to solve this particular need, but perhaps str could have something like this: > > spam = "docs.python.org" > eggs = "wiki." + spam['.'][1:] > print(eggs) #wiki.python.org -1. I see no compelling reason to overload __getitem__ to provide a synonym for the split method. eggs = "wiki." + spam.split('.')[1:] Besides, you can already make such replacements more efficiently with eggs = spam.replace('docs', 'wiki') or, for more complex replacements, eggs = re.sub('^docs', 'wiki', spam) -- Clint From desmoulinmichel at gmail.com Sun Mar 4 15:11:16 2018 From: desmoulinmichel at gmail.com (Michel Desmoulin) Date: Sun, 4 Mar 2018 12:11:16 -0800 Subject: [Python-ideas] Split, slice, join and return "syntax" for str In-Reply-To: References: Message-ID: Even if replace would be a better fit, I can see why doing those 3 operations in one row can be valuable. But, first, they are not common enough so that it's hard to do: spam = docs.python.org" eggs = 'wiki.' + '.'.join(spams.split('.')[1:]) It's not that long to type, and certainly is not happening in every single script you do. But let's say for the sake of argument you do a lot of cmd parsing, with a lot of split and join. Just make a helper: def rearrange(string, sep=None, start=None, stop=None, step=None): return sep.join(string.split(sep)[start:stop:step]) And then you can do: spam = docs.python.org" eggs = 'wiki.' + rearrange(spam, '.', 1) Simple, easy, no need to change Python. Le 04/03/2018 ? 09:59, Andr?s Delfino a ?crit?: > Hi! > > I was thinking: perhaps it would be nice to be able to quicky split a > string, do some slicing, and then obtaining the joined string back. > > Say we have the string: "docs.python.org ", and > we want to change "docs" to "wiki". Of course, there are a ton of > simpler ways to solve this particular need, but perhaps str could have > something like this: > > spam = "docs.python.org " > eggs = "wiki." + spam['.'][1:] > print(eggs) #wiki.python.org > > A quick implementation to get the idea and try it: > > class Mystr(str): > ??? def __getitem__(self, item): > ??????? if isinstance(item, str): > ??????????? return Mystr_helper(self, item) > ??????? else: > ??????????? return super().__getitem__(item) > > class Mystr_helper: > ??? def __init__(self, obj, sep): > ??????? self.obj = obj > ??????? self.sep = sep > ??? def __getitem__(self, item): > ??????? return self.sep.join(self.obj.split(self.sep)[item]) > > What are your thoughts? > > Greetings from Argentina. > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From steve at pearwood.info Sun Mar 4 18:23:49 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Mar 2018 10:23:49 +1100 Subject: [Python-ideas] Split, slice, join and return "syntax" for str In-Reply-To: References: Message-ID: <20180304232346.GF18868@ando.pearwood.info> On Sun, Mar 04, 2018 at 01:44:20PM -0500, Clint Hepner wrote: > -1. I see no compelling reason to overload __getitem__ to provide a synonym for the split method. > > eggs = "wiki." + spam.split('.')[1:] Fair point. Neither do I. But your next comment: > Besides, you can already make such replacements more efficiently with > > eggs = spam.replace('docs', 'wiki') is not suitable, because it assumes that the substring "docs" appears only once in the string. In a more realistic example, you don't know what the string contains, only that it is delimited by dots and that you want to replace the n-th field (whatever it contains) with "wiki". -- Steve From steve at pearwood.info Sun Mar 4 19:12:50 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 5 Mar 2018 11:12:50 +1100 Subject: [Python-ideas] Split, slice, join and return "syntax" for str In-Reply-To: References: Message-ID: <20180305001247.GG18868@ando.pearwood.info> On Sun, Mar 04, 2018 at 12:11:16PM -0800, Michel Desmoulin wrote: > But, first, they are not common enough so that it's hard to do: > > spam = docs.python.org" > eggs = 'wiki.' + '.'.join(spams.split('.')[1:]) In a more realistic case, the delimiter is not necessarily a constant, nor will you always want to replace the first item. So you would be writing: delimiter.join( spam.split(delimiter)[:n-1] + ['wiki'] + spam.split(delimiter)[n+1:] ) Suddenly it's a lot less attractive to be typing out each time. It is a good candidate for a helper function: def replace_field(string, delimiter, position, new, start=0, end=None): fields = string[start:end].split(delimiter) fields[position] = new string = (string[:start] + delimiter.join(fields) + ('' if end is None else string[end:])) return string That's not the most efficient implementation, and it isn't thoroughly tested/debugged, but it ought to be good enough for casual use. Personally, if this were available as a string method I think I'd use this quite frequently, certainly more often than I use some other string methods like str.partition. > It's not that long to type, and certainly is not happening in every > single script you do. Speak for yourself :-) No, not literally every script. (If that were the requirement to be a string method, strings would have no methods.) But I think it is common enough that I'd be happy for it to be a string method. -- Steve From python-ideas at mgmiller.net Sun Mar 4 22:39:54 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sun, 4 Mar 2018 19:39:54 -0800 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <5A9B430C.2030200@canterbury.ac.nz> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <5A9B430C.2030200@canterbury.ac.nz> Message-ID: <546cf374-4bb3-ac55-d5c5-3593c81013f8@mgmiller.net> On 2018-03-03 16:51, Greg Ewing wrote: >> 2018-03-03 8:40 GMT+01:00 Nick Coghlan : >>> ?? pairs = [(f(y), g(y)) for x in things with bind(h(x)) as y] > > I don't mucn like "with bind(h(x)) as y" because it's kind of > like an abstraction inversion -- you're building something > complicated on top of something complicated in order to get > something simple, instead of just having the simple thing > to begin with. If nothing else, it has a huge runtime cost > for the benefit it gives. Reading this thread I was thinking that the assignment part was happening too far away from where the action was and came up with this variant: [ f(y), g(y) for x, y as h(x) in things ] Plus or minus an extra set of parentheses for clarity. -Mike From rosuav at gmail.com Sun Mar 4 22:45:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Mar 2018 14:45:21 +1100 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <546cf374-4bb3-ac55-d5c5-3593c81013f8@mgmiller.net> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <5A9B430C.2030200@canterbury.ac.nz> <546cf374-4bb3-ac55-d5c5-3593c81013f8@mgmiller.net> Message-ID: On Mon, Mar 5, 2018 at 2:39 PM, Mike Miller wrote: > > On 2018-03-03 16:51, Greg Ewing wrote: >>> >>> 2018-03-03 8:40 GMT+01:00 Nick Coghlan : >>>> >>>> pairs = [(f(y), g(y)) for x in things with bind(h(x)) as y] >> >> >> I don't mucn like "with bind(h(x)) as y" because it's kind of >> like an abstraction inversion -- you're building something >> complicated on top of something complicated in order to get >> something simple, instead of just having the simple thing >> to begin with. If nothing else, it has a huge runtime cost >> for the benefit it gives. > > > Reading this thread I was thinking that the assignment part was happening > too far away from where the action was and came up with this variant: > > [ f(y), g(y) for x, y as h(x) in things ] > > Plus or minus an extra set of parentheses for clarity. Did you mean: [ f(y), g(y) for x, h(x) as y in things ] ? Elsewhere in Python, "a as b" takes "a" and binds it to the name "b". Otherwise, I'm not sure what you meant. ChrisA From python-ideas at mgmiller.net Sun Mar 4 22:49:39 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Sun, 4 Mar 2018 19:49:39 -0800 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <5A9B430C.2030200@canterbury.ac.nz> <546cf374-4bb3-ac55-d5c5-3593c81013f8@mgmiller.net> Message-ID: <7de82d71-d562-4638-d084-880b75ad39a4@mgmiller.net> Yes, thanks: [ f(y), g(y) for x, h(x) as y in things ] Dyslexics untie! On 2018-03-04 19:45, Chris Angelico wrote: From rosuav at gmail.com Sun Mar 4 23:04:41 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 5 Mar 2018 15:04:41 +1100 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: <7de82d71-d562-4638-d084-880b75ad39a4@mgmiller.net> References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <5A9B430C.2030200@canterbury.ac.nz> <546cf374-4bb3-ac55-d5c5-3593c81013f8@mgmiller.net> <7de82d71-d562-4638-d084-880b75ad39a4@mgmiller.net> Message-ID: On Mon, Mar 5, 2018 at 2:49 PM, Mike Miller wrote: > Yes, thanks: > > [ f(y), g(y) for x, h(x) as y in things ] > > > Dyslexics untie! :) Hmm. The trouble here is that a 'for' loop is basically doing assignment. When you say "for x, h(x) as y in things", what Python does is (simplified): _it = iter(things) while not StopIteration: x, (h(x) as y) = next(_it) ... loop body ... So what you're asking for is something that doesn't behave like an assignment target, but just does its own assignment. Bear in mind, too, that "x = next(_it)" is very different from "x, = next(_it)"; which one is "x, (h(x) as y) = next(_it)" more like? ChrisA From neatnate at gmail.com Sun Mar 4 23:15:07 2018 From: neatnate at gmail.com (Nathan Schneider) Date: Sun, 4 Mar 2018 23:15:07 -0500 Subject: [Python-ideas] An alternative to PEP 572's Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <5A99FC27.3000809@canterbury.ac.nz> <5A9B430C.2030200@canterbury.ac.nz> <546cf374-4bb3-ac55-d5c5-3593c81013f8@mgmiller.net> <7de82d71-d562-4638-d084-880b75ad39a4@mgmiller.net> Message-ID: On Sun, Mar 4, 2018 at 11:04 PM, Chris Angelico wrote: > On Mon, Mar 5, 2018 at 2:49 PM, Mike Miller > wrote: > > Yes, thanks: > > > > [ f(y), g(y) for x, h(x) as y in things ] > > > > > > Dyslexics untie! > > :) > > Hmm. The trouble here is that a 'for' loop is basically doing > assignment. When you say "for x, h(x) as y in things", what Python > does is (simplified): > > _it = iter(things) > while not StopIteration: > x, (h(x) as y) = next(_it) > ... loop body ... > > So what you're asking for is something that doesn't behave like an > assignment target, but just does its own assignment. Bear in mind, > too, that "x = next(_it)" is very different from "x, = next(_it)"; > which one is "x, (h(x) as y) = next(_it)" more like? > > If I understood Mike's proposal correctly it was to allow "," to mean 'given' in this context when followed by EXPRESSION "as" VARIABLE, rather than its usual iterable-unpacking meaning. But I think this would cause confusion. Suppose `things` contains pair-tuples. Then you could have [ f(y), g(y) for x1, x2, h(x1) as y in things ] which makes it look like (x1, x2, h(x1)) go together rather than just (x1, x2). Nathan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Mon Mar 5 09:42:04 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Mon, 05 Mar 2018 06:42:04 -0800 Subject: [Python-ideas] Class autoload In-Reply-To: References: <93a472ca-21eb-0b0a-7554-c1fca8416247@trueblade.com> Message-ID: <5A9D573C.8000504@stoneleaf.us> On 03/03/2018 10:46 AM, Jamesie Pic wrote: > While i understand it would be harder to make it memory efficient, but this is python not go, and also this sort feature > could be easily optional, also, it might even help against circular import issues, whoever hasn't imported a module from > inside a function in their life may throw the first rock at my face (kidding) > > Yes i know it's sad php has this feature and python does not and again i'm not proud to say this but it's true. Not sad at all. It's a misfeature that Python is better off without. -- ~Ethan~ From rhodri at kynesim.co.uk Tue Mar 6 09:51:32 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Tue, 6 Mar 2018 14:51:32 +0000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <3ef29439-079d-e349-75d8-cdd813f94233@kynesim.co.uk> <82c73823-7ea3-0e7e-cd4e-1b10b2678b44@kynesim.co.uk> Message-ID: On 02/03/18 20:02, Jonathan Goble wrote: > On Fri, Mar 2, 2018 at 2:56 PM Robert Vanden Eynde > wrote: > >> @Rhodri, this is what Everybody does because you hit the "reply to all" >> button, but, we don't receive two copies on the mail, I don't know why (but >> that's a good thing). >> >> Le 2 mars 2018 20:48, "Rhodri James" a ?crit : >> >> On 02/03/18 18:27, Jelle Zijlstra wrote: >> >>> 2018-03-02 7:03 GMT-08:00 Robert Vanden Eynde : >>> >> >> Guys, please don't email to me *and* the mailing list. Getting two copies >> of your deathless prose makes me less likely to pay attention to you, not >> more. >> >> >> -- >> Rhodri James *-* Kynesim Ltd >> >> > *de-lurks* There is an option in your personal settings for the list to > choose whether to receive duplicate copies of messages sent to both you and > the list. It is usually wise to set that to "no". > > To check this option, go to > https://mail.python.org/mailman/options/python-ideas, enter your email > address and password (if you don't recall your password, just enter your > email address and click "Remind" at the bottom of the page), and click "Log > in". From there, you can change a variety of options, including the one > about receiving duplicate copies of messages aimed at both you and the list. I checked, and I have "Avoid duplicate copies" set to "Yes". Curious. Not worth chasing up if we're going to change software soon. -- Rhodri James *-* Kynesim Ltd From mistersheik at gmail.com Tue Mar 6 12:59:00 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 6 Mar 2018 09:59:00 -0800 (PST) Subject: [Python-ideas] Consider adding reverse argument to heapq methods. Message-ID: <60901cf5-9bee-4586-b26c-031331806a2b@googlegroups.com> heapq creates and works with min-heaps. Currently, the only way to do this is to use _heapify_max instead of heapify, heapq._heappop_max instead of heapq.heappop, etc. These methods should be exposed using a reverse keyword argument rather than as private methods just like sort. Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikez302 at gmail.com Thu Mar 8 02:29:12 2018 From: mikez302 at gmail.com (Elias Zamaria) Date: Wed, 7 Mar 2018 23:29:12 -0800 (PST) Subject: [Python-ideas] Consider adding reverse argument to heapq methods. In-Reply-To: <60901cf5-9bee-4586-b26c-031331806a2b@googlegroups.com> References: <60901cf5-9bee-4586-b26c-031331806a2b@googlegroups.com> Message-ID: Or if we are willing to make bigger changes to the module, there should be a sort of class that can remember if it is supposed to represent a min-heap or a max-heap. This has been discussed several times: - https://groups.google.com/d/topic/python-ideas/h3WLmpBG4ug/discussion - https://groups.google.com/d/topic/python-ideas/WQMhF79oCzw/discussion - https://groups.google.com/d/topic/python-ideas/NmL53DLTqpo/discussion - https://groups.google.com/d/topic/python-ideas/cLIAhBbQ8xA/discussion But nothing definite has come out of those discussions, as far as I can tell. On Tuesday, March 6, 2018 at 9:59:00 AM UTC-8, Neil Girdhar wrote: > > heapq creates and works with min-heaps. Currently, the only way to do > this is to use _heapify_max instead of heapify, heapq._heappop_max instead > of heapq.heappop, etc. > > These methods should be exposed using a reverse keyword argument rather > than as private methods just like sort. > > Best, > > Neil > -------------- next part -------------- An HTML attachment was scrubbed... URL: From waksman at gmail.com Thu Mar 8 03:47:43 2018 From: waksman at gmail.com (George Leslie-Waksman) Date: Thu, 08 Mar 2018 08:47:43 +0000 Subject: [Python-ideas] Argparse subparsers add required keyword Message-ID: With the stdlib argparse, subparsers can be defined and they can be marked as required (though this is not documented) but they do not support a "required" keyword. I think it would make everything more consistent if the keyword existed. This won't require any functional changes under the hood. Right now, this works (and behaves as expected): parser = argparse.ArgumentParser(...) subparsers = parser.add_subparsers(...) subparsers.required = True but this does not: parser = argparse.ArgumentParser(...) subparsers = parser.add_subparsers(..., required=True) --George -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucas.wiman at gmail.com Fri Mar 9 19:48:26 2018 From: lucas.wiman at gmail.com (Lucas Wiman) Date: Fri, 9 Mar 2018 16:48:26 -0800 Subject: [Python-ideas] Add MutableSet.update? Message-ID: I was trying to use `collections.abc.MutableSet` today, and noticed that it does not define an `update` method. This is a very useful method that is present on the builtin `set` class, and seems to fit naturally with the other methods. Was omitting this method intentional? Thanks & best wishes, Lucas Wiman -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Fri Mar 9 20:15:36 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 9 Mar 2018 17:15:36 -0800 Subject: [Python-ideas] Add MutableSet.update? In-Reply-To: References: Message-ID: Yes, you can use the |= operator instead. On Fri, Mar 9, 2018 at 4:48 PM, Lucas Wiman wrote: > I was trying to use `collections.abc.MutableSet` today, and noticed that > it does not define an `update` method. This is a very useful method that is > present on the builtin `set` class, and seems to fit naturally with the > other methods. > > Was omitting this method intentional? > > Thanks & best wishes, > Lucas Wiman > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From python at mrabarnett.plus.com Fri Mar 9 21:24:07 2018 From: python at mrabarnett.plus.com (MRAB) Date: Sat, 10 Mar 2018 02:24:07 +0000 Subject: [Python-ideas] Add MutableSet.update? In-Reply-To: References: Message-ID: <4de9f5c5-895a-9c8c-aa06-3a0d3986e885@mrabarnett.plus.com> On 2018-03-10 01:15, Guido van Rossum wrote: > Yes, you can use the |= operator instead. > |= is not quite the same as .update because it rebinds, so if the name on the LHS isn't local it'll raise NameError. Does that matter? > On Fri, Mar 9, 2018 at 4:48 PM, Lucas Wiman > wrote: > > I was trying to use `collections.abc.MutableSet` today, and noticed > that it does not define an `update` method. This is a very useful > method that is present on the builtin `set` class, and seems to fit > naturally with the other methods. > > Was omitting this method intentional? > > Thanks & best wishes, > Lucas Wiman > From tomtervoort at gmail.com Sat Mar 10 10:35:47 2018 From: tomtervoort at gmail.com (Tom Tervoort) Date: Sat, 10 Mar 2018 16:35:47 +0100 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) Message-ID: Hi all, PEP 506, which introduces the secrets module, states the following: Default arguments One difficult question is "How many bytes should my token be?". We can help with this question by providing a default amount of entropy for the "token_*" functions. If the nbytes argument is None or not given, the default entropy will be used. This default value should be large enough to be expected to be secure for medium-security uses, but is expected to change in the future, possibly even in a maintenance release [14]. Currently, the default output size of the secrets.token_* functions is 32 bytes. This is definitely more than enough to protect against brute-force attacks in almost any feasible scenario, and that is a good thing. It's always better to pick a conservative default. If smaller tokens could introduce a security risk in 1% of use cases, I agree smaller tokens should not be used. However, I would like to argue that 32-byte tokens are unnecessarily long for far more than 99% of use cases. Instead, I'd suggest using a default length of 16 bytes (128 bits), which already has a very high security margin. Brute-force attacks and cryptography ------------------------------------ Most cryptographic systems are inherently vulnerable to brute-force attacks. For example, if you were somehow capable of performing 2^128 AES decryption operations, you would be able to crack AES. If you could perform the same amount of SHA256 hash iterations, you could find a hash collision. Doing 2^128 computations of any kind is completely and utterly infeasible. All (super-)computers in the world combined would not even come close in a million years. That's why cryptographers typically use this 128 bits "security level" as a de facto standard for how many brute-force operations a cryptosystem should tolerate. This influences properties such as key length and signature sizes. However, there is an important difference between "offline" and "online" brute-force attacks. In an online scenario, an attacker can not immediately determine whether their guess was correct; they first have to submit the result to another party (e.g. a server) which can either accept or reject it. In such a case, it is acceptable when an attacker needs to perform 2^64 operations in order to break the system (cryptographic authenticator CBC-MAC-AES can be broken after this many forgery attempt, for example). Note that when 2^64 guesses will break the system, an attacker doing random guesses is expected to perform the correct guess halfway through (i.e. after about 2^63 attempts). Basically, a 128 bit security level entails the following assumptions: - An attacker is not capable of performing 2^64 online guesses. - An attacker (without a quantum computer) is not capable of performing 2^128 offline guesses. - An attacker with a large quantum computer, capable of performing Grover's algorithm, is not capable of doing a search through a space of 2^256 elements (which is why quantum-resistant symmetric ciphers use 32-byte keys). The birthday paradox -------------------- When you keep drawing random integers between 0 and 2^N, it will take about sqrt(2^N) = 2^(N/2) draws before you encounter a number a second time (see https://en.wikipedia.org/wiki/Birthday_problem#An_upper_bound). This is relevant for hash collisions, for example, and the reason we have SHA256 instead of SHA128. If a system would allow an attacker to create a resource that is identified by a 64-bit UUID, they would only need to create about 2^32 resources (not infeasible) before they would have two that share an identifier, which could lead to all sorts of problems. So this is something that should be taken into account when generating random tokens. Use-cases of secret tokens -------------------------- Let's consider some of the typical security-sensitive applications for unguessable tokens, and see how long tokens should be in order to achieve the security level described above: Authenticity proof ~~~~~~~~~~~~~~~~~~ A user needs to prove that they know a specific secret value, such as a generated password, license code or anti-CSRF token. An 8-byte random secret is sufficient here: guessing the secret only has a 1 in 2^64 chance of success. Secret index ~~~~~~~~~~~~ The user provides a secret which identifies them or a certain resource. The application stores many different secret indices, and someone who knows secret A should not be able to guess secret B. These secrets could e.g. be session identifiers, password reset tokens, or secret URL's, 16 random bytes are certainly sufficient here, even in the extreme case that a server would store 2^64 (more than nine quintillion) records; since the chance of the attacker guessing one of them is again 1 in 2^64. Nonces and salts ~~~~~~~~~~~~~~~~ A nonce is a value that should only be used once, and is often used for symmetric ciphers or for replay attack protection. Salts used for password hashing are also an example of nonces. Seeds for deterministic CSRNG's can also be considered as (secret) nonces. When nonces are randomly generated (the easiest way to come up with a unique one), making them 16 bytes long is sufficient: due to the birthday problem one would need to generate about 2^64 of them before a collision occurs. Cryptographic keys ~~~~~~~~~~~~~~~~~~ The required length of a key depends on the cryptographic system for which it is used. If a crypto library does not enforce length requirements, I'd consider that a bug in the library. Well, okay. The HMAC algorithm poses no length requirements on keys and the Python library reflects that. For HMAC, 16 byte keys are okay, unless you want post-quantum security (which you won't get anyway when you use HMAC-MD5, which is still the default). Except for the far-fetched post-quantum HMAC scenario, I can't really think of a reasonably realistic situation where a security issue is introduced because the result of secrets.token_bytes is 16 bytes long instead of 32. Can anyone else here think of an example? Wouldn't changing this result in compatibility issues? ------------------------------------------------------ The docs describe the default value of nbytes as "a reasonable default" and state that "That default is subject to change at any time, including during maintenance releases.". Since the secrets module is still relatively new, this would be a good moment to change it, right? 32 > 16, so what's the problem? ------------------------------- First of all, I'd like to point out that the three examples in the docs (https://docs.python.org/3.6/library/secrets.html#generating-tokens) explicitly generate 16-byte tokens. If the current default is the most secure solution, why deviate from it here? I think there a quite some situations where unnecessarily long codes can cause usability problems: when using a system where users have to manually type in a random code (I frequently have to do that when using the password manager on my phone, for example) it's nice if you can save half of the time they have to spend on that. Shorter codes can also be converted to smaller QR codes, and to nicer URLs. Sure, a user can always choose to just pick a value for nbytes and set their own length. However, if they want to pick a secure value, they are being tasked with considering the implications. A user should be able to rely on the library picking a good secure default. I think security functionality should be made as ergonomic as possible, without compromising security. When security features unnecessarily create usability issues, people are discouraged from using them in the first place. So yeah, that was my argument for making tokens 16 bytes long instead of 32. Please let me know if and why I am completely wrong:P Regards, Tom -------------- next part -------------- An HTML attachment was scrubbed... URL: From lucas.wiman at gmail.com Sat Mar 10 17:13:10 2018 From: lucas.wiman at gmail.com (Lucas Wiman) Date: Sat, 10 Mar 2018 14:13:10 -0800 Subject: [Python-ideas] Add MutableSet.update? In-Reply-To: <4de9f5c5-895a-9c8c-aa06-3a0d3986e885@mrabarnett.plus.com> References: <4de9f5c5-895a-9c8c-aa06-3a0d3986e885@mrabarnett.plus.com> Message-ID: On Fri, Mar 9, 2018 at 6:24 PM, MRAB wrote: > On 2018-03-10 01:15, Guido van Rossum wrote: > >> Yes, you can use the |= operator instead. >> >> |= is not quite the same as .update because it rebinds, so if the name on > the LHS isn't local it'll raise NameError. > > Does that matter? Not for my use case; that should work great. Thanks Guido! - Lucas -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Sat Mar 10 17:45:59 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Sat, 10 Mar 2018 23:45:59 +0100 Subject: [Python-ideas] Add MutableSet.update? In-Reply-To: <4de9f5c5-895a-9c8c-aa06-3a0d3986e885@mrabarnett.plus.com> References: <4de9f5c5-895a-9c8c-aa06-3a0d3986e885@mrabarnett.plus.com> Message-ID: On Sat, Mar 10, 2018 at 3:24 AM, MRAB wrote: > On 2018-03-10 01:15, Guido van Rossum wrote: > >> Yes, you can use the |= operator instead. >> >> |= is not quite the same as .update because it rebinds, isn't that an "in-place operator" i.e. if it's a mutable object it should mutate rather than rebinding? -CHB > so if the name on the LHS isn't local it'll raise NameError. > > Does that matter? > > > On Fri, Mar 9, 2018 at 4:48 PM, Lucas Wiman > > wrote: >> >> I was trying to use `collections.abc.MutableSet` today, and noticed >> that it does not define an `update` method. This is a very useful >> method that is present on the builtin `set` class, and seems to fit >> naturally with the other methods. >> >> Was omitting this method intentional? >> >> Thanks & best wishes, >> Lucas Wiman >> >> _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sat Mar 10 18:20:57 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 10 Mar 2018 15:20:57 -0800 Subject: [Python-ideas] Add MutableSet.update? In-Reply-To: References: <4de9f5c5-895a-9c8c-aa06-3a0d3986e885@mrabarnett.plus.com> Message-ID: On Sat, Mar 10, 2018 at 2:45 PM, Chris Barker wrote: > On Sat, Mar 10, 2018 at 3:24 AM, MRAB wrote: >> >> On 2018-03-10 01:15, Guido van Rossum wrote: >>> >>> Yes, you can use the |= operator instead. >>> >> |= is not quite the same as .update because it rebinds, > > > isn't that an "in-place operator" i.e. if it's a mutable object it should > mutate rather than rebinding? Normally on a mutable object, |= will mutate the object in-place AND ALSO rebind the name to the same object it started with (like writing 'x = x'). The latter part raises an error if 'x' is not a legal target for assignment. -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Mon Mar 12 09:45:42 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 12 Mar 2018 23:45:42 +1000 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) In-Reply-To: References: Message-ID: On 11 March 2018 at 01:35, Tom Tervoort wrote: > 32 > 16, so what's the problem? > ------------------------------- > > First of all, I'd like to point out that the three examples in the docs > (https://docs.python.org/3.6/library/secrets.html#generating-tokens) > explicitly generate 16-byte tokens. If the current > default is the most secure solution, why deviate from it here? > 1. For readability (as you say, shorter tokens *are* easier to read) 2. For independence from the default length setting > I think there a quite some situations where unnecessarily long codes can > cause usability problems: when using a system > where users have to manually type in a random code (I frequently have to > do that when using the password manager on my > phone, for example) it's nice if you can save half of the time they have > to spend on that. Shorter codes can also be > converted to smaller QR codes, and to nicer URLs. > > Sure, a user can always choose to just pick a value for nbytes and set > their own length. However, if they want to pick > a secure value, they are being tasked with considering the implications. A > user should be able to rely on the library > picking a good secure default. > I think security functionality should be made as ergonomic as possible, > without compromising security. When security > features unnecessarily create usability issues, people are discouraged > from using them in the first place. > "32 bytes is the conservative default, while 16 bytes is a common explicit override to improve token readability with little or no reduction in pragmatic security" is still pretty easy to use. With the API token generation examples already explicitly specifying 16 bytes, the only change needed to get to that state would be to amend https://docs.python.org/3/library/secrets.html#how-many-bytes-should-tokens-use to mention the readability question, and the fact that 16 is a reasonable choice in most cases. With the current setup, the ideal adoption cycle will see someone going: * It works, yay! * The tokens are a bit hard to read though, how do I improve that? * OK, I don't need the super-conservative 32 bytes, 16 bytes is fine, so I'll pass that in explicitly > > So yeah, that was my argument for making tokens 16 bytes long instead of > 32. Please let me know if and why I am > completely wrong:P > I don't think you're completely wrong, and I suspect if anyone had gone through this level of detailed analysis prior to 3.6, we might have made the default 128 bits of entropy instead of 256. As it is though, I think you've made the case for a docs change to make it explicit that 16 bytes of entropy is almost certainly going to be fine for practical purposes (which will benefit all users of 3.6+), but not for a reduction in the actual default (which would require invoking the "Hey, we said we might change it at any time!" caveat on the default length, and we included that caveat because we weren't sure of the potential future security implications of quantum computing, not because 32 byte tokens are harder to read than 16 byte ones) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Mon Mar 12 10:04:32 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Mar 2018 15:04:32 +0100 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) References: Message-ID: <20180312150432.4a6bebbb@fsol> On Sat, 10 Mar 2018 16:35:47 +0100 Tom Tervoort wrote: > > I think there a quite some situations where unnecessarily long codes can > cause usability problems: when using a system > where users have to manually type in a random code (I frequently have to do > that when using the password manager on my > phone, for example) it's nice if you can save half of the time they have to > spend on that. Shorter codes can also be > converted to smaller QR codes, and to nicer URLs. If you want shorter codes for specific scenarios then it's your responsibility (as an application developer) to adapt the token width *and* ensure that the chosen code length is still non-vulnerable. I think defaulting to 32 bytes for the Python standard library is good as: 1) it's more future-proof, even in the face of algorithm weaknesses which may make available better-than-brute-force methods in the future 2) it teaches developers the value of having sufficient entropy in tokens Also I disagree with the claim that 16 bytes is somehow better for usability. It's still a terribly long random string to type by hand and I hope nobody is inflicting that on their users. (I'm not sure what "smaller QR code" means. Given a QR code is basically a computer analysis-friendly glyph that you show your phone or other device to perform pattern recognition on, why does it matter whether the QR code is "small" or not?) > A user should be able to rely on the library > picking a good secure default. And that's exactly what the library does, apparently! Regards Antoine. From rosuav at gmail.com Mon Mar 12 10:07:50 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Mar 2018 01:07:50 +1100 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) In-Reply-To: References: Message-ID: On Tue, Mar 13, 2018 at 12:45 AM, Nick Coghlan wrote: > As it is though, I think you've made the case for a docs change to make it > explicit that 16 bytes of entropy is almost certainly going to be fine for > practical purposes (which will benefit all users of 3.6+), but not for a > reduction in the actual default (which would require invoking the "Hey, we > said we might change it at any time!" caveat on the default length, and we > included that caveat because we weren't sure of the potential future > security implications of quantum computing, not because 32 byte tokens are > harder to read than 16 byte ones) +1 on the docs change. Is there value in exposing a couple of guiding constants secrets.DEFAULT_ENTROPY and secrets.MINIMUM_ENTROPY ? The former is already present (and would simply be documented as public), and the latter would be "whatever seems like a basic minimum", such that MIN <= DEFAULT, and the two of them could be increased at any time (together or independently). For applications where readability and practicality are more important than absolute best quality security, secrets.token_hex(secrets.MINIMUM_ENTROPY) would be future-proofed while still being shorter than secrets.token_hex(). Unlike the pure docs change, this would only benefit users of 3.8+ (or maybe 3.7+ if this is considered a trivial change), but it'd give a way to avoid hardcoding a length that might later be considered weak. ChrisA From rosuav at gmail.com Mon Mar 12 10:10:33 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Mar 2018 01:10:33 +1100 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) In-Reply-To: <20180312150432.4a6bebbb@fsol> References: <20180312150432.4a6bebbb@fsol> Message-ID: On Tue, Mar 13, 2018 at 1:04 AM, Antoine Pitrou wrote: > (I'm not sure what "smaller QR code" means. Given a QR code is > basically a computer analysis-friendly glyph that you show your phone > or other device to perform pattern recognition on, why does it matter > whether the QR code is "small" or not?) A QR code encoding more data requires finer resolution at the same size. That means the camera needs to be closer to it, all else being equal. If you keep stuffing more data into it, eventually the QR code ends up looking like the patented Monitor Grit that we do our best to avoid. :) ChrisA From solipsis at pitrou.net Mon Mar 12 11:10:41 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Mar 2018 16:10:41 +0100 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) References: <20180312150432.4a6bebbb@fsol> Message-ID: <20180312161041.6b29dfe6@fsol> On Tue, 13 Mar 2018 01:10:33 +1100 Chris Angelico wrote: > On Tue, Mar 13, 2018 at 1:04 AM, Antoine Pitrou wrote: > > (I'm not sure what "smaller QR code" means. Given a QR code is > > basically a computer analysis-friendly glyph that you show your phone > > or other device to perform pattern recognition on, why does it matter > > whether the QR code is "small" or not?) > > A QR code encoding more data requires finer resolution at the same > size. That means the camera needs to be closer to it, all else being > equal. Is that important here? I would expect the user to be (physically) close to the QR code. It's not like a QR code containing secret credentials will be posted on a wall in a random street or subway station (at least I hope so :-)). Regards Antoine. From rosuav at gmail.com Mon Mar 12 14:03:21 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Mar 2018 05:03:21 +1100 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) In-Reply-To: <20180312161041.6b29dfe6@fsol> References: <20180312150432.4a6bebbb@fsol> <20180312161041.6b29dfe6@fsol> Message-ID: On Tue, Mar 13, 2018 at 2:10 AM, Antoine Pitrou wrote: > On Tue, 13 Mar 2018 01:10:33 +1100 > Chris Angelico wrote: >> On Tue, Mar 13, 2018 at 1:04 AM, Antoine Pitrou wrote: >> > (I'm not sure what "smaller QR code" means. Given a QR code is >> > basically a computer analysis-friendly glyph that you show your phone >> > or other device to perform pattern recognition on, why does it matter >> > whether the QR code is "small" or not?) >> >> A QR code encoding more data requires finer resolution at the same >> size. That means the camera needs to be closer to it, all else being >> equal. > > Is that important here? I would expect the user to be (physically) > close to the QR code. It's not like a QR code containing secret > credentials will be posted on a wall in a random street or subway > station (at least I hope so :-)). Depends what you mean by "secret". Let's suppose you host a video sharing site (we'll call it, say, "me tube") and need to create URLs for videos as they get uploaded. These URLs should be impossible to predict, but easy to share. How long do they need to be? If they're encoded using token_urlsafe (base 64), you get six bits of randomness per character of URL; the default entropy looks like http://metube.example/SoO8IclkLFcfPX2pA7okFHdoSrZjKtrAmDdmFvC2O6Y which is going to make a large and complicated QR code that you have to be very close to. But you don't really need these to be THAT secure. It'd be fine to use token_urlsafe(16) to make something like http://metube.example/9IoJVtQrhic4Xi633mJ7MQ; and our nearest competitor uses even shorter URLs like http://youtu.be/B7xai5u_tnk (about equivalent to token_urlsafe(9)). Let's look at those URLs: 32: http://metube.example/SoO8IclkLFcfPX2pA7okFHdoSrZjKtrAmDdmFvC2O6Y 16: http://metube.example/9IoJVtQrhic4Xi633mJ7MQ 09: http://metube.example/ziCHRKMlr8rX YT: http://youtu.be/B7xai5u_tnk Using the 'secrets' module to generate URLs like this isn't wrong; since these URLs have to be unguessable (you shouldn't be able to type http://metube.example/aaaaac and get someone's secret unlisted video), their identifiers have to be functionally equivalent to session IDs and such. And since advertisers *do* want to put links to their videos onto billboards, QR codes are definitely a thing; and companies won't use metube if its competitor's QR codes can be scanned reliably from two platforms across and ours need to be scanned from right up next to it. As you can see from this analysis, the boundary for "good enough" is incredibly rubbery, but there is definitely value in making shorter URLs. 32: https://chart.googleapis.com/chart?cht=qr&chl=http%3A%2F%2Fmetube.example%2FSoO8IclkLFcfPX2pA7okFHdoSrZjKtrAmDdmFvC2O6Y&chs=180x180&choe=UTF-8&chld=L|2 16: https://chart.googleapis.com/chart?cht=qr&chl=http%3A%2F%2Fmetube.example%2F9IoJVtQrhic4Xi633mJ7MQ&chs=180x180&choe=UTF-8&chld=L|2 09: https://chart.googleapis.com/chart?cht=qr&chl=http%3A%2F%2Fmetube.example%2FziCHRKMlr8rX&chs=180x180&choe=UTF-8&chld=L|2 (and YT: https://chart.googleapis.com/chart?cht=qr&chl=http%3A%2F%2Fyoutu.be%2FB7xai5u_tnk&chs=180x180&choe=UTF-8&chld=L|2 for comparison) The longer the URL, the noisier the image, and thus the nearer you need to be for a reliable scan. ChrisA From solipsis at pitrou.net Mon Mar 12 14:32:55 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Mon, 12 Mar 2018 19:32:55 +0100 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) References: <20180312150432.4a6bebbb@fsol> <20180312161041.6b29dfe6@fsol> Message-ID: <20180312193255.56d1b291@fsol> On Tue, 13 Mar 2018 05:03:21 +1100 Chris Angelico wrote: > > Using the 'secrets' module to generate URLs like this isn't wrong; > since these URLs have to be unguessable (you shouldn't be able to type > http://metube.example/aaaaac and get someone's secret unlisted video), > their identifiers have to be functionally equivalent to session IDs > and such. And since advertisers *do* want to put links to their videos > onto billboards, QR codes are definitely a thing; and companies won't > use metube if its competitor's QR codes can be scanned reliably from > two platforms across and ours need to be scanned from right up next to > it. Yeah. So people building such a platform can use a custom token length. Still, I think it's better to have a future-proof default token length. People will know if they need to shorten it for usability reasons. However, if we default to shorter tokens, people won't know whether they need to ask for a longer length for security reasons. "Secure by default, better usability with a simple parameter tweak" sounds like a sane API guideline. Regards Antoine. From rosuav at gmail.com Mon Mar 12 14:40:11 2018 From: rosuav at gmail.com (Chris Angelico) Date: Tue, 13 Mar 2018 05:40:11 +1100 Subject: [Python-ideas] PEP 506: some thoughts on the output length of token_bytes (and why I think it should be split in half) In-Reply-To: <20180312193255.56d1b291@fsol> References: <20180312150432.4a6bebbb@fsol> <20180312161041.6b29dfe6@fsol> <20180312193255.56d1b291@fsol> Message-ID: On Tue, Mar 13, 2018 at 5:32 AM, Antoine Pitrou wrote: > On Tue, 13 Mar 2018 05:03:21 +1100 > Chris Angelico wrote: >> >> Using the 'secrets' module to generate URLs like this isn't wrong; >> since these URLs have to be unguessable (you shouldn't be able to type >> http://metube.example/aaaaac and get someone's secret unlisted video), >> their identifiers have to be functionally equivalent to session IDs >> and such. And since advertisers *do* want to put links to their videos >> onto billboards, QR codes are definitely a thing; and companies won't >> use metube if its competitor's QR codes can be scanned reliably from >> two platforms across and ours need to be scanned from right up next to >> it. > > Yeah. So people building such a platform can use a custom token > length. Still, I think it's better to have a future-proof default token > length. People will know if they need to shorten it for usability > reasons. However, if we default to shorter tokens, people won't > know whether they need to ask for a longer length for security reasons. > > "Secure by default, better usability with a simple parameter tweak" > sounds like a sane API guideline. Yep, I think we're on the same page here! ChrisA From george at fischhof.hu Mon Mar 12 16:57:32 2018 From: george at fischhof.hu (George Fischhof) Date: Mon, 12 Mar 2018 21:57:32 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations Message-ID: Good day all, as it seemed to be a good idea, I wrote a PEP proposal for pathlib to contain file operations. Here is the draft. What do you think about this? BR, George --------------------------- PEP: 9999 Title: Pathlib Module Should Contain All File Operations Author: George Fischhof Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Mar-2018 Python-Version: 3.8 Post-History: Abstract ======== This PEP proposes pathlib module to be a centralized place for all file-system related operations. Rationale ========= Right now we have several modules that contain functions related to file-system operations mainly the os, pathlib and shutil. For beginners it is quite hard to remember where can he / she find a function (copy resides in shutil, but the remove function can be found in the os module. (And sometimes developers with moderate experience have to check the documentation as well.) After the release of version 3.6 several methods became aware of path-like object. There are only a few ones which does not support the path-like object. After making these methods path-like object aware, these functions could be added to pathlib. With functions in pathlib the developers should not have to think on which method (function) can be found in which module. Makes the life easier. Implementation ============== For compatibility reasons the pathlib should contain wrappers to the original functions. The original functions should remain at their original place. (Or if pathlib contains the function, the original modules should have a wrapper to it.) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Mon Mar 12 17:16:59 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 12 Mar 2018 21:16:59 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: On 12 March 2018 at 20:57, George Fischhof wrote: > Good day all, > > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to > contain file operations. > > Here is the draft. What do you think about this? I don't know for certain what I think I feel about the idea - in general, it seems plausible. But I think you'll need to get down to specifics in the PEP, exactly what functions do you suggest get added to pathlib? Paul From george at fischhof.hu Mon Mar 12 17:56:38 2018 From: george at fischhof.hu (George Fischhof) Date: Mon, 12 Mar 2018 22:56:38 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: 2018-03-12 22:16 GMT+01:00 Paul Moore : > On 12 March 2018 at 20:57, George Fischhof wrote: > > Good day all, > > > > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to > > contain file operations. > > > > Here is the draft. What do you think about this? > > I don't know for certain what I think I feel about the idea - in > general, it seems plausible. But I think you'll need to get down to > specifics in the PEP, exactly what functions do you suggest get added > to pathlib? > > Paul > Basically file and directory operations: copy, rename, move, delete. With all variants. I will collect them, and put intot PEP George -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Mar 13 08:17:36 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 13 Mar 2018 23:17:36 +1100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: <20180313121736.GT18868@ando.pearwood.info> On Mon, Mar 12, 2018 at 09:57:32PM +0100, George Fischhof wrote: > Right now we have several modules that contain functions related > to file-system operations mainly the os, pathlib and shutil. > For beginners it is quite hard to remember where can he / she find > a function (copy resides in shutil, but the remove function can be > found in the os module. (And sometimes developers with moderate > experience have to check the documentation as well.) This is certainly a problem. Not a big problem, but it is an annoyance. > With functions in pathlib the developers should not have to think > on which method (function) can be found in which module. > > Makes the life easier. I don't know that this will be true. It makes one problem better: you no longer have to remember which module the function is in. But it makes other things worse: - code and/or API duplication: for backwards compatibility, every existing function must be in two places, the original and in pathlib; - if new file functions are added, they will go only in pathlib, which makes pathlib effectively mandatory; - the pathlib API becomes even more complicated: not only have you got all the methods of pathlib objects, but you have all the shutil and os functions as well. I think this is a good place for an experiment. You could write a function which monkey-patches pathlib: from pathlib import Path import os import shutil def monkeypatch(): Path.remove = os.remove # etc. Then we can see how many functions are involved, how large this makes the Path object, and try it out and see whether it is better. -- Steve From rhodri at kynesim.co.uk Tue Mar 13 11:35:56 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Tue, 13 Mar 2018 15:35:56 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: <494794ce-29f7-fc90-cc7b-a4c665c90e69@kynesim.co.uk> On 12/03/18 20:57, George Fischhof wrote: > Good day all, > > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to > contain file operations. > > Here is the draft. What do you think about this? I am mildly negative about this. In my copious spare time (ho ho) I have been considering the use of pathlib with things that aren't filing systems, such as http or ftp. Moving the file operations into pathlib would strain that idea even harder than my (extremely superficial) current thinking. -- Rhodri James *-* Kynesim Ltd From mistersheik at gmail.com Tue Mar 13 20:24:41 2018 From: mistersheik at gmail.com (Neil Girdhar) Date: Tue, 13 Mar 2018 17:24:41 -0700 (PDT) Subject: [Python-ideas] Pythonic access protection Message-ID: <8496d049-c96b-476a-9bfd-8fcec08c7ccd@googlegroups.com> Now that Python has type annotations has there been any thought into having access protection annotations. This would be for same reason as type annotations: to give the linter the opportunity to catch bugs early, and to give the programmer a uniform way to specify invariants. Unlike type annotations, this wouldn't need any new syntax, but it should be well thought-out. My rough idea is to have some decorator on methods and properties describing who is allowed to call those methods and access those properties. The allowed caller list could be things base classes. I think this would need to be better thought-out, but something like this would be really useful for me as it's hard to keep track of who is allowed to call what. Best, Neil -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrisjbillington at gmail.com Wed Mar 14 01:20:20 2018 From: chrisjbillington at gmail.com (Chris Billington) Date: Wed, 14 Mar 2018 16:20:20 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names Message-ID: The Python "double import" problem (discussed here ) leads to subtle bugs in code that change depending on what directory the code was run from. It's hard to think of reasons why importing the same filepath multiple times resulting in two copies of a module should be allowed, it almost always indicates a mistake in code. One possible solution I though of is to return the existing module upon import under a different name. Then you would still have two names for the same module, but at least it would be the same module object, not two copies of it loaded separately from the same file. However, then you would have the situation where a modules __name__ attribute would not match at least one of the names it was imported as. Instead, maybe a user should just get a big fat error if they try to import the same file twice under different names. For frozen modules, and other fancy imports, users should be able to do what they like, but for ordinary file-based imports, it's easy enough to detect if a module has been imported under two different names (by inspecting the filepath of the module being loaded and seeing if that filepath has previously been loaded under a different name) and throw an exception. I've recently made an addition to a project of mine to turn this type of detection on all the time so that we don't miss these subtle bugs (we encountered some whilst porting to Python 3 since we started using absolute imports). I wonder if there's any reason something like this shouldn't be built into Python's default import system. -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Mar 14 01:58:28 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 14 Mar 2018 16:58:28 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: Message-ID: <20180314055828.GV18868@ando.pearwood.info> On Wed, Mar 14, 2018 at 04:20:20PM +1100, Chris Billington wrote: > Instead, maybe a user should just get a big fat error if they try to import > the same file twice under different names. Absolutely not. Suppose I import a library, Spam, which does "import numpy". Now I try to "import numpy as np", and I get an error. Besides, there is no reason why I shouldn't be able to import Spam import Spam as spam import Spam as Ham *even in the same script* if I so choose. It's not an error or a mistake to have multiple names for a single object. > I wonder if there's any reason something like this shouldn't be built into > Python's default import system. Wrong question. The question should not be "Why shouldn't we do this?" but "why should we do this?". -- Steve From rosuav at gmail.com Wed Mar 14 02:06:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Wed, 14 Mar 2018 17:06:02 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: <20180314055828.GV18868@ando.pearwood.info> References: <20180314055828.GV18868@ando.pearwood.info> Message-ID: On Wed, Mar 14, 2018 at 4:58 PM, Steven D'Aprano wrote: > On Wed, Mar 14, 2018 at 04:20:20PM +1100, Chris Billington wrote: > >> Instead, maybe a user should just get a big fat error if they try to import >> the same file twice under different names. > > Absolutely not. > > Suppose I import a library, Spam, which does "import numpy". > > Now I try to "import numpy as np", and I get an error. That's not the same thing. Both of those statements are importing the same file under the same name, "numpy"; one of them then assigns that to a different local name. But in sys.modules, they're the exact same thing. The double import problem comes when the same file gets imported under two different names *in sys.modules*. Everything else isn't a problem, because you get the same module object - if you "import numpy; import numpy as np; assert np is numpy", you're not seeing a double import problem. ChrisA From chrisjbillington at gmail.com Wed Mar 14 02:20:41 2018 From: chrisjbillington at gmail.com (Chris Billington) Date: Wed, 14 Mar 2018 17:20:41 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: <20180314055828.GV18868@ando.pearwood.info> Message-ID: Exactly. The example in the "if __name__ == '__main__'" block of my module I linked imports numpy as np, and then adds the numpy package directory to sys.path and imports linalg. This is detected as numpy.linalg being imported twice, once as numpy.linalg, and once as linalg. The output error is below. As for "why should we do this", well, it helps prevent bugs that are pretty hard to notice and debug, so that's a plus. I can't think of any other pluses, so it's down to thinking of minuses in order to see if it's a good idea on-net. Traceback (most recent call last): File "double_import_denier.py", line 153, in _raise_error exec('raise RuntimeError(msg) from None') File "", line 1, in RuntimeError: Double import! The same file has been imported under two different names, resulting in two copies of the module. This is almost certainly a mistake. If you are running a script from within a package and want to import another submodule of that package, import it by its full path: 'import module.submodule' instead of just 'import submodule.' Path imported: /home/bilbo/anaconda3/lib/python3.6/site-packages/numpy/ linalg Traceback (first time imported, as numpy.linalg): ------------ File "double_import_denier.py", line 195, in test1() File "double_import_denier.py", line 185, in test1 import numpy as np File "/home/bilbo/anaconda3/lib/python3.6/site-packages/numpy/__init__.py", line 142, in from . import add_newdocs File "/home/bilbo/anaconda3/lib/python3.6/site-packages/numpy/add_newdocs.py", line 13, in from numpy.lib import add_newdoc File "/home/bilbo/anaconda3/lib/python3.6/site-packages/numpy/lib/__init__.py", line 19, in from .polynomial import * File "/home/bilbo/anaconda3/lib/python3.6/site-packages/numpy/lib/polynomial.py", line 20, in from numpy.linalg import eigvals, lstsq, inv ------------ Traceback (second time imported, as linalg): ------------ File "double_import_denier.py", line 195, in test1() File "double_import_denier.py", line 188, in test1 import linalg ------------ On Wed, Mar 14, 2018 at 5:06 PM, Chris Angelico wrote: > On Wed, Mar 14, 2018 at 4:58 PM, Steven D'Aprano > wrote: > > On Wed, Mar 14, 2018 at 04:20:20PM +1100, Chris Billington wrote: > > > >> Instead, maybe a user should just get a big fat error if they try to > import > >> the same file twice under different names. > > > > Absolutely not. > > > > Suppose I import a library, Spam, which does "import numpy". > > > > Now I try to "import numpy as np", and I get an error. > > That's not the same thing. Both of those statements are importing the > same file under the same name, "numpy"; one of them then assigns that > to a different local name. But in sys.modules, they're the exact same > thing. > > The double import problem comes when the same file gets imported under > two different names *in sys.modules*. Everything else isn't a problem, > because you get the same module object - if you "import numpy; import > numpy as np; assert np is numpy", you're not seeing a double import > problem. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Wed Mar 14 07:18:08 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 14 Mar 2018 22:18:08 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: <20180314055828.GV18868@ando.pearwood.info> Message-ID: <20180314111808.GW18868@ando.pearwood.info> On Wed, Mar 14, 2018 at 05:06:02PM +1100, Chris Angelico wrote: > On Wed, Mar 14, 2018 at 4:58 PM, Steven D'Aprano wrote: > > On Wed, Mar 14, 2018 at 04:20:20PM +1100, Chris Billington wrote: > > > >> Instead, maybe a user should just get a big fat error if they try to import > >> the same file twice under different names. > > > > Absolutely not. > > > > Suppose I import a library, Spam, which does "import numpy". > > > > Now I try to "import numpy as np", and I get an error. > > That's not the same thing. Both of those statements are importing the > same file under the same name, "numpy"; one of them then assigns that > to a different local name. Hence, two different names. > But in sys.modules, they're the exact same thing. Of course they are. But it wasn't clear to me that the alternative was what Chris was referring to. Either I read carelessly, or he never mentioned anything about duplicate entries in sys.modules. > The double import problem comes when the same file gets imported under > two different names *in sys.modules*. It actually requires more than that to cause an actual problem. For starters, merely having two keys in sys.modules isn't a problem if they both refer to the same module object: sys.modules['maths'] = sys.modules['math'] is harmless. Even if you have imported two distinct copies of the same logical module as separate module objects -- which is hardly something you can do by accident, apart from one special case -- it won't necessarily be a problem. For instance, if the module consists of nothing but pure functions with no state, then the worst you have is a redundant copy and some wasted memory. It can even be useful, e.g. I have a module that uses global variables (I know, I know, "global variables considered harmful"...) and sometimes it is useful to import it twice as two independent copies. That's better than literally duplicating the .py file, and faster than re-writing the module and changing the scripts that rely on it. On Linux, I can make a hard-link of spam.py as ham.py. But if my file system doesn't support hard-linking, or I can't do that for some other reason, the next best thing is to intentionally subvert the import system and/or sys.modules in order to get two distinct copies. import spam sys.modules['ham'] = sys.modules['spam'] del sys.modules['spam'] import spam, ham will do it. (I don't know if there are any easier ways.) There's nothing wrong with doing this intentionally. Consenting adults and all that. So the question is, how can you do this *by accident*? The only way I know of to get a module *accidentally* imported twice is when a module imports itself in a script: # spam.py if __name__ == '__main__': import spam does not do what you expect. Now your script is loaded as two independent copies, once under 'spam' and once under '__main__'. The simple fix is: Unless you know what you are doing, don't have your runnable scripts import themselves when running. Apart from intentionally manipulating sys.modules or the import system, or playing file system tricks like hard-linking your files, under what circumstances can this occur by accident? -- Steve From chrisjbillington at gmail.com Wed Mar 14 07:33:18 2018 From: chrisjbillington at gmail.com (Chris Billington) Date: Wed, 14 Mar 2018 22:33:18 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: <20180314111808.GW18868@ando.pearwood.info> References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> Message-ID: The (perhaps minor) problem with simply having the same module object have two entries in sys.modules is that at least one of them will have a different __name__ attribute than the corresponding key in sys.modules. It's not the end of the world (os.path.__name__ is 'posixpath' on linux, not 'os.path', for example), but it could nonetheless trip some code up. Multiple copies of a module even when they are stateless can trip things up. The bug that bit us recently was that a function was checking if its arguments was an instance of a particular class defined in the same module as that function. However the argument had been instantiated using the other copy of the module, and hence as far as Python was concerned, the two classes were different classes and isinstance() returned False. It was very confusing. The problem occurs by accident primarily when you run scripts as __main__ from within a package directory, because the current working directory is in sys.path. I know I know, you're not supposed to do this. But this double import problem is exactly *why* you're not supposed to do this, and despite the advice people do it all the time: for example they might store scripts within a package directory to be run by calling code using subprocess.Popen, or manually by a developer to generate resources, or they might be in the process of turning their pile of scripts into a package, and encounter the problem during the transition whilst they are still running from within the directory. Or they might have had their program os.chdir() into the package directory in order to be able to use relative paths for resources the program needs to load. On Wed, Mar 14, 2018 at 10:18 PM, Steven D'Aprano wrote: > On Wed, Mar 14, 2018 at 05:06:02PM +1100, Chris Angelico wrote: > > On Wed, Mar 14, 2018 at 4:58 PM, Steven D'Aprano > wrote: > > > On Wed, Mar 14, 2018 at 04:20:20PM +1100, Chris Billington wrote: > > > > > >> Instead, maybe a user should just get a big fat error if they try to > import > > >> the same file twice under different names. > > > > > > Absolutely not. > > > > > > Suppose I import a library, Spam, which does "import numpy". > > > > > > Now I try to "import numpy as np", and I get an error. > > > > That's not the same thing. Both of those statements are importing the > > same file under the same name, "numpy"; one of them then assigns that > > to a different local name. > > Hence, two different names. > > > But in sys.modules, they're the exact same thing. > > Of course they are. But it wasn't clear to me that the alternative was > what Chris was referring to. Either I read carelessly, or he never > mentioned anything > about duplicate entries in sys.modules. > > > The double import problem comes when the same file gets imported under > > two different names *in sys.modules*. > > It actually requires more than that to cause an actual problem. > > For starters, merely having two keys in sys.modules isn't a problem if > they both refer to the same module object: > > sys.modules['maths'] = sys.modules['math'] > > is harmless. Even if you have imported two distinct copies of the same > logical module as separate module objects -- which is hardly something > you can do by accident, apart from one special case -- it won't > necessarily be a problem. > > For instance, if the module consists of nothing but pure functions with > no state, then the worst you have is a redundant copy and some wasted > memory. > > It can even be useful, e.g. I have a module that uses global variables > (I know, I know, "global variables considered harmful"...) and sometimes > it is useful to import it twice as two independent copies. > > That's better than literally duplicating the .py file, and faster than > re-writing the module and changing the scripts that rely on it. > > On Linux, I can make a hard-link of spam.py as ham.py. But if my file > system doesn't support hard-linking, or I can't do that for some other > reason, the next best thing is to intentionally subvert the import > system and/or sys.modules in order to get two distinct copies. > > import spam > sys.modules['ham'] = sys.modules['spam'] > del sys.modules['spam'] > import spam, ham > > will do it. (I don't know if there are any easier ways.) There's nothing > wrong with doing this intentionally. Consenting adults and all that. > > So the question is, how can you do this *by accident*? > > The only way I know of to get a module *accidentally* imported > twice is when a module imports itself in a script: > > # spam.py > if __name__ == '__main__': > import spam > > > does not do what you expect. Now your script is loaded as two > independent copies, once under 'spam' and once under '__main__'. > > The simple fix is: > > Unless you know what you are doing, don't have your runnable scripts > import themselves when running. > > Apart from intentionally manipulating sys.modules or the import system, > or playing file system tricks like hard-linking your files, under what > circumstances can this occur by accident? > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From facundobatista at gmail.com Wed Mar 14 08:18:30 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 14 Mar 2018 09:18:30 -0300 Subject: [Python-ideas] Descouraging the implicit string concatenation Message-ID: Hello! What would you think about formally descouraging the following idiom? long_string = ( "some part of the string " "with more words, actually is the same " "string that the compiler puts together") We should write the following, instead: long_string = ( "some part of the string " + "with more words, actually is the same " + "string that the compiler puts together") I know that "no change to Python itself" is needed, but having a formal discouragement of the idiom will help in avoiding people to fall in mistakes like: fruits = { "apple", "orange" "banana", "melon", } (and even making the static analysers, like pyflakes or pylint, to show that as a warning) Note that there's no penalty in adding the '+' between the strings, those are resolved at compilation time. Thanks!! -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From fiskomaten at gmail.com Wed Mar 14 08:43:53 2018 From: fiskomaten at gmail.com (=?UTF-8?B?U8O4cmVuIFBpbGfDpXJk?=) Date: Wed, 14 Mar 2018 13:43:53 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: On Wed, Mar 14, 2018 at 1:18 PM, Facundo Batista wrote: > Hello! > > What would you think about formally descouraging the following idiom? > > long_string = ( > "some part of the string " > "with more words, actually is the same " > "string that the compiler puts together") > > We should write the following, instead: > > long_string = ( > "some part of the string " + > "with more words, actually is the same " + > "string that the compiler puts together") > > I know that "no change to Python itself" is needed, but having a > formal discouragement of the idiom will help in avoiding people to > fall in mistakes like: > > fruits = { > "apple", > "orange" > "banana", > "melon", > } > > (and even making the static analysers, like pyflakes or pylint, to > show that as a warning) > > Note that there's no penalty in adding the '+' between the strings, > those are resolved at compilation time. > > Thanks!! > I agree that implicit concatenation is a bad feature. I have seen it cause multiple stupid errors while I have never actually used it for anything useful. I have also experienced beginners asking why you can do `x = "abc" "def"` but not `a = "abc"; b = "def"; x = a b` and then you have to either explain them the differences between strings and string literals or just tell them to always use `+`. I think it breaks multiple parts of the zen of python: Explicit is better than implicit. Special cases aren't special enough to break the rules. Errors should never pass silently. There should be one-- and preferably only one --obvious way to do it. From eryksun at gmail.com Wed Mar 14 08:56:43 2018 From: eryksun at gmail.com (eryk sun) Date: Wed, 14 Mar 2018 12:56:43 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: On Wed, Mar 14, 2018 at 12:18 PM, Facundo Batista wrote: > > Note that there's no penalty in adding the '+' between the strings, > those are resolved at compilation time. The above statement is not true for versions prior to 3.7. Previously the addition of string literals was optimized by the peephole optimizer, with a limit of 20 characters. Do you mean to formally discourage implicit string-literal concatenation only for 3.7+? From steve at pearwood.info Wed Mar 14 09:15:52 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Mar 2018 00:15:52 +1100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: <20180314131552.GX18868@ando.pearwood.info> On Wed, Mar 14, 2018 at 09:18:30AM -0300, Facundo Batista wrote: > Hello! > > What would you think about formally descouraging the following idiom? > > long_string = ( > "some part of the string " > "with more words, actually is the same " > "string that the compiler puts together") I would hate that. > We should write the following, instead: > > long_string = ( > "some part of the string " + > "with more words, actually is the same " + > "string that the compiler puts together") Should we? I disagree. Of course you're welcome to specify that in your own style-guide for your own code, but I won't be following that recommendation. > I know that "no change to Python itself" is needed, but having a > formal discouragement of the idiom will help in avoiding people to > fall in mistakes like: > > fruits = { > "apple", > "orange" > "banana", > "melon", > } People can make all sorts of mistakes through carlessness. I wrote {y, x*3} the other day instead of {y: x**3}. (That's *two* errors in one simple expression. I wish I could say it was a record for me.) Should we "discourage" exponentiation and dict displays and insist on writing dict((y, x*x*x)) to avoid the risk of errors? I don't think so. I think string concatenation falls into the same category. Sometimes even the most careful writer makes a mistake (let alone careless writers). That's life. Not every problem needs a technical "solution". Sometimes the right solution is to proof-read your code, or have code review by a fresh pair of eyes. And tests, of course. [...] > Note that there's no penalty in adding the '+' between the strings, > those are resolved at compilation time. That is an implementation feature, not a language requirement. Not all Python interpreters will do that, and they are free to put limits on how much they optimize. Here's Python 3.5: py> import dis py> dis.dis('s = "abcdefghijklmnopqrs" + "t"') 1 0 LOAD_CONST 3 ('abcdefghijklmnopqrst') 3 STORE_NAME 0 (s) 6 LOAD_CONST 2 (None) 9 RETURN_VALUE But now see this: py> dis.dis('s = "abcdefghijklmnopqrs" + "tu"') 1 0 LOAD_CONST 0 ('abcdefghijklmnopqrs') 3 LOAD_CONST 1 ('tu') 6 BINARY_ADD 7 STORE_NAME 0 (s) 10 LOAD_CONST 2 (None) 13 RETURN_VALUE And older versions of CPython didn't optimize this at all, and some day there could be a command-line switch or environment variable to turn these optimizations off. With string concatentation having potential O(N**2) performance, if you're writing explicit string concatenations with +, they could potentially be *very* expensive at runtime. -- Steve From steve at pearwood.info Wed Mar 14 09:26:17 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Mar 2018 00:26:17 +1100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: <20180314132617.GY18868@ando.pearwood.info> On Wed, Mar 14, 2018 at 01:43:53PM +0100, S?ren Pilg?rd wrote: [...] > I have also experienced beginners asking why you can do `x = "abc" > "def"` but not `a = "abc"; b = "def"; x = a b` and then you have to > either explain them the differences between strings and string > literals or just tell them to always use `+`. If you tell them to "always use `+`", you are teaching them to be cargo-cult coders who write code they don't understand for reasons they don't know. Life is too short to learn *everything*, I know that, and I've certainly copied lots of code I don't understand (and hoped I'd never need to debug it!). If that makes me a cargo-cult coder too, so be it. But never over something as simple as the difference between names a and b, and string literals "abc", "def". -- Steve From jsbueno at python.org.br Wed Mar 14 09:32:50 2018 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Wed, 14 Mar 2018 10:32:50 -0300 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314131552.GX18868@ando.pearwood.info> References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: While I personally dislike implicit string concatenation, I've worked in projects that use it, and I, more often than I'd like, use them myself. The problem is that there is no "less typing" way of entering a correct multi-line string - due to multiline string literals preserving the indentation characters. It is a bit cumbersome to do something like from textwrap import dedent as D a = D("""\ My code-aligned, multiline text with no spaces to the left goes here """) And people then resort for implicit concatenation instead of that. I am sure this has been discussed before, and decided against, but I'd be all for taking the occasion to add a "D" prefix to multiline strings to apply de-denting to the literals. Regards, js -><- On 14 March 2018 at 10:15, Steven D'Aprano wrote: > On Wed, Mar 14, 2018 at 09:18:30AM -0300, Facundo Batista wrote: >> Hello! >> >> What would you think about formally descouraging the following idiom? >> >> long_string = ( >> "some part of the string " >> "with more words, actually is the same " >> "string that the compiler puts together") > > I would hate that. > > >> We should write the following, instead: >> >> long_string = ( >> "some part of the string " + >> "with more words, actually is the same " + >> "string that the compiler puts together") > > Should we? I disagree. > > Of course you're welcome to specify that in your own style-guide for > your own code, but I won't be following that recommendation. > > >> I know that "no change to Python itself" is needed, but having a >> formal discouragement of the idiom will help in avoiding people to >> fall in mistakes like: >> >> fruits = { >> "apple", >> "orange" >> "banana", >> "melon", >> } > > People can make all sorts of mistakes through carlessness. I wrote > > {y, x*3} > > the other day instead of {y: x**3}. (That's *two* errors in one simple > expression. I wish I could say it was a record for me.) Should we > "discourage" exponentiation and dict displays and insist on writing > dict((y, x*x*x)) to avoid the risk of errors? I don't think so. > > I think string concatenation falls into the same category. Sometimes > even the most careful writer makes a mistake (let alone careless > writers). That's life. Not every problem needs a technical "solution". > Sometimes the right solution is to proof-read your code, or have code > review by a fresh pair of eyes. > > And tests, of course. > > > [...] >> Note that there's no penalty in adding the '+' between the strings, >> those are resolved at compilation time. > > That is an implementation feature, not a language requirement. Not all > Python interpreters will do that, and they are free to put limits on > how much they optimize. Here's Python 3.5: > > py> import dis > py> dis.dis('s = "abcdefghijklmnopqrs" + "t"') > 1 0 LOAD_CONST 3 ('abcdefghijklmnopqrst') > 3 STORE_NAME 0 (s) > 6 LOAD_CONST 2 (None) > 9 RETURN_VALUE > > But now see this: > > py> dis.dis('s = "abcdefghijklmnopqrs" + "tu"') > 1 0 LOAD_CONST 0 ('abcdefghijklmnopqrs') > 3 LOAD_CONST 1 ('tu') > 6 BINARY_ADD > 7 STORE_NAME 0 (s) > 10 LOAD_CONST 2 (None) > 13 RETURN_VALUE > > And older versions of CPython didn't optimize this at all, and some day > there could be a command-line switch or environment variable to turn > these optimizations off. > > With string concatentation having potential O(N**2) performance, if > you're writing explicit string concatenations with +, they could > potentially be *very* expensive at runtime. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From fiskomaten at gmail.com Wed Mar 14 09:40:47 2018 From: fiskomaten at gmail.com (=?UTF-8?B?U8O4cmVuIFBpbGfDpXJk?=) Date: Wed, 14 Mar 2018 14:40:47 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314131552.GX18868@ando.pearwood.info> References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: On Wed, Mar 14, 2018 at 2:15 PM, Steven D'Aprano wrote: > On Wed, Mar 14, 2018 at 09:18:30AM -0300, Facundo Batista wrote: >> Hello! >> >> What would you think about formally descouraging the following idiom? >> >> long_string = ( >> "some part of the string " >> "with more words, actually is the same " >> "string that the compiler puts together") > > I would hate that. > > >> We should write the following, instead: >> >> long_string = ( >> "some part of the string " + >> "with more words, actually is the same " + >> "string that the compiler puts together") > > Should we? I disagree. > > Of course you're welcome to specify that in your own style-guide for > your own code, but I won't be following that recommendation. > > >> I know that "no change to Python itself" is needed, but having a >> formal discouragement of the idiom will help in avoiding people to >> fall in mistakes like: >> >> fruits = { >> "apple", >> "orange" >> "banana", >> "melon", >> } > > People can make all sorts of mistakes through carlessness. I wrote > > {y, x*3} > > the other day instead of {y: x**3}. (That's *two* errors in one simple > expression. I wish I could say it was a record for me.) Should we > "discourage" exponentiation and dict displays and insist on writing > dict((y, x*x*x)) to avoid the risk of errors? I don't think so. > > I think string concatenation falls into the same category. Sometimes > even the most careful writer makes a mistake (let alone careless > writers). That's life. Not every problem needs a technical "solution". > Sometimes the right solution is to proof-read your code, or have code > review by a fresh pair of eyes. > > And tests, of course. > > > [...] >> Note that there's no penalty in adding the '+' between the strings, >> those are resolved at compilation time. > > That is an implementation feature, not a language requirement. Not all > Python interpreters will do that, and they are free to put limits on > how much they optimize. Here's Python 3.5: > > py> import dis > py> dis.dis('s = "abcdefghijklmnopqrs" + "t"') > 1 0 LOAD_CONST 3 ('abcdefghijklmnopqrst') > 3 STORE_NAME 0 (s) > 6 LOAD_CONST 2 (None) > 9 RETURN_VALUE > > But now see this: > > py> dis.dis('s = "abcdefghijklmnopqrs" + "tu"') > 1 0 LOAD_CONST 0 ('abcdefghijklmnopqrs') > 3 LOAD_CONST 1 ('tu') > 6 BINARY_ADD > 7 STORE_NAME 0 (s) > 10 LOAD_CONST 2 (None) > 13 RETURN_VALUE > > And older versions of CPython didn't optimize this at all, and some day > there could be a command-line switch or environment variable to turn > these optimizations off. > > With string concatentation having potential O(N**2) performance, if > you're writing explicit string concatenations with +, they could > potentially be *very* expensive at runtime. > > > Of course you can always make error, even in a single letter. But I think there is a big difference between mixing up +-/* and ** where the operator is in "focus" and the implicit concatenation where there is no operator. A common problem is that you have something like foo(["a", "b", "c" ]).bar() but then you remember that there also needs to be a "d" so you just add foo(["a", "b", "c" "d" ]).bar() Causing an error, not with the "d" expression you are working on but due to what you thought was the previous expression but python turns it into one. The , is seen as a delimiter by the programmer not as part of the operation (or the lack of the ,). We can't remove all potential pitfalls, but I do think there is value in evaluating whether something has bigger potential to cause harm than the benefits it brings, especially if there are other ways to do the same. From fiskomaten at gmail.com Wed Mar 14 09:48:52 2018 From: fiskomaten at gmail.com (=?UTF-8?B?U8O4cmVuIFBpbGfDpXJk?=) Date: Wed, 14 Mar 2018 14:48:52 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314132617.GY18868@ando.pearwood.info> References: <20180314132617.GY18868@ando.pearwood.info> Message-ID: On Wed, Mar 14, 2018 at 2:26 PM, Steven D'Aprano wrote: > On Wed, Mar 14, 2018 at 01:43:53PM +0100, S?ren Pilg?rd wrote: > > [...] >> I have also experienced beginners asking why you can do `x = "abc" >> "def"` but not `a = "abc"; b = "def"; x = a b` and then you have to >> either explain them the differences between strings and string >> literals or just tell them to always use `+`. > > If you tell them to "always use `+`", you are teaching them to be > cargo-cult coders who write code they don't understand for reasons they > don't know. > > Life is too short to learn *everything*, I know that, and I've certainly > copied lots of code I don't understand (and hoped I'd never need to > debug it!). If that makes me a cargo-cult coder too, so be it. > > But never over something as simple as the difference between names a and > b, and string literals "abc", "def". > > Yes of course the skilled programmer needs to understand the difference. But I am talking about absolute beginners where python in many regards is an excelent first language. And in time they will probably get an understanding of why there is a difference, but when you still have trouble telling the difference between 5 and "5" then telling them the difference between strings and string litterals will only confuse them more. Trying to teach a person everything at once does not work well, limiting the curriculum and defering some parts for later study is hardly cargo-cult coding. At least not more than all the other things that just work the way they do because thats how it works - from the beginners point of view. From p.f.moore at gmail.com Wed Mar 14 09:54:33 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Mar 2018 13:54:33 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: On 14 March 2018 at 13:40, S?ren Pilg?rd wrote: > We can't remove all potential pitfalls, but I do think there is value > in evaluating whether something has bigger potential to cause harm > than the benefits it brings, especially if there are other ways to do > the same. Certainly. However, in this case opinions differ (I, like Steven, have no problem with implicit string concatenation, and in certain cases prefer it over using explicit addition and relying on the compiler optimising that away for me). Also, we need to consider the significant body of code that would break if this construct were prohibited. Even if we were to agree that the harm caused by implicit concatenation outweighed the benefits, that would *still* not justify making it illegal, unless the net gain could be shown to justify the cost of forcing every project that currently uses implicit concatenation to change their code, debug those changes, make new releases, etc. Paul. From fiskomaten at gmail.com Wed Mar 14 09:57:37 2018 From: fiskomaten at gmail.com (=?UTF-8?B?U8O4cmVuIFBpbGfDpXJk?=) Date: Wed, 14 Mar 2018 14:57:37 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: On Wed, Mar 14, 2018 at 2:54 PM, Paul Moore wrote: > On 14 March 2018 at 13:40, S?ren Pilg?rd wrote: >> We can't remove all potential pitfalls, but I do think there is value >> in evaluating whether something has bigger potential to cause harm >> than the benefits it brings, especially if there are other ways to do >> the same. > > Certainly. However, in this case opinions differ (I, like Steven, have > no problem with implicit string concatenation, and in certain cases > prefer it over using explicit addition and relying on the compiler > optimising that away for me). > > Also, we need to consider the significant body of code that would > break if this construct were prohibited. Even if we were to agree that > the harm caused by implicit concatenation outweighed the benefits, > that would *still* not justify making it illegal, unless the net gain > could be shown to justify the cost of forcing every project that > currently uses implicit concatenation to change their code, debug > those changes, make new releases, etc. > I guess that is why the proposal is to discourage the use, not removing it. From phd at phdru.name Wed Mar 14 09:31:58 2018 From: phd at phdru.name (Oleg Broytman) Date: Wed, 14 Mar 2018 14:31:58 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: <20180314133158.4ldr2usx7je45zpx@phdru.name> Hi! On Wed, Mar 14, 2018 at 09:18:30AM -0300, Facundo Batista wrote: > Hello! > > What would you think about formally descouraging the following idiom? > > long_string = ( > "some part of the string " > "with more words, actually is the same " > "string that the compiler puts together") > > We should write the following, instead: > > long_string = ( > "some part of the string " + > "with more words, actually is the same " + > "string that the compiler puts together") > > I know that "no change to Python itself" is needed, but having a > formal discouragement of the idiom will help in avoiding people to > fall in mistakes like: > > fruits = { > "apple", > "orange" > "banana", > "melon", > } I am all for that, with both hands! +1000000, dammit! Once I stumbled over a bug caused by this in legacy code in production. Fixing it was quite painful! The problem with the idiom is it's too easy to make the mistake and rather hard to find. > (and even making the static analysers, like pyflakes or pylint, to > show that as a warning) > > Note that there's no penalty in adding the '+' between the strings, > those are resolved at compilation time. > > Thanks!! > > -- > . Facundo > > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > Twitter: @facundobatista Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From phd at phdru.name Wed Mar 14 10:01:07 2018 From: phd at phdru.name (Oleg Broytman) Date: Wed, 14 Mar 2018 15:01:07 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314131552.GX18868@ando.pearwood.info> References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: <20180314140107.wrq4ke6qkap7zxh4@phdru.name> On Thu, Mar 15, 2018 at 12:15:52AM +1100, Steven D'Aprano wrote: > On Wed, Mar 14, 2018 at 09:18:30AM -0300, Facundo Batista wrote: > > We should write the following, instead: > > > > long_string = ( > > "some part of the string " + > > "with more words, actually is the same " + > > "string that the compiler puts together") > > Should we? I disagree. > > Of course you're welcome to specify that in your own style-guide for > your own code, but I won't be following that recommendation. > > > > I know that "no change to Python itself" is needed, but having a > > formal discouragement of the idiom will help in avoiding people to > > fall in mistakes like: > > > > fruits = { > > "apple", > > "orange" > > "banana", > > "melon", > > } > > People can make all sorts of mistakes through carlessness. I wrote > > {y, x*3} > > the other day instead of {y: x**3}. (That's *two* errors in one simple > expression. I wish I could say it was a record for me.) Should we > "discourage" exponentiation and dict displays and insist on writing > dict((y, x*x*x)) to avoid the risk of errors? I don't think so. We should fix what causes real problems, not convoluted ones. And this particular misfeature caused problems for me. > -- > Steve Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From stephanh42 at gmail.com Wed Mar 14 10:09:28 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Wed, 14 Mar 2018 15:09:28 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314140107.wrq4ke6qkap7zxh4@phdru.name> References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: Note that this has already been proposed and rejected before: https://www.python.org/dev/peps/pep-3126/ Stephan Op 14 mrt. 2018 15:01 schreef "Oleg Broytman" : > On Thu, Mar 15, 2018 at 12:15:52AM +1100, Steven D'Aprano < > steve at pearwood.info> wrote: > > On Wed, Mar 14, 2018 at 09:18:30AM -0300, Facundo Batista wrote: > > > We should write the following, instead: > > > > > > long_string = ( > > > "some part of the string " + > > > "with more words, actually is the same " + > > > "string that the compiler puts together") > > > > Should we? I disagree. > > > > Of course you're welcome to specify that in your own style-guide for > > your own code, but I won't be following that recommendation. > > > > > > > I know that "no change to Python itself" is needed, but having a > > > formal discouragement of the idiom will help in avoiding people to > > > fall in mistakes like: > > > > > > fruits = { > > > "apple", > > > "orange" > > > "banana", > > > "melon", > > > } > > > > People can make all sorts of mistakes through carlessness. I wrote > > > > {y, x*3} > > > > the other day instead of {y: x**3}. (That's *two* errors in one simple > > expression. I wish I could say it was a record for me.) Should we > > "discourage" exponentiation and dict displays and insist on writing > > dict((y, x*x*x)) to avoid the risk of errors? I don't think so. > > We should fix what causes real problems, not convoluted ones. And > this particular misfeature caused problems for me. > > > -- > > Steve > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From phd at phdru.name Wed Mar 14 10:16:42 2018 From: phd at phdru.name (Oleg Broytman) Date: Wed, 14 Mar 2018 15:16:42 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: <20180314141642.d5nob6fnxg57hk7z@phdru.name> On Wed, Mar 14, 2018 at 03:09:28PM +0100, Stephan Houben wrote: > Note that this has already been proposed and rejected before: > > https://www.python.org/dev/peps/pep-3126/ Not the same. The current proposal is to discourage, not remove the misfeature. > Stephan Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From steve at pearwood.info Wed Mar 14 10:18:06 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Mar 2018 01:18:06 +1100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314133158.4ldr2usx7je45zpx@phdru.name> References: <20180314133158.4ldr2usx7je45zpx@phdru.name> Message-ID: <20180314141806.GZ18868@ando.pearwood.info> On Wed, Mar 14, 2018 at 02:31:58PM +0100, Oleg Broytman wrote: [...] > > fruits = { > > "apple", > > "orange" > > "banana", > > "melon", > > } > > I am all for that, with both hands! +1000000, dammit! > > Once I stumbled over a bug caused by this in legacy code in > production. Fixing it was quite painful! Did you mean that *finding* the bug was painful? Fixing it should be trivially easy: add a comma. But honestly, I'm having trouble seeing why it would be painful to find the bug. As soon as you see "orangebanana" where you expected "orange" or "banana", that should give you a very strong clue. But without knowing the details of your code, and how convoluted the problem was, I can't really judge. I can only say that in *my personal experience* finding and fixing these sorts of errors is usually simple. > The problem with the idiom is it's too easy to make the mistake and > rather hard to find. I disagree that its easy to make or hard to find, but I accept that not everyone will have the same experience as me. For those who have this problem, perhaps you should use a linter to check your code and warn about string concatenation. -- Steve From facundobatista at gmail.com Wed Mar 14 10:22:10 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 14 Mar 2018 11:22:10 -0300 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: 2018-03-14 10:54 GMT-03:00 Paul Moore : > Also, we need to consider the significant body of code that would > break if this construct were prohibited. Even if we were to agree that > the harm caused by implicit concatenation outweighed the benefits, > that would *still* not justify making it illegal, unless the net gain > could be shown to justify the cost of forcing every project that > currently uses implicit concatenation to change their code, debug > those changes, make new releases, etc. I never said to prohibit it, just discourage the idiom. -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From facundobatista at gmail.com Wed Mar 14 10:23:25 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 14 Mar 2018 11:23:25 -0300 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: 2018-03-14 11:09 GMT-03:00 Stephan Houben : > Note that this has already been proposed and rejected before: > > https://www.python.org/dev/peps/pep-3126/ What was proposed and reject was the *removal* of the feature/syntax. I propose the discouragement of the idiom. Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From stephanh42 at gmail.com Wed Mar 14 10:30:33 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Wed, 14 Mar 2018 15:30:33 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: Op 14 mrt. 2018 15:23 schreef "Facundo Batista" : I propose the discouragement of the idiom. What does that mean? Stephan Regards, -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Wed Mar 14 10:48:26 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Mar 2018 14:48:26 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: On 14 March 2018 at 14:22, Facundo Batista wrote: > 2018-03-14 10:54 GMT-03:00 Paul Moore : > >> Also, we need to consider the significant body of code that would >> break if this construct were prohibited. Even if we were to agree that >> the harm caused by implicit concatenation outweighed the benefits, >> that would *still* not justify making it illegal, unless the net gain >> could be shown to justify the cost of forcing every project that >> currently uses implicit concatenation to change their code, debug >> those changes, make new releases, etc. > > I never said to prohibit it, just discourage the idiom. Apologies. I misread. How would you propose discouraging the idiom? (That may be why I misread - I can't see any practical way of implementing such a "discouragement"). Paul From robertve92 at gmail.com Wed Mar 14 10:51:22 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Wed, 14 Mar 2018 15:51:22 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: +1 on general idea of discouraging it. > [...] mistakes like: > > fruits = { > "apple", > "orange" > "banana", > "melon", > } +1 > (and even making the static analysers, like pyflakes or pylint, to > show that as a warning) +1 > I agree that implicit concatenation is a bad feature. I have seen it > cause multiple stupid errors while I have never actually used it for > anything useful. [...] Zen of Python. +1. But I think a counter argument would be a real world example. > We should write the following, instead: > > long_string = ( > "some part of the string " + > "with more words, actually is the same " + > "string that the compiler puts together") Maybe the "+" at the beginnig as PEP8 said, to clearly mark it present and explicit. long_string = ( "some part of the string " + "with more words, actually is the same " + "string that the compiler puts together") or long_string = ("" + "some part of the string " + "with more words, actually is the same " + "string that the compiler puts together") > People can make all sorts of mistakes through carlessness. I wrote > {y, x*3} > the other day instead of {y: x**3} Other comments said this isn't really the same kind of error, I agree to them. >> Note that there's no penalty in adding the '+' between the strings, >> those are resolved at compilation time. > That is an implementation feature, not a language requirement. Not all > Python interpreters will do that, and they are free to put limits on > how much they optimize. Here's Python 3.5: > With string concatentation having potential O(N**2) performance, if > you're writing explicit string concatenations with +, they could > potentially be *very* expensive at runtime. I completely agree, by emitting warning if this syntax is used, codes Could be less efficient, but that's also the job of the implementations to be fast. This is the only counter-argument I have to OP, even if I really like the proposal. > Life is too short to learn *everything*, I know that, and I've certainly > copied lots of code I don't understand (and hoped I'd never need to > debug it!). If that makes me a cargo-cult coder too, so be it. I find that learning `x = "e" "f"` does indeed compile is also learning and surprising (especially for beginners). > While I personally dislike implicit string concatenation, > I've worked in projects that use it, and I, more often than > I'd like, use them myself. We would like to see more real world example vs real world mistakes. > It is a bit cumbersome to do something like > from textwrap import dedent as D > a = D("""\ > My code-aligned, multiline text > with no spaces to the left goes here > """) +1. But you should add "\n" to have a = ("hello\n" "world\n" "how") Which would be maybe better written as a = '\n'.join([ "hello", "world", "how"]) (put the spaces where you like) which can easily be switched to ''.join if needed. > A common problem is that you have something like > foo(["a", > "b", > "c" > ]).bar() > but then you remember that there also needs to be a "d" so you just add +1 > Yes of course the skilled programmer needs to understand the > difference. But I am talking about absolute beginners where python in > many regards is an excelent first language. Agreed. 2018-03-14 15:30 GMT+01:00 Stephan Houben : > > > Op 14 mrt. 2018 15:23 schreef "Facundo Batista" : > > > I propose the discouragement of the idiom. > > > > What does that mean? > > Stephan > > > Regards, > > -- > . Facundo > > Blog: http://www.taniquetil.com.ar/plog/ > PyAr: http://www.python.org/ar/ > Twitter: @facundobatista > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From phd at phdru.name Wed Mar 14 10:52:53 2018 From: phd at phdru.name (Oleg Broytman) Date: Wed, 14 Mar 2018 15:52:53 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314141806.GZ18868@ando.pearwood.info> References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> Message-ID: <20180314145253.egtmioqik4dq2shg@phdru.name> On Thu, Mar 15, 2018 at 01:18:06AM +1100, Steven D'Aprano wrote: > On Wed, Mar 14, 2018 at 02:31:58PM +0100, Oleg Broytman wrote: > > Once I stumbled over a bug caused by this in legacy code in > > production. Fixing it was quite painful! > > Did you mean that *finding* the bug was painful? Fixing it should be > trivially easy: add a comma. The most painful was that the bug destroyed important information that was hard to fix later (enterprise code in production, the code is certificated and hard to replace). The second pain was to understand what was going on. When a program report "The OID in the certificate doesn't belong to the proper root OID" it's simpler to decide that the certificate is wrong, not that the list of root OIDs to check is wrong. The third pain was to find the bug. It's hard to spot a problem in the list oids = [ "1.2.6.254.14." "19.9.91.281", "1.2.6.263.12." "481.7.9.6.3.87" "1.2.7.4.214.7." "9.1.52.12", ] There were about 50 OIDs like these, and the missing comma was in the middle of the list. > -- > Steve Oleg. -- Oleg Broytman http://phdru.name/ phd at phdru.name Programmers don't die, they just GOSUB without RETURN. From gvanrossum at gmail.com Wed Mar 14 11:03:02 2018 From: gvanrossum at gmail.com (Guido van Rossum) Date: Wed, 14 Mar 2018 15:03:02 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314145253.egtmioqik4dq2shg@phdru.name> References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: I use the feature regularly for long error messages, and when combined with .format() or % it's not so easy to replace it with a + (it would require adding parentheses). So I am against formally discouraging it or adding something to PEP 8. However linters could warn about lists of comma-separated strings with a missing comma. On Wed, Mar 14, 2018, 07:54 Oleg Broytman wrote: > On Thu, Mar 15, 2018 at 01:18:06AM +1100, Steven D'Aprano < > steve at pearwood.info> wrote: > > On Wed, Mar 14, 2018 at 02:31:58PM +0100, Oleg Broytman wrote: > > > Once I stumbled over a bug caused by this in legacy code in > > > production. Fixing it was quite painful! > > > > Did you mean that *finding* the bug was painful? Fixing it should be > > trivially easy: add a comma. > > The most painful was that the bug destroyed important information > that was hard to fix later (enterprise code in production, the code is > certificated and hard to replace). > > The second pain was to understand what was going on. When a program > report "The OID in the certificate doesn't belong to the proper root > OID" it's simpler to decide that the certificate is wrong, not that the > list of root OIDs to check is wrong. > > The third pain was to find the bug. It's hard to spot a problem in > the list > > oids = [ > "1.2.6.254.14." > "19.9.91.281", > "1.2.6.263.12." > "481.7.9.6.3.87" > "1.2.7.4.214.7." > "9.1.52.12", > ] > > There were about 50 OIDs like these, and the missing comma was in the > middle of the list. > > > -- > > Steve > > Oleg. > -- > Oleg Broytman http://phdru.name/ phd at phdru.name > Programmers don't die, they just GOSUB without RETURN. > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertvandeneynde at hotmail.com Wed Mar 14 11:10:44 2018 From: robertvandeneynde at hotmail.com (Robert Vanden Eynde) Date: Wed, 14 Mar 2018 15:10:44 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: Indeed, linters are the place to go, but I think there is no "official" linter (am I wrong ?), are pyflakes and pylint independant projects ? 2018-03-14 16:03 GMT+01:00 Guido van Rossum : > I use the feature regularly for long error messages, and when combined with > .format() or % it's not so easy to replace it with a + (it would require > adding parentheses). > > So I am against formally discouraging it or adding something to PEP 8. > > However linters could warn about lists of comma-separated strings with a > missing comma. > > On Wed, Mar 14, 2018, 07:54 Oleg Broytman wrote: >> >> On Thu, Mar 15, 2018 at 01:18:06AM +1100, Steven D'Aprano >> wrote: >> > On Wed, Mar 14, 2018 at 02:31:58PM +0100, Oleg Broytman wrote: >> > > Once I stumbled over a bug caused by this in legacy code in >> > > production. Fixing it was quite painful! >> > >> > Did you mean that *finding* the bug was painful? Fixing it should be >> > trivially easy: add a comma. >> >> The most painful was that the bug destroyed important information >> that was hard to fix later (enterprise code in production, the code is >> certificated and hard to replace). >> >> The second pain was to understand what was going on. When a program >> report "The OID in the certificate doesn't belong to the proper root >> OID" it's simpler to decide that the certificate is wrong, not that the >> list of root OIDs to check is wrong. >> >> The third pain was to find the bug. It's hard to spot a problem in >> the list >> >> oids = [ >> "1.2.6.254.14." >> "19.9.91.281", >> "1.2.6.263.12." >> "481.7.9.6.3.87" >> "1.2.7.4.214.7." >> "9.1.52.12", >> ] >> >> There were about 50 OIDs like these, and the missing comma was in the >> middle of the list. >> >> > -- >> > Steve >> >> Oleg. >> -- >> Oleg Broytman http://phdru.name/ phd at phdru.name >> Programmers don't die, they just GOSUB without RETURN. >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From steve at pearwood.info Wed Mar 14 11:33:45 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Mar 2018 02:33:45 +1100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314140107.wrq4ke6qkap7zxh4@phdru.name> References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: <20180314153345.GA18868@ando.pearwood.info> On Wed, Mar 14, 2018 at 03:01:07PM +0100, Oleg Broytman wrote: > > People can make all sorts of mistakes through carlessness. I wrote > > > {y, x*3} > > > > the other day instead of {y: x**3}. (That's *two* errors in one simple > > expression. I wish I could say it was a record for me.) Should we > > "discourage" exponentiation and dict displays and insist on writing > > dict((y, x*x*x)) to avoid the risk of errors? I don't think so. > > We should fix what causes real problems, not convoluted ones. And > this particular misfeature caused problems for me. I don't think that using a single asterisk instead of two is a "convoluted" problem. Please Oleg, if you want others to respect your personal experiences, you should return the favour and respect theirs. Don't try to dismiss them as "convoluted" or not "real". I am far more likely to mistype exponentiation with a single asterisk than to accidentally leave out a comma from a list display of strings. But I recognise that this is *my* problem, not the language. You want some sort of "official" recommendation not to use implicit string concatenation. What benefit will that give you, except to nag those who like and use the feature? - It won't give you a error or warning when your code accidentally uses string concatenation; you still need to be alert for the possibility, or use a linter. - It won't teach people not to use this feature; you still need to point them at the recommendation and have them read it and agree to follow it. - It won't automatically remove string concatenation from the code you use, nor prevent it from happening again, or even change your project style-guide. - It won't even add any information to the community that doesn't already exist. I have no doubt there are many blog posts and Stackoverflow discussions about this. I know I've talked about it many times on comp.lang.python. If you know how to google, you can find a post from Guido talking about getting "a mysterious argument count error" due to this. If you want something to point people at, bookmark Guido's comment. But for that matter, what will this statement say? "Don't use implicit concatenation. Even though this is not deprecated and will remain legal, you mustn't use it because you might use it in a list or function call and forget to use a comma between two string literals and get a mysterious error." Um, yeah, okay, I get lots of mysterious errors caused by carelessness. Some of them are easy to fix. Some are much harder to track down and fix than a missing comma. Do they all need official warnings in the docs? "Don't misspell None as 'none' in three dozen places. And if you do, don't spend twenty minutes fixing them one at a time, use your editor's search and replace function." (That really was not my best day.) I don't think this is any worse or more common than any other gotcha or silly error that might trip people up occasionally, and I think that singling it out in the Python docs for special warning is unnecessary. -- Steve From storchaka at gmail.com Wed Mar 14 11:34:33 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 14 Mar 2018 17:34:33 +0200 Subject: [Python-ideas] Implicit string literal concatenation considered harmful? In-Reply-To: References: Message-ID: 10.05.13 21:48, Guido van Rossum ????: > I just spent a few minutes staring at a bug caused by a missing comma > -- I got a mysterious argument count error because instead of foo('a', > 'b') I had written foo('a' 'b'). > > This is a fairly common mistake, and IIRC at Google we even had a lint > rule against this (there was also a Python dialect used for some > specific purpose where this was explicitly forbidden). > > Now, with modern compiler technology, we can (and in fact do) evaluate > compile-time string literal concatenation with the '+' operator, so > there's really no reason to support 'a' 'b' any more. (The reason was > always rather flimsy; I copied it from C but the reason why it's > needed there doesn't really apply to Python, as it is mostly useful > inside macros.) > > Would it be reasonable to start deprecating this and eventually remove > it from the language? This already was discussed 5 years ago. See the topic "Implicit string literal concatenation considered harmful?" started by GvR. https://mail.python.org/pipermail/python-ideas/2013-May/020527.html First that reviving this discussion please take a look at arguments made at former discussion, and make sure that your arguments are new. From storchaka at gmail.com Wed Mar 14 11:38:49 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 14 Mar 2018 17:38:49 +0200 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: 14.03.18 14:18, Facundo Batista ????: > What would you think about formally descouraging the following idiom? > > long_string = ( > "some part of the string " > "with more words, actually is the same " > "string that the compiler puts together") > > We should write the following, instead: > > long_string = ( > "some part of the string " + > "with more words, actually is the same " + > "string that the compiler puts together") > > I know that "no change to Python itself" is needed, but having a > formal discouragement of the idiom will help in avoiding people to > fall in mistakes like: > > fruits = { > "apple", > "orange" > "banana", > "melon", > } > > (and even making the static analysers, like pyflakes or pylint, to > show that as a warning) > > Note that there's no penalty in adding the '+' between the strings, > those are resolved at compilation time. This already was discussed 5 years ago. See the topic "Implicit string literal concatenation considered harmful?" started by GvR. https://mail.python.org/pipermail/python-ideas/2013-May/020527.html First that reviving this discussion please take a look at arguments made at former discussion, and make sure that your arguments are new. From storchaka at gmail.com Wed Mar 14 11:45:22 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 14 Mar 2018 17:45:22 +0200 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314153345.GA18868@ando.pearwood.info> References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> <20180314153345.GA18868@ando.pearwood.info> Message-ID: 14.03.18 17:33, Steven D'Aprano ????: > Um, yeah, okay, I get lots of mysterious errors caused by carelessness. > Some of them are easy to fix. Some are much harder to track down and fix > than a missing comma. Do they all need official warnings in the docs? > > "Don't misspell None as 'none' in three dozen places. And if > you do, don't spend twenty minutes fixing them one at a time, > use your editor's search and replace function." And don't replace the plane "none", but use regular expressions to avoid replacing "none" in "allow_none" and "nonexisting"! (And there should be a warning about replacing the word "none" in comments and strings). From robertve92 at gmail.com Wed Mar 14 12:00:05 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Wed, 14 Mar 2018 17:00:05 +0100 Subject: [Python-ideas] Adding shallow argument in filecmp.dircmp Message-ID: In module filecmp, the normal functions cmp and cmpfiles do have a shallow attribute. But the dircmp class doesn't provide one, could we add it in the constructor ? In the implementation, it would only change the call to cmpfiles in def phase3 ? It could be useful for performance (same reason, the "shallow" parameter exists in the other two functions), when quickly comparing two directories. From rhodri at kynesim.co.uk Wed Mar 14 12:03:21 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Wed, 14 Mar 2018 16:03:21 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: <5afc9250-fe3f-dffd-c7f7-95dea6d2ff4e@kynesim.co.uk> On 14/03/18 12:18, Facundo Batista wrote: > What would you think about formally descouraging the following idiom? > > long_string = ( > "some part of the string " > "with more words, actually is the same " > "string that the compiler puts together") -1. I use it a fair bit, and prefer it to explicit concatenation. But then I come from C, so I'm used to it. > Note that there's no penalty in adding the '+' between the strings, > those are resolved at compilation time. Is that assertion true? In all Python compilers? I would expect the constant string implicit concatenation to produce single constants efficiently, but explicit concatenation doing the constant folding is a much less sure bet. -- Rhodri James *-* Kynesim Ltd From p.f.moore at gmail.com Wed Mar 14 12:20:46 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Mar 2018 16:20:46 +0000 Subject: [Python-ideas] Implicit string literal concatenation considered harmful? In-Reply-To: References: Message-ID: On 14 March 2018 at 15:34, Serhiy Storchaka wrote: > 10.05.13 21:48, Guido van Rossum ????: >> >> I just spent a few minutes staring at a bug caused by a missing comma >> -- I got a mysterious argument count error because instead of foo('a', >> 'b') I had written foo('a' 'b'). >> >> This is a fairly common mistake, and IIRC at Google we even had a lint >> rule against this (there was also a Python dialect used for some >> specific purpose where this was explicitly forbidden). >> >> Now, with modern compiler technology, we can (and in fact do) evaluate >> compile-time string literal concatenation with the '+' operator, so >> there's really no reason to support 'a' 'b' any more. (The reason was >> always rather flimsy; I copied it from C but the reason why it's >> needed there doesn't really apply to Python, as it is mostly useful >> inside macros.) >> >> Would it be reasonable to start deprecating this and eventually remove >> it from the language? > > > This already was discussed 5 years ago. See the topic "Implicit string > literal concatenation considered harmful?" started by GvR. > > https://mail.python.org/pipermail/python-ideas/2013-May/020527.html > > First that reviving this discussion please take a look at arguments made at > former discussion, and make sure that your arguments are new. To be fair, Guido's post was about removing implicit concatenation, and much of the subsequent thread was only somewhat related (proposals for alternative syntax). And the ultimate conclusion from Guido was that the breakage would be too high. None of which really relates to the proposal here, which is that we should "discourage" use without any language change. I'm just not clear how such discouragement would make any practical difference (I for one would likely just ignore it). Paul From p.f.moore at gmail.com Wed Mar 14 12:29:33 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Wed, 14 Mar 2018 16:29:33 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: On 14 March 2018 at 15:10, Robert Vanden Eynde wrote: > Indeed, linters are the place to go, but I think there is no > "official" linter (am I wrong ?), are pyflakes and pylint independant > projects ? Ironically, many of the places I see implicit concatenation used are where people need to work around linters complaining about line lengths. I understand the benefits of projects that mandate code passing lint checks, but I foresee sequences of commits: "Modify error X to be clearer" "Linter fix - split new message onto 2 lines to avoid long line" "Linter fix - explicitly concatenate with + because implicit concatenation is discouraged" "Linter fix - go back to the old message because the stupid linter complains about unnecessary addition of 2 constant values and I can't be bothered any more" Ok, the last one is a joke - but getting a set of rules that doesn't back the programmer into a corner where it's non-obvious how to achieve a simple goal could easily become a non-trivial task. Paul From facundobatista at gmail.com Wed Mar 14 12:47:53 2018 From: facundobatista at gmail.com (Facundo Batista) Date: Wed, 14 Mar 2018 13:47:53 -0300 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: 2018-03-14 11:30 GMT-03:00 Stephan Houben : > Op 14 mrt. 2018 15:23 schreef "Facundo Batista" : > > I propose the discouragement of the idiom. > > > > What does that mean? That we say "hey, this works but please don't use it, because it tends to a error prone way of writing some code, instead do this". -- . Facundo Blog: http://www.taniquetil.com.ar/plog/ PyAr: http://www.python.org/ar/ Twitter: @facundobatista From mal at egenix.com Wed Mar 14 12:59:39 2018 From: mal at egenix.com (M.-A. Lemburg) Date: Wed, 14 Mar 2018 17:59:39 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: Hi Facundo, On 14.03.2018 17:47, Facundo Batista wrote: > 2018-03-14 11:30 GMT-03:00 Stephan Houben : > >> Op 14 mrt. 2018 15:23 schreef "Facundo Batista" : >> >> I propose the discouragement of the idiom. >> >> >> >> What does that mean? > > That we say "hey, this works but please don't use it, because it tends > to a error prone way of writing some code, instead do this". I believe this falls under coding style and is not something we should actively discourage. It's an idiom that many programming languages share with Python. You may also want to look at this PEP for a longer discussion on the topic: https://www.python.org/dev/peps/pep-3126/ Even pylint rejected a request to have a rule added for this: https://github.com/PyCQA/pylint/issues/1589 FWIW: I use implicit concats a lot and find them very useful for e.g. writing long SQL queries or longer string literals at indented levels where the triple quote approach doesn't work well. -- Marc-Andre Lemburg eGenix.com Professional Python Services directly from the Experts (#1, Mar 14 2018) >>> Python Projects, Coaching and Consulting ... http://www.egenix.com/ >>> Python Database Interfaces ... http://products.egenix.com/ >>> Plone/Zope Database Interfaces ... http://zope.egenix.com/ ________________________________________________________________________ ::: We implement business ideas - efficiently in both time and costs ::: eGenix.com Software, Skills and Services GmbH Pastor-Loeh-Str.48 D-40764 Langenfeld, Germany. CEO Dipl.-Math. Marc-Andre Lemburg Registered at Amtsgericht Duesseldorf: HRB 46611 http://www.egenix.com/company/contact/ http://www.malemburg.com/ From carl at oddbird.net Wed Mar 14 12:53:42 2018 From: carl at oddbird.net (Carl Meyer) Date: Wed, 14 Mar 2018 09:53:42 -0700 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: On 3/14/18 8:03 AM, Guido van Rossum wrote: > I use the feature regularly for long error messages, and when combined > with .format() or % it's not so easy to replace it with a + (it would > require adding parentheses). > > So I am against formally discouraging it or adding something to PEP 8. > > However linters could warn about lists of comma-separated strings with a > missing comma. +1 to all of this. It's possible to write a linter rule that will reliably catch the potential missing-comma errors, while still allowing implicit string concatenation where it's useful and readable. The rule is that implicit string concatenation should only be disallowed if there are any commas present at the same "parenthesization level". Thus, this is allowed: raise SomeException( "This is a really long exception message " "which I've split over two lines." ) While both of these are disallowed: fruits = [ 'orange', 'grape' 'banana', ] some_str.replace( "This is a long string " "split over two lines.", "And this is another string." ) For the latter case, you'd need to wrap the multiline string in its own parens, use +, and/or previously assign it to its own variable. (I don't think this should be controversial; the example is just asking for trouble.) With this rule the only missing-comma that can slip through is when you've forgotten _all_ the intervening commas in your sequence of strings. That's much less likely. Carl From robertve92 at gmail.com Wed Mar 14 13:09:47 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Wed, 14 Mar 2018 17:09:47 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: Le mer. 14 mars 2018 ? 17:29, Paul Moore a ?crit : > Ironically, many of the places I see implicit concatenation used are > where people need to work around linters complaining about line > lengths. I understand the benefits of projects that mandate code > passing lint checks, but I foresee sequences of commits: > > "Modify error X to be clearer" > "Linter fix - split new message onto 2 lines to avoid long line" > "Linter fix - explicitly concatenate with + because implicit > concatenation is discouraged" > "Linter fix - go back to the old message because the stupid linter > complains about unnecessary addition of 2 constant values and I can't > be bothered any more" > Haha, agreed, didn't think of that. In my humble opinion, long lines are less of a problem with raw strings (personal opinion based on html background). -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Wed Mar 14 13:13:17 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Wed, 14 Mar 2018 17:13:17 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: Le mer. 14 mars 2018 ? 18:04, Carl Meyer a ?crit : > On 3/14/18 8:03 AM, Guido van Rossum wrote: > > I use the feature regularly for long error messages, and when combined > > with .format() or % it's not so easy to replace it with a + (it would > > require adding parentheses). > > > > So I am against formally discouraging it or adding something to PEP 8. > > > > However linters could warn about lists of comma-separated strings with a > > missing comma. > > +1 to all of this. > > It's possible to write a linter rule that will reliably catch the > potential missing-comma errors, while still allowing implicit string > concatenation where it's useful and readable. The rule is that implicit > string concatenation should only be disallowed if there are any commas > present at the same "parenthesization level". Thus, this is allowed: > > raise SomeException( > "This is a really long exception message " > "which I've split over two lines." > ) > > While both of these are disallowed: > > fruits = [ > 'orange', > 'grape' > 'banana', > ] > > some_str.replace( > "This is a long string " > "split over two lines.", > "And this is another string." > ) > > +1... if this is indeed implemented in a linter ! -------------- next part -------------- An HTML attachment was scrubbed... URL: From solipsis at pitrou.net Wed Mar 14 13:36:47 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2018 18:36:47 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations References: <494794ce-29f7-fc90-cc7b-a4c665c90e69@kynesim.co.uk> Message-ID: <20180314183647.4dc606aa@fsol> On Tue, 13 Mar 2018 15:35:56 +0000 Rhodri James wrote: > On 12/03/18 20:57, George Fischhof wrote: > > Good day all, > > > > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to > > contain file operations. > > > > Here is the draft. What do you think about this? > > I am mildly negative about this. In my copious spare time (ho ho) I > have been considering the use of pathlib with things that aren't filing > systems, such as http or ftp. Moving the file operations into pathlib > would strain that idea even harder than my (extremely superficial) > current thinking. pathlib is meant for local filesystem paths. Dealing with arbitrary "things which look like paths or have paths in them" (URLs?) is a topic for another library. Regards Antoine. From solipsis at pitrou.net Wed Mar 14 13:41:57 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 14 Mar 2018 18:41:57 +0100 Subject: [Python-ideas] Descouraging the implicit string concatenation References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: <20180314184157.2ac74172@fsol> On Wed, 14 Mar 2018 15:03:02 +0000 Guido van Rossum wrote: > I use the feature regularly for long error messages, and when combined with > .format() or % it's not so easy to replace it with a + (it would require > adding parentheses). Same here. Implicit string concatenation together with string formatting is quite common for me when writing error messages. Regards Antoine. From rosuav at gmail.com Wed Mar 14 13:57:51 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Mar 2018 04:57:51 +1100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: On Thu, Mar 15, 2018 at 12:40 AM, S?ren Pilg?rd wrote: > Of course you can always make error, even in a single letter. > But I think there is a big difference between mixing up +-/* and ** > where the operator is in "focus" and the implicit concatenation where > there is no operator. > A common problem is that you have something like > foo(["a", > "b", > "c" > ]).bar() > but then you remember that there also needs to be a "d" so you just add > foo(["a", > "b", > "c" > "d" > ]).bar() > Causing an error, not with the "d" expression you are working on but > due to what you thought was the previous expression but python turns > it into one. > The , is seen as a delimiter by the programmer not as part of the > operation (or the lack of the ,). You're creating a list. Put a comma at the end of every line; problem solved. Your edit would be from this: foo(["a", "b", "c", ]).bar() to this: foo(["a", "b", "c", "d", ]).bar() and there is no bug. In fact, EVERY time you lay out a list display vertically (or dict or set or any equivalent), ALWAYS put commas. That way, you can reorder the lines freely (you don't special-case the last one), you can append a line without changing the previous one (no noise in the diff), etc, etc, etc. ChrisA From wes.turner at gmail.com Wed Mar 14 14:22:54 2018 From: wes.turner at gmail.com (Wes Turner) Date: Wed, 14 Mar 2018 14:22:54 -0400 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: path.py conveniently defines very many (most?) file operations as methods: https://github.com/jaraco/path.py/blob/master/path.py https://pathpy.readthedocs.io/en/latest/api.html The division operator is mapped to os.path.join: Path('a') / '/root' == Path('/root') On Monday, March 12, 2018, George Fischhof wrote: > > > 2018-03-12 22:16 GMT+01:00 Paul Moore : > >> On 12 March 2018 at 20:57, George Fischhof wrote: >> > Good day all, >> > >> > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to >> > contain file operations. >> > >> > Here is the draft. What do you think about this? >> >> I don't know for certain what I think I feel about the idea - in >> general, it seems plausible. But I think you'll need to get down to >> specifics in the PEP, exactly what functions do you suggest get added >> to pathlib? >> >> Paul >> > > > Basically file and directory operations: copy, rename, move, delete. > With all variants. > I will collect them, and put intot PEP > > George > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Wed Mar 14 14:25:28 2018 From: brenbarn at brenbarn.net (Brendan Barnwell) Date: Wed, 14 Mar 2018 11:25:28 -0700 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: <20180314111808.GW18868@ando.pearwood.info> References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> Message-ID: <5AA96918.40605@brenbarn.net> On 2018-03-14 04:18, Steven D'Aprano wrote: > Apart from intentionally manipulating sys.modules or the import system, > or playing file system tricks like hard-linking your files, under what > circumstances can this occur by accident? It can occur if a given directory winds up appearing twice on the import path. For instance, if /foo is on the path and /foo/bar is a package directory with /foo/bar/baz as a subpackage directory, then you can do "from bar import baz" and "import baz" and wind up with two different module objects referring to the same module. This usually happens when code starts adding paths to sys.path. This is in some sense "manipulating the import system" but it's something that a fair number of libraries do in various contexts, in order to be able to do things like import plugins without requiring the user to make those plugins available on the default import path. For what it's worth, I have been bitten by the problem a few times, although it's not very common. I think it's worth considering the proposal, but not sure if any change is justified given that the issue is fairly obscure. -- Brendan Barnwell "Do not follow where the path may lead. Go, instead, where there is no path, and leave a trail." --author unknown From rosuav at gmail.com Wed Mar 14 14:39:19 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Mar 2018 05:39:19 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: <5AA96918.40605@brenbarn.net> References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> Message-ID: On Thu, Mar 15, 2018 at 5:25 AM, Brendan Barnwell wrote: > On 2018-03-14 04:18, Steven D'Aprano wrote: >> >> Apart from intentionally manipulating sys.modules or the import system, >> or playing file system tricks like hard-linking your files, under what >> circumstances can this occur by accident? > > > It can occur if a given directory winds up appearing twice on the > import path. For instance, if /foo is on the path and /foo/bar is a package > directory with /foo/bar/baz as a subpackage directory, then you can do "from > bar import baz" and "import baz" and wind up with two different module > objects referring to the same module. > > This usually happens when code starts adding paths to sys.path. > This is in some sense "manipulating the import system" but it's something > that a fair number of libraries do in various contexts, in order to be able > to do things like import plugins without requiring the user to make those > plugins available on the default import path. Or by running something that's part of a package, thus making the current directory automatically available. spam/ __init__.py spam.py ham.py python3 spam/spam.py If it now says "from spam import ham", is it going to get ham.py, or is it going to try to pull up ham from spam.py? ChrisA From robertvandeneynde at hotmail.com Wed Mar 14 10:09:49 2018 From: robertvandeneynde at hotmail.com (Robert Vanden Eynde) Date: Wed, 14 Mar 2018 14:09:49 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314140107.wrq4ke6qkap7zxh4@phdru.name> References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: +1 on general idea of discouraging it. > [...] mistakes like: > > fruits = { > "apple", > "orange" > "banana", > "melon", > } +1 > (and even making the static analysers, like pyflakes or pylint, to > show that as a warning) +1 > I agree that implicit concatenation is a bad feature. I have seen it > cause multiple stupid errors while I have never actually used it for > anything useful. [...] Zen of Python. +1. But I think a counter argument would be a real world example. > We should write the following, instead: > > long_string = ( > "some part of the string " + > "with more words, actually is the same " + > "string that the compiler puts together") Maybe the "+" at the beginnig as PEP8 said, to clearly mark it present and explicit. long_string = ( "some part of the string " + "with more words, actually is the same " + "string that the compiler puts together") or long_string = ("" + "some part of the string " + "with more words, actually is the same " + "string that the compiler puts together") > People can make all sorts of mistakes through carlessness. I wrote > {y, x*3} > the other day instead of {y: x**3} Other comments said this isn't really the same kind of error, I agree to them. >> Note that there's no penalty in adding the '+' between the strings, >> those are resolved at compilation time. > That is an implementation feature, not a language requirement. Not all > Python interpreters will do that, and they are free to put limits on > how much they optimize. Here's Python 3.5: > With string concatentation having potential O(N**2) performance, if > you're writing explicit string concatenations with +, they could > potentially be *very* expensive at runtime. I completely agree, by emitting warning if this syntax is used, codes Could be less efficient, but that's also the job of the implementations to be fast. This is the only counter-argument I have to OP, even if I really like the proposal. > Life is too short to learn *everything*, I know that, and I've certainly > copied lots of code I don't understand (and hoped I'd never need to > debug it!). If that makes me a cargo-cult coder too, so be it. I find that learning `x = "e" "f"` does indeed compile is also learning and surprising (especially for beginners). > While I personally dislike implicit string concatenation, > I've worked in projects that use it, and I, more often than > I'd like, use them myself. We would like to see more real world example vs real world mistakes. > It is a bit cumbersome to do something like > from textwrap import dedent as D > a = D("""\ > My code-aligned, multiline text > with no spaces to the left goes here > """) +1. But you should add "\n" to have a = ("hello\n" "world\n" "how") Which would be maybe better written as a = '\n'.join([ "hello", "world", "how"]) (put the spaces where you like) which can easily be switched to ''.join if needed. > A common problem is that you have something like > foo(["a", > "b", > "c" > ]).bar() > but then you remember that there also needs to be a "d" so you just add +1 > Yes of course the skilled programmer needs to understand the > difference. But I am talking about absolute beginners where python in > many regards is an excelent first language. Agreed. From elazarg at gmail.com Wed Mar 14 14:59:55 2018 From: elazarg at gmail.com (Elazar) Date: Wed, 14 Mar 2018 18:59:55 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: On Wed, Mar 14, 2018 at 8:43 PM Robert Vanden Eynde < robertvandeneynde at hotmail.com> wrote: > > We should write the following, instead: > > > > long_string = ( > > "some part of the string " + > > "with more words, actually is the same " + > > "string that the compiler puts together") > > Maybe the "+" at the beginnig as PEP8 said, to clearly mark it present > and explicit. > > long_string = ( > "some part of the string " > + "with more words, actually is the same " > + "string that the compiler puts together") > > or > > long_string = ("" > + "some part of the string " > + "with more words, actually is the same " > + "string that the compiler puts together") > > > Or allow unary plus: long_string = ( + "some part of the string " + "with more words, actually is the same " + "string that the compiler puts together" ) Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Wed Mar 14 15:02:37 2018 From: njs at pobox.com (Nathaniel Smith) Date: Wed, 14 Mar 2018 12:02:37 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: On Mar 12, 2018 1:57 PM, "George Fischhof" wrote: This PEP proposes pathlib module to be a centralized place for all file-system related operations. I'd find this useful for another reason that hasn't been mentioned yet: having a single class collecting all the common/basic file operations makes it much easier to provide a convenient async version of that interface. For example: https://trio.readthedocs.io/en/latest/reference-io.html#trio.Path Obviously it can never be complete, since there are always going to be standalone functions that take paths and work on them internally (for example in third party libraries), but the operations we're talking about here are all pretty basic primitives. -n -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Wed Mar 14 15:09:55 2018 From: guido at python.org (Guido van Rossum) Date: Wed, 14 Mar 2018 12:09:55 -0700 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: <5AA96918.40605@brenbarn.net> References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> Message-ID: Yeah, one should never add a module to sys.path that has a __init__.py file. On Wed, Mar 14, 2018 at 11:25 AM, Brendan Barnwell wrote: > On 2018-03-14 04:18, Steven D'Aprano wrote: > >> Apart from intentionally manipulating sys.modules or the import system, >> or playing file system tricks like hard-linking your files, under what >> circumstances can this occur by accident? >> > > It can occur if a given directory winds up appearing twice on the > import path. For instance, if /foo is on the path and /foo/bar is a > package directory with /foo/bar/baz as a subpackage directory, then you can > do "from bar import baz" and "import baz" and wind up with two different > module objects referring to the same module. > > This usually happens when code starts adding paths to sys.path. > This is in some sense "manipulating the import system" but it's something > that a fair number of libraries do in various contexts, in order to be able > to do things like import plugins without requiring the user to make those > plugins available on the default import path. > > For what it's worth, I have been bitten by the problem a few > times, although it's not very common. I think it's worth considering the > proposal, but not sure if any change is justified given that the issue is > fairly obscure. > > -- > Brendan Barnwell > "Do not follow where the path may lead. Go, instead, where there is no > path, and leave a trail." > --author unknown > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Wed Mar 14 16:09:04 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2018 16:09:04 -0400 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: On 3/14/2018 8:18 AM, Facundo Batista wrote: > Hello! > > What would you think about formally descouraging the following idiom? > > long_string = ( > "some part of the string " > "with more words, actually is the same " > "string that the compiler puts together") -1 > We should write the following, instead: > > long_string = ( > "some part of the string " + > "with more words, actually is the same " + > "string that the compiler puts together") > > Note that there's no penalty in adding the '+' between the strings, > those are resolved at compilation time. Not always, as Eryk Sun discussed. When f-strings are thrown into the mix, as when creating multi-line messages with interpolated values, not even true in 3.7. Compare the following: >>> def f(): x = input('x:') print(f'a{x}bc' 'def') >>> dis.dis(f) 2 0 LOAD_GLOBAL 0 (input) 2 LOAD_CONST 1 ('x:') 4 CALL_FUNCTION 1 6 STORE_FAST 0 (x) 3 8 LOAD_GLOBAL 1 (print) 10 LOAD_CONST 2 ('a') 12 LOAD_FAST 0 (x) 14 FORMAT_VALUE 0 16 LOAD_CONST 3 ('bcdef') 18 BUILD_STRING 3 20 CALL_FUNCTION 1 22 POP_TOP 24 LOAD_CONST 0 (None) 26 RETURN_VALUE >>> def f(): x = input('x:') print(f'a{x}bc' + 'def') >>> dis.dis(f) 2 0 LOAD_GLOBAL 0 (input) 2 LOAD_CONST 1 ('x:') 4 CALL_FUNCTION 1 6 STORE_FAST 0 (x) 3 8 LOAD_GLOBAL 1 (print) 10 LOAD_CONST 2 ('a') 12 LOAD_FAST 0 (x) 14 FORMAT_VALUE 0 16 LOAD_CONST 3 ('bc') 18 BUILD_STRING 3 20 LOAD_CONST 4 ('def') 22 BINARY_ADD 24 CALL_FUNCTION 1 26 POP_TOP 28 LOAD_CONST 0 (None) 30 RETURN_VALUE The '+' prevents compile time concatenation of 'bc' and 'def' and adds a load and binary add. -- Terry Jan Reedy From tjreedy at udel.edu Wed Mar 14 16:11:05 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2018 16:11:05 -0400 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: On 3/14/2018 11:03 AM, Guido van Rossum wrote: > I use the feature regularly for long error messages, and when combined > with .format() or % it's not so easy to replace it with a + (it would > require adding parentheses). I like using f-strings for this now, and adding '+' stops compile-time concatenation across literal boundaries. See my response to FB's original post. -- Terry Jan Reedy From barry at barrys-emacs.org Wed Mar 14 16:40:26 2018 From: barry at barrys-emacs.org (Barry) Date: Wed, 14 Mar 2018 20:40:26 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: <59747DEF-054E-40A4-B4AD-ED34D79F4D59@barrys-emacs.org> > On 12 Mar 2018, at 21:56, George Fischhof wrote: > > > > 2018-03-12 22:16 GMT+01:00 Paul Moore : >> On 12 March 2018 at 20:57, George Fischhof wrote: >> > Good day all, >> > >> > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to >> > contain file operations. >> > >> > Here is the draft. What do you think about this? >> >> I don't know for certain what I think I feel about the idea - in >> general, it seems plausible. But I think you'll need to get down to >> specifics in the PEP, exactly what functions do you suggest get added >> to pathlib? >> >> Paul > > > Basically file and directory operations: copy, rename, move, delete. With all variants. > I will collect them, and put intot PEP > I thought that because of __fspath__ conversions pathlib can be use with any function that needs a path. So long as shutil is Pathlib friendly isn?t this asolv d problem? No need to extend pathlib for every file op, just pass the existed file ops a Path. Wasn?t this discussed at the time fspath was added? Barry > George > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Wed Mar 14 18:04:29 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Mar 2018 11:04:29 +1300 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: <5AA99C6D.20804@canterbury.ac.nz> Facundo Batista wrote: > What was proposed and reject was the *removal* of the feature/syntax. > > I propose the discouragement of the idiom. Providing a linter option to catch it would be fine. Officially discouraging it would be going too far, I think. It's one thing to promote style guidelines concerning how to use or not use certain language features. But this would be telling people they should *never* use an entire feature -- effectively deprecating it even if there is no intention to remove it -- for very subjective reasons. That comes across as excessively moralistic to me. I object to someone telling me what I should or shouldn't do in the privacy of my own code. -- Greg From klahnakoski at mozilla.com Wed Mar 14 18:06:56 2018 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Wed, 14 Mar 2018 18:06:56 -0400 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: Message-ID: <6347159f-3375-2f71-e478-e0b93216e572@mozilla.com> On 2018-03-14 08:18, Facundo Batista wrote: > Hello! > > What would you think about formally descouraging the following idiom? > > long_string = ( > "some part of the string " > "with more words, actually is the same " > "string that the compiler puts together") > Thank you for bring this up!? A regex through one of my programs revealed three bugs caused by implicit concatenation in lists.? Here is one of them: https://github.com/klahnakoski/ActiveData-ETL/blob/etl/activedata_etl/transforms/pulse_block_to_task=_cluster.py#L897? The missing commas were not caught until now because of the lists are long, and deal with rare cases. I did use implicit concatenation for a long SQL statement, and a few long error messages, but the byte savings is not worth the increased bug density. >??? self.db.execute( >??????? "CREATE TABLE files (" + >??????? "?? bucket TEXT," + >??????? "?? key TEXT," + >??????? "?? name TEXT," + >??????? "?? last_modified REAL," + >??????? "?? size INTEGER," + >??????? "?? annotate TEXT, " + >??????? "?? CONSTRAINT pk PRIMARY KEY (bucket, name)" + >??????? ")" >??? ) is almost identical to >??? self.db.execute( >??????? "CREATE TABLE files (" >??????? "?? bucket TEXT," >??????? "?? key TEXT," >??????? "?? name TEXT," >??????? "?? last_modified REAL," >??????? "?? size INTEGER," >??????? "?? annotate TEXT, " >??????? "?? CONSTRAINT pk PRIMARY KEY (bucket, name)" >??????? ")" >??? ) ??? It looks like I am in the market for a linter that prevents implicit string concatenation! From steve at pearwood.info Wed Mar 14 18:20:42 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Mar 2018 09:20:42 +1100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: <20180314222042.GC18868@ando.pearwood.info> On Wed, Mar 14, 2018 at 01:47:53PM -0300, Facundo Batista wrote: > 2018-03-14 11:30 GMT-03:00 Stephan Houben : > > > Op 14 mrt. 2018 15:23 schreef "Facundo Batista" : > > > > I propose the discouragement of the idiom. > > > > > > > > What does that mean? > > That we say "hey, this works but please don't use it, because it tends > to a error prone way of writing some code, instead do this". It's not error prone. It works perfectly. The error you are complaining about has absolutely nothing to do with the intentional use of implicit string concatenation. The error you describe comes from accidentally leaving out a comma. Suppose I follow this advice. I spend a month going through all my code, removing every example of implicit string concatenation and replace it with some work-around like runtime string concatenation or calling textwrap.dedent() on triple quoted strings. And I swear to never, ever use implicit string concatenation again. And the very next day, I accidently leave out a comma in a list of strings. How exactly does it help me to follow this advice? All it does is give me a lot of extra work to do, by avoiding a useful and reliable feature, without preventing the error you are concerned about. -- Steve From greg.ewing at canterbury.ac.nz Wed Mar 14 18:22:53 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Mar 2018 11:22:53 +1300 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <20180314145253.egtmioqik4dq2shg@phdru.name> References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: <5AA9A0BD.5090402@canterbury.ac.nz> Oleg Broytman wrote: > The most painful was that the bug destroyed important information > that was hard to fix later (enterprise code in production, the code is > certificated and hard to replace). Seems to me there are a number of problems right there: * Important information that is not backed up or is hard to recover from wherever it is backed up * A rigid deployment process without correspondingly thorough testing of what is to be deployed Blaming the debacle on one particular language feature, when something similar could have happened for any number of other reasons, seems like scapegoat-hunting. > The third pain was to find the bug. It's hard to spot a problem in > the list > > oids = [ > "1.2.6.254.14." > "19.9.91.281", > "1.2.6.263.12." > "481.7.9.6.3.87" > "1.2.7.4.214.7." > "9.1.52.12", > ] Again, testing. You're about to deploy a hard-coded list of IDs to a mission-critical environment where it will be very difficult to change anything later. Might it perhaps be prudent to do something like checking that they're all valid? A typo in one of the IDs would have given you the same result, without involving any language features of dubious merit. -- Greg From rosuav at gmail.com Wed Mar 14 18:31:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Thu, 15 Mar 2018 09:31:02 +1100 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <6347159f-3375-2f71-e478-e0b93216e572@mozilla.com> References: <6347159f-3375-2f71-e478-e0b93216e572@mozilla.com> Message-ID: On Thu, Mar 15, 2018 at 9:06 AM, Kyle Lahnakoski wrote: > I did use implicit concatenation for a long SQL statement, and a few > long error messages, but the byte savings is not worth the increased bug > density. > >> self.db.execute( >> "CREATE TABLE files (" + >> " bucket TEXT," + >> " key TEXT," + >> " name TEXT," + >> " last_modified REAL," + >> " size INTEGER," + >> " annotate TEXT, " + >> " CONSTRAINT pk PRIMARY KEY (bucket, name)" + >> ")" >> ) > > is almost identical to > >> self.db.execute( >> "CREATE TABLE files (" >> " bucket TEXT," >> " key TEXT," >> " name TEXT," >> " last_modified REAL," >> " size INTEGER," >> " annotate TEXT, " >> " CONSTRAINT pk PRIMARY KEY (bucket, name)" >> ")" >> ) > > It looks like I am in the market for a linter that prevents implicit > string concatenation! Even better: self.db.execute(""" CREATE TABLE files ( bucket TEXT, key TEXT, name TEXT, last_modified REAL, size INTEGER, annotate TEXT, CONSTRAINT pk PRIMARY KEY (bucket, name) ) """) ChrisA From greg.ewing at canterbury.ac.nz Wed Mar 14 18:37:16 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Mar 2018 11:37:16 +1300 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> <20180314140107.wrq4ke6qkap7zxh4@phdru.name> Message-ID: <5AA9A41C.7030502@canterbury.ac.nz> Facundo Batista wrote: > That we say "hey, this works but please don't use it, because it tends > to a error prone way of writing some code, instead do this". The trouble is that there is nowhere near widespread agreement that it is error prone enough to be worth such a drastic measure as deprecating the whole feature. -- Greg From greg.ewing at canterbury.ac.nz Wed Mar 14 18:40:54 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Thu, 15 Mar 2018 11:40:54 +1300 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> Message-ID: <5AA9A4F6.8050104@canterbury.ac.nz> Carl Meyer wrote: > With this rule the only missing-comma that can slip through is when > you've forgotten _all_ the intervening commas in your sequence of > strings. That's much less likely. Not so unlikely when the argument list is of length 2. -- Greg From bzvi7919 at gmail.com Wed Mar 14 19:37:02 2018 From: bzvi7919 at gmail.com (Bar Harel) Date: Wed, 14 Mar 2018 23:37:02 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: <5AA9A4F6.8050104@canterbury.ac.nz> References: <20180314133158.4ldr2usx7je45zpx@phdru.name> <20180314141806.GZ18868@ando.pearwood.info> <20180314145253.egtmioqik4dq2shg@phdru.name> <5AA9A4F6.8050104@canterbury.ac.nz> Message-ID: I'm with Steven D`Aprano here. Implicit string concat, in my experience, is something clear, concise and convenient. I've had a bug because of it only once or twice during my whole career, and not only that "occasional" bug would have occurred either way because of a missing comma or plus sign, but it saves the trouble of having to add more symbols which usually means less surface area for bugs. If you feel it affects you all so often, I would totally understand the addition of that feature in a linter, but not endorsed by the python community in any way, and with defaults probably set to disabled. I wouldn't want to add any superfluous symbols to my otherwise (at least in my opinion) cleaner code. All in all, -1 from me. On Thu, Mar 15, 2018 at 12:41 AM Greg Ewing wrote: > Carl Meyer wrote: > > With this rule the only missing-comma that can slip through is when > > you've forgotten _all_ the intervening commas in your sequence of > > strings. That's much less likely. > > Not so unlikely when the argument list is of length 2. > > -- > Greg > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Wed Mar 14 20:32:35 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Thu, 15 Mar 2018 01:32:35 +0100 Subject: [Python-ideas] Syntax idea for 2D lists\arrays Message-ID: Once there was a discussion about alternative list and arrays creation and assignment syntaxes. I think this is one of things with room for ideas and has frequent usage. I've had some ideas for syntax long ago, and now I remembered it because of the recent topic "Descouraging the implicit string concatenation". Idea is a concept for 2D arrays/lists syntax, which should simplify some editing boilerplate while working with arrays and improve readability for bigger arrays. Lets start with a simple list example : L === 1 5 9 155 53 44 44 34 returns a 2d list: [[1, 5, 9, 155], [53, 44, 44, 34]] Syntax rules here: - "===" indicates parsing start (may be some other decorator) - data block *always* starts on the new line and parsing always like in a "def:" block (by indentation, till next negative indent) - new line is a next element - white space separated inner elements (tabs, spaces) Line continuation as usual: L === 2 1 5 \ 3 4 6 returns: [[2, 1, 5, 3, 4, 5]] SPECIAL CASE: 1d list could be explicitly marked with e.g. *: L ===* 2 1 5 3 4 6 returns : [2, 1, 5, 3, 4, 5] Also in 1d case one might omit the line continuation \: L ===* "msg1" "msg2" "error" returns: ["msg1", "msg2", "error"] IMO the latter looks nice and very useful for pretty-formatting of string lists. ---- *Examples for working with 3D arrays:* import numpy as np A = np.zeros((2,3,2)) A[0,0] === 8 8 A[1] === 0 0 155 1 1 1 returns: [[[ 8. 8.] [ 0. 0.] [ 0. 0.]] [[ 0. 0.] [ 155. 1.] [ 1. 1.]]] Compared to same operation in current syntax: A[0, 0] = [8, 8] A[1] = [ [0, 0], [155, 1], [1, 1] ] (which is ugly, though can be a bit more compact in simplest case) I think these examples show the general idea already. Pros: - clean readable look - copy-pasting tables from e.g. excel with little to no re-typing - no more editing annoyance with trailing commas/brackets Cons: - less compact for trivail cases (newline always added) ----- Problems that I see with this syntax: *Comma-separated vs space separated*. Space separated IMO better readable in general, though single space can be too dense with some proportional fonts. In this case two or more spaces could be better. Also consider expressions inside arrays, e.g.: SS ===* 11 + 3 5 6 SS ===* 11 + 3, 5, 6 I would not say comma really improves things here, but opinions may vary. Still, all in all, I am on the side of space-separated. Also numpy uses space-separated with print() method by default. ------ How do you like such syntax? Maybe someone can come up with counter-examples where such syntax will lead to ugly look? Mikhail From tjreedy at udel.edu Wed Mar 14 21:10:05 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Wed, 14 Mar 2018 21:10:05 -0400 Subject: [Python-ideas] Syntax idea for 2D lists\arrays In-Reply-To: References: Message-ID: On 3/14/2018 8:32 PM, Mikhail V wrote: > Once there was a discussion about alternative list and arrays > creation and assignment syntaxes. I think this is one of things with room for > ideas and has frequent usage. > I've had some ideas for syntax long ago, and now I remembered it because of > the recent topic "Descouraging the implicit string concatenation". > > Idea is a concept for 2D arrays/lists syntax, which should simplify > some editing boilerplate while working with arrays and improve > readability for bigger arrays. > > Lets start with a simple list example : > > L === > 1 5 9 155 > 53 44 44 34 > > returns a 2d list: > [[1, 5, 9, 155], [53, 44, 44, 34]] I would rather type this, with or without a return. Literal arrays are not this common. When the entries are constant, a tuple of tuples is much more efficient as it is built at compile time >>> dis.dis("((1,2),(3,4))") 1 0 LOAD_CONST 0 (((1, 2), (3, 4))) 2 RETURN_VALUE >>> dis.dis("[[1,2],[3,4]]") 1 0 LOAD_CONST 0 (1) 2 LOAD_CONST 1 (2) 4 BUILD_LIST 2 6 LOAD_CONST 2 (3) 8 LOAD_CONST 3 (4) 10 BUILD_LIST 2 12 BUILD_LIST 2 14 RETURN_VALUE > > Syntax rules here: > - "===" indicates parsing start (may be some other decorator) > - data block *always* starts on the new line and parsing always > like in a "def:" block (by indentation, till next negative indent) > - new line is a next element > - white space separated inner elements (tabs, spaces) > > Line continuation as usual: > L === > 2 1 5 \ > 3 4 6 > returns: > [[2, 1, 5, 3, 4, 5]] > > > SPECIAL CASE: 1d list could be explicitly marked with e.g. *: > > L ===* > 2 1 5 3 4 6 > returns : > [2, 1, 5, 3, 4, 5] > > Also in 1d case one might omit the line continuation \: > > L ===* > "msg1" > "msg2" > "error" > > returns: > ["msg1", "msg2", "error"] No, this would return "msg1msg2error" > IMO the latter looks nice and very useful for pretty-formatting > of string lists. > > ---- > > *Examples for working with 3D arrays:* > > import numpy as np > A = np.zeros((2,3,2)) > > A[0,0] === > 8 8 > A[1] === > 0 0 > 155 1 > 1 1 > > returns: > [[[ 8. 8.] > [ 0. 0.] > [ 0. 0.]] > [[ 0. 0.] > [ 155. 1.] > [ 1. 1.]]] > > Compared to same operation in current syntax: > > A[0, 0] = [8, 8] > A[1] = [ > [0, 0], > [155, 1], > [1, 1] ] > > (which is ugly, though can be a bit more compact in simplest case) > > I think these examples show the general idea already. > > Pros: > - clean readable look > - copy-pasting tables from e.g. excel with little to no re-typing > - no more editing annoyance with trailing commas/brackets > > Cons: > - less compact for trivail cases (newline always added) > > ----- > > Problems that I see with this syntax: > *Comma-separated vs space separated*. > > Space separated IMO better readable in general, > though single space can be too dense with some proportional > fonts. In this case two or more spaces could be better. > > Also consider expressions inside arrays, e.g.: > SS ===* > 11 + 3 5 6 > SS ===* > 11 + 3, 5, 6 > > I would not say comma really improves things here, but > opinions may vary. > Still, all in all, I am on the side of space-separated. > Also numpy uses space-separated with print() method by default. > > ------ > > How do you like such syntax? Maybe someone can come up > with counter-examples where such syntax will lead to ugly look? > > > > Mikhail > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- Terry Jan Reedy From mikhailwas at gmail.com Wed Mar 14 23:32:28 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Thu, 15 Mar 2018 04:32:28 +0100 Subject: [Python-ideas] Syntax idea for 2D lists\arrays In-Reply-To: References: Message-ID: On Thu, Mar 15, 2018 at 2:10 AM, Terry Reedy wrote: > On 3/14/2018 8:32 PM, Mikhail V wrote: >> >> ... >> >> Also in 1d case one might omit the line continuation \: >> >> L ===* >> "msg1" >> "msg2" >> "error" >> >> returns: >> ["msg1", "msg2", "error"] > > > No, this would return "msg1msg2error" > Ah well, if we already in implementation details - sure, a complete solution would be a whole new layer in parser - lists, tuples, lists of tuples, dicts. It'd be what I suppose is called a [new to python] micro-language, with own preprocessing. From steve at pearwood.info Thu Mar 15 01:15:26 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Thu, 15 Mar 2018 16:15:26 +1100 Subject: [Python-ideas] Syntax idea for 2D lists\arrays In-Reply-To: References: Message-ID: <20180315051525.GB16661@ando.pearwood.info> On Thu, Mar 15, 2018 at 01:32:35AM +0100, Mikhail V wrote: > Idea is a concept for 2D arrays/lists syntax, which should simplify > some editing boilerplate while working with arrays and improve > readability for bigger arrays. I don't understand; we already have perfectly good syntax for working with 2D arrays. > Lets start with a simple list example : > > L === > 1 5 9 155 > 53 44 44 34 > > returns a 2d list: > [[1, 5, 9, 155], [53, 44, 44, 34]] We already have: L = [[1, 5, 9, 155], [53, 44, 44, 34]] which is more compact (one line rather than two) and explicitly delimits the start and end of each list. Like everything else in Python, it uses commas to separate items, not whitespace. If you prefer: L = [[1, 5, 9, 155], [53, 44, 44, 34]] Using spaces to separate items has the fatal flaw that it cannot distinguish x - y 0 # two items, the expression `x - y` and the integer 0 from: x - y 0 # three items, `x`, `-y`, and 0 making it ambiguous. I stopped reading your post once I realised that. -- Steve From ncoghlan at gmail.com Thu Mar 15 06:25:22 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Thu, 15 Mar 2018 20:25:22 +1000 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: Message-ID: On 14 March 2018 at 15:20, Chris Billington wrote: > > I wonder if there's any reason something like this shouldn't be built into > Python's default import system. > There are two main challenges with enforcing such a check, one affecting end users in general, one affecting standard library maintainers in particular: * the user facing problem is a backwards compatibility one: while double-imports usually aren't what people wanted, they're also typically fairly harmless. As a result, elevating them from "sometimes a source of obscure bugs" to "categorically prohibited" risks breaking currently working code. While a conventional deprecation cycle should be possible, it isn't clear whether or not the problem occurs frequently enough to warrant that effort. * the maintainer level problem is that we actually do this on purpose in the standard library's test suite in order to test both pure Python and C accelerated variants of various modules. That could be handled by being careful about exactly where the reverse lookup cache from filenames back to module names is checked and updated, but it does make the problem a bit trickier than just "maintain a reverse lookup table from filesystem paths to module names and complain if an import gets a hit in that table" I'm definitely sympathetic to the idea, though. If we did head in this direction, then we'd also need to accept & implement PEP 499 [1] (which proposes aliasing __main__ as __main__.__spec__.name in sys.modules when executed with "-m") to avoid causing problems. Cheers, Nick. [1] https://www.python.org/dev/peps/pep-0499/ -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From stephanh42 at gmail.com Thu Mar 15 07:21:47 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 15 Mar 2018 12:21:47 +0100 Subject: [Python-ideas] Syntax idea for 2D lists\arrays In-Reply-To: <20180315051525.GB16661@ando.pearwood.info> References: <20180315051525.GB16661@ando.pearwood.info> Message-ID: I'd just do this, which works today: ================== import numpy import io ar = numpy.loadtxt(io.StringIO(""" 1 5 9 155 53 44 44 34 """)) ================== Of course, this is only worth the trouble if you are somehow loading a very large matrix. (And then, are you sure you want to embed it in your code?) Stephan 2018-03-15 6:15 GMT+01:00 Steven D'Aprano : > On Thu, Mar 15, 2018 at 01:32:35AM +0100, Mikhail V wrote: > > > Idea is a concept for 2D arrays/lists syntax, which should simplify > > some editing boilerplate while working with arrays and improve > > readability for bigger arrays. > > I don't understand; we already have perfectly good syntax for working > with 2D arrays. > > > Lets start with a simple list example : > > > > L === > > 1 5 9 155 > > 53 44 44 34 > > > > returns a 2d list: > > [[1, 5, 9, 155], [53, 44, 44, 34]] > > We already have: > > L = [[1, 5, 9, 155], [53, 44, 44, 34]] > > which is more compact (one line rather than two) and explicitly delimits > the start and end of each list. Like everything else in Python, it uses > commas to separate items, not whitespace. If you prefer: > > L = [[1, 5, 9, 155], > [53, 44, 44, 34]] > > > Using spaces to separate items has the fatal flaw that it cannot > distinguish > > x - y 0 # two items, the expression `x - y` and the integer 0 > > from: > > x - y 0 # three items, `x`, `-y`, and 0 > > > making it ambiguous. I stopped reading your post once I realised that. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From george at fischhof.hu Thu Mar 15 09:38:04 2018 From: george at fischhof.hu (George Fischhof) Date: Thu, 15 Mar 2018 14:38:04 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: <20180313121736.GT18868@ando.pearwood.info> References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: 2018-03-13 13:17 GMT+01:00 Steven D'Aprano : > On Mon, Mar 12, 2018 at 09:57:32PM +0100, George Fischhof wrote: > > > Right now we have several modules that contain functions related > > to file-system operations mainly the os, pathlib and shutil. > > For beginners it is quite hard to remember where can he / she find > > a function (copy resides in shutil, but the remove function can be > > found in the os module. (And sometimes developers with moderate > > experience have to check the documentation as well.) > > This is certainly a problem. Not a big problem, but it is an annoyance. > > > > With functions in pathlib the developers should not have to think > > on which method (function) can be found in which module. > > > > Makes the life easier. > > I don't know that this will be true. It makes one problem better: you no > longer have to remember which module the function is in. But it makes > other things worse: > > - code and/or API duplication: for backwards compatibility, every > existing function must be in two places, the original and in > pathlib; > > - if new file functions are added, they will go only in pathlib, > which makes pathlib effectively mandatory; > > - the pathlib API becomes even more complicated: not only have you > got all the methods of pathlib objects, but you have all the shutil > and os functions as well. > > > I think this is a good place for an experiment. You could write a > function which monkey-patches pathlib: > > from pathlib import Path > import os > import shutil > > def monkeypatch(): > Path.remove = os.remove > # etc. > > Then we can see how many functions are involved, how large this makes > the Path object, and try it out and see whether it is better. > > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Duplication: it is true, but it is true for several other modules as well. I checked the pathlib module: right now more than 50% of the functions are duplicate - mainly from os - so it seems that pathlib already started to develop this way ;-) " if new file functions are added, they will go only in pathlib, which makes pathlib effectively mandatory;" Yes but I think this part of the evolution: slowly everyone will shift to pathlib, and being mandatory is true for the current status as well: if you need a function, you need the module. Right now if you wan to execute some file operations, you need os plus shutil, because the half of the functions are in one of them, the other half is in the other module I collected the functions that sould be put into pathlib: - os.remove - os.removedirs (shutil.rmtree has the same functionalaty) - os.truncate - shutil.copyfileobj - shutil.copyfile - shutil.copymode - shutil.copystat - shutil.copy - shutil.copy2 - shutil.copytree with shutil.ignore_patterns - shutil.move - shutil.disk_usage - shutil.chown - os.link => path.hardlink_to - os.mkfifo - os.readlink Sum: 16 functuins And all functions from os module accept path-like objects, and none of the shutil functions. Pathlib already contains 17 functions from os an shutil modules. George -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Thu Mar 15 12:55:05 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Thu, 15 Mar 2018 17:55:05 +0100 Subject: [Python-ideas] Syntax idea for 2D lists\arrays In-Reply-To: <20180315051525.GB16661@ando.pearwood.info> References: <20180315051525.GB16661@ando.pearwood.info> Message-ID: On Thu, Mar 15, 2018 at 6:15 AM, Steven D'Aprano wrote: > On Thu, Mar 15, 2018 at 01:32:35AM +0100, Mikhail V wrote: > > Using spaces to separate items has the fatal flaw that it cannot > distinguish > > x - y 0 # two items, the expression `x - y` and the integer 0 > > from: > > x - y 0 # three items, `x`, `-y`, and 0 > When you say "it cannot distinguish" what is "it"? you mean current parser can't separate items due to the allowed unary negation operator? Well then it is not like I would parse the table, for this case my parsing rule would be: x - y 0 -> [x - y, 0] x-y -> [x - y, 0] x -y 0 -> [x, -y, 0] That's in case I use same char for unary negation and minus operator (which I find rather inconvinient for parsing). Mikhail From rosuav at gmail.com Thu Mar 15 13:15:11 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 16 Mar 2018 04:15:11 +1100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: On Fri, Mar 16, 2018 at 12:38 AM, George Fischhof wrote: > > > " if new file functions are added, they will go only in pathlib, > which makes pathlib effectively mandatory;" > Yes but I think this part of the evolution: slowly everyone will shift to > pathlib, > and being mandatory is true for the current status as well: if you need a > function, you need the module. > Right now if you wan to execute some file operations, you need os plus > shutil, because the half of the > functions are in one of them, the other half is in the other module The os module is cheap; pathlib has a definite cost. If every file operation goes through pathlib, that basically means pathlib becomes part of the startup cost: rosuav at sikorsky:~$ python3 -m timeit -s 'import subprocess, sys, pathlib' 'subprocess.check_call([sys.executable, "-c", "import os"])' 50 loops, best of 5: 8.82 msec per loop rosuav at sikorsky:~$ python3 -m timeit -s 'import subprocess, sys, pathlib' 'subprocess.check_call([sys.executable, "-c", "import pathlib"])' 20 loops, best of 5: 15.9 msec per loop rosuav at sikorsky:~$ python3.6 -m timeit -s 'import subprocess, sys, pathlib' 'subprocess.check_call([sys.executable, "-c", "import os"])' 100 loops, best of 3: 14.1 msec per loop rosuav at sikorsky:~$ python3.6 -m timeit -s 'import subprocess, sys, pathlib' 'subprocess.check_call([sys.executable, "-c", "import pathlib"])' 10 loops, best of 3: 19.7 msec per loop rosuav at sikorsky:~$ python3.5 -m timeit -s 'import subprocess, sys, pathlib' 'subprocess.check_call([sys.executable, "-c", "import os"])' 100 loops, best of 3: 10.6 msec per loop rosuav at sikorsky:~$ python3.5 -m timeit -s 'import subprocess, sys, pathlib' 'subprocess.check_call([sys.executable, "-c", "import pathlib"])' 100 loops, best of 3: 18.7 msec per loop And this is with warm caches; for a true first-time startup, the cost could be significantly higher. ChrisA From solipsis at pitrou.net Thu Mar 15 13:48:22 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Thu, 15 Mar 2018 18:48:22 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: <20180315184822.3b50fb48@fsol> On Fri, 16 Mar 2018 04:15:11 +1100 Chris Angelico wrote: > On Fri, Mar 16, 2018 at 12:38 AM, George Fischhof wrote: > > > > > > " if new file functions are added, they will go only in pathlib, > > which makes pathlib effectively mandatory;" > > Yes but I think this part of the evolution: slowly everyone will shift to > > pathlib, > > and being mandatory is true for the current status as well: if you need a > > function, you need the module. > > Right now if you wan to execute some file operations, you need os plus > > shutil, because the half of the > > functions are in one of them, the other half is in the other module > > The os module is cheap; pathlib has a definite cost. If every file > operation goes through pathlib, that basically means pathlib becomes > part of the startup cost: > > rosuav at sikorsky:~$ python3 -m timeit -s 'import subprocess, sys, > pathlib' 'subprocess.check_call([sys.executable, "-c", "import os"])' > 50 loops, best of 5: 8.82 msec per loop > rosuav at sikorsky:~$ python3 -m timeit -s 'import subprocess, sys, > pathlib' 'subprocess.check_call([sys.executable, "-c", "import > pathlib"])' > 20 loops, best of 5: 15.9 msec per loop Which version is this "python3"? Regards Antoine. From rosuav at gmail.com Thu Mar 15 13:58:24 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 16 Mar 2018 04:58:24 +1100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: <20180315184822.3b50fb48@fsol> References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> Message-ID: On Fri, Mar 16, 2018 at 4:48 AM, Antoine Pitrou wrote: > On Fri, 16 Mar 2018 04:15:11 +1100 > Chris Angelico wrote: >> On Fri, Mar 16, 2018 at 12:38 AM, George Fischhof wrote: >> > >> > >> > " if new file functions are added, they will go only in pathlib, >> > which makes pathlib effectively mandatory;" >> > Yes but I think this part of the evolution: slowly everyone will shift to >> > pathlib, >> > and being mandatory is true for the current status as well: if you need a >> > function, you need the module. >> > Right now if you wan to execute some file operations, you need os plus >> > shutil, because the half of the >> > functions are in one of them, the other half is in the other module >> >> The os module is cheap; pathlib has a definite cost. If every file >> operation goes through pathlib, that basically means pathlib becomes >> part of the startup cost: >> >> rosuav at sikorsky:~$ python3 -m timeit -s 'import subprocess, sys, >> pathlib' 'subprocess.check_call([sys.executable, "-c", "import os"])' >> 50 loops, best of 5: 8.82 msec per loop >> rosuav at sikorsky:~$ python3 -m timeit -s 'import subprocess, sys, >> pathlib' 'subprocess.check_call([sys.executable, "-c", "import >> pathlib"])' >> 20 loops, best of 5: 15.9 msec per loop > > Which version is this "python3"? 3.8, but I also tested with 3.7, 3.6, and 3.5. All of them showed the same kind of difference, but the numbers were a bit higher in the older versions. I fully expect that Python 3.4 (the first one where pathlib existed) would show the same too. ChrisA From guillaumeplique at gmail.com Thu Mar 15 15:39:27 2018 From: guillaumeplique at gmail.com (PLIQUE Guillaume) Date: Thu, 15 Mar 2018 20:39:27 +0100 Subject: [Python-ideas] Adding quantile to the statistics module Message-ID: Hello everyone, Sorry if this subject has already been covered in the mailing list but I could not find it. My question is very simple: should the `quantile` function be added to python `statistics` module. I was very happy to learn the existence of this module in python3 only to later be forced to install numpy to compute a .7 quantile which is even more frustrating since the module actually provides an implementation of the median. I would therefore be willing to submit a PR to add the `quantile` function to the module. The function would have the following signature: ``` # data -> your sequence # p -> the quantile, between 0 & 1 def quantile(data, p) # example quantile([1, 2, 3, 4, 5], 0.5) >>> 3 ``` This would also mean implementing very simply the `quantile_low` & `quantile_high` function as their median counterparts in the module. I am unclear, however, how to implement a hypothetic `quantile_grouped`. What do you think? -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Thu Mar 15 17:42:01 2018 From: njs at pobox.com (Nathaniel Smith) Date: Thu, 15 Mar 2018 14:42:01 -0700 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: References: Message-ID: On Thu, Mar 15, 2018 at 12:39 PM, PLIQUE Guillaume wrote: > Hello everyone, > > Sorry if this subject has already been covered in the mailing list but I > could not find it. > > My question is very simple: should the `quantile` function be added to > python `statistics` module. This seems like a reasonable idea to me -- but be warned that there are actually quite a few slightly-different definitions of "quantile" in use. R supports 9 different methods of calculating quantiles (exposed via an interesting API: their quantile function takes a type= argument, which is an integer between 1 and 9; the default is 7). And there's currently an open issue at numpy discussing whether numpy implements the right approaches: https://github.com/numpy/numpy/issues/10736 So this would require some research to decide on which definition(s) you wanted to support. -n -- Nathaniel J. Smith -- https://vorpus.org From steve at pearwood.info Thu Mar 15 19:46:44 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Fri, 16 Mar 2018 10:46:44 +1100 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: References: Message-ID: <20180315234644.GD16661@ando.pearwood.info> On Thu, Mar 15, 2018 at 02:42:01PM -0700, Nathaniel Smith wrote: > This seems like a reasonable idea to me -- but be warned that there > are actually quite a few slightly-different definitions of "quantile" > in use. R supports 9 different methods of calculating quantiles Indeed. I've been considering quantiles and quartiles for a long time, and I've found at least ten different definitions for quantiles and sixteen for quartiles. -- Steve From rob.cliffe at btinternet.com Thu Mar 15 23:58:05 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Fri, 16 Mar 2018 03:58:05 +0000 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: On 14/03/2018 17:57, Chris Angelico wrote: > On Thu, Mar 15, 2018 at 12:40 AM, S?ren Pilg?rd wrote: >> Of course you can always make error, even in a single letter. >> But I think there is a big difference between mixing up +-/* and ** >> where the operator is in "focus" and the implicit concatenation where >> there is no operator. >> A common problem is that you have something like >> foo(["a", >> "b", >> "c" >> ]).bar() >> but then you remember that there also needs to be a "d" so you just add >> foo(["a", >> "b", >> "c" >> "d" >> ]).bar() >> Causing an error, not with the "d" expression you are working on but >> due to what you thought was the previous expression but python turns >> it into one. >> The , is seen as a delimiter by the programmer not as part of the >> operation (or the lack of the ,). > You're creating a list. Put a comma at the end of every line; problem > solved. Your edit would be from this: > > foo(["a", > "b", > "c", > ]).bar() > > to this: > > foo(["a", > "b", > "c", > "d", > ]).bar() > > and there is no bug. In fact, EVERY time you lay out a list display > vertically (or dict or set or any equivalent), ALWAYS put commas. That > way, you can reorder the lines freely (you don't special-case the last > one), you can append a line without changing the previous one (no > noise in the diff), etc, etc, etc. > > My thoughts exactly.? I make it a personal rule to ALWAYS add a comma to every line, including the last one, in this kind of list (/dict/set etc.). *Python allows it - take advantage of it!* (A perhaps minor-seeming feature of the language which actually is a big benefit if you use it.)? Preferably with all the commas vertically aligned to highlight the structure (I'm a great believer BTW in using vertical alignment, even if it means violating Python near-taboos such as more that one statement per line).? Also I would automatically put the first string (as well as the last) on a line by itself: foo([ ??? ??? "short string" , ??? ??? "extremely looooooooooooong string" , ??? ??? "some other string"?????????????????????????????????? , ]) Then as Chris says (sorry to keep banging the drum), the lines can trivially be reordered, and adding more lines never causes a problem as long as I stick to the rule.? Which I do automatically because I think my code looks prettier that way. From a purist angle, implicit string concatenation is somewhat inelegant (where else in Python can you have two adjacent operands not separated by an operator/comma/whatever?? We don't use reverse Polish notation).? And I could live without it.? But I have found it useful from time to time: constructing SQL queries or error messages or other long strings that I needed for some reason, where triple quotes would be more awkward (and I find line continuation backslashes ugly, *especially* in mid-string).? I guess my attitude is: "If you want to read my Python code, 90%+ of the time it will be *obvious* that these strings are *meant* to be concatenated.? But if it isn't, then you need to learn some Python basics first (Sorry!).". I have never as far as I can remember had a bug caused by string concatenation.? However, it is possible that I am guilty of selective memory. +1, though, for linters to point out possible errors from possible accidental omission of a comma (if they don't already).? It never hurts to have our code checked. Best wishes Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From turnbull.stephen.fw at u.tsukuba.ac.jp Fri Mar 16 00:36:31 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Fri, 16 Mar 2018 13:36:31 +0900 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: <20180315234644.GD16661@ando.pearwood.info> References: <20180315234644.GD16661@ando.pearwood.info> Message-ID: <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> Steven D'Aprano writes: > Indeed. I've been considering quantiles and quartiles for a long time, > and I've found at least ten different definitions for quantiles and > sixteen for quartiles. My take is "don't let the perfect be the enemy of the good." Quantiles are used a lot for "government work". Pick a definition that's good enough for that. Pick one that is commonly used and has nice invariance properties if any are applicable (e.g., quartile1(list) == quartile3(reversed(list), although I'm not even sure that is appropriate!), document it carefully, and give (some) examples of the edge cases that affect comparability of the statistics module's computation to alternative formulae. I'd like to see your list written up. To my mind, it would be of enough general interest that you could probably publish the results of your research in a political science or maybe psychology journal, or a statistics journal oriented to practitioners and/or educators[1]. I'm not suggesting you should do the work involved in actually submitting unless there's benefit to you in it, I'm just saying I think it's that interesting. Regards, Steve Footnotes: [1] I'd find it useful in talking to my students about trusting computers, for one thing. From liik.joonas at gmail.com Fri Mar 16 07:34:30 2018 From: liik.joonas at gmail.com (Joonas Liik) Date: Fri, 16 Mar 2018 13:34:30 +0200 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> Message-ID: If there it is desireable to have pathlib used to represent paths that do not map directly to the filesystem.. then it might be an acceptable compromise to have yet another... package that just imports os, pathlib, shutil etc and re-exports all relevant functions. i mean we are talking about convenience here anyway not new functionality and this would enable 1-import convenience that gives you all-you-would-ever-want but at the cost of maybe a bit more startup time if you only needed 1 of the 3 modules that do actual work. and avoid polluting pathlib with things that are... too useful in the real world :P -------------- next part -------------- An HTML attachment was scrubbed... URL: From guillaumeplique at gmail.com Fri Mar 16 10:32:55 2018 From: guillaumeplique at gmail.com (PLIQUE Guillaume) Date: Fri, 16 Mar 2018 15:32:55 +0100 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> References: <20180315234644.GD16661@ando.pearwood.info> <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> Message-ID: That's really interesting. I did not know there were so many way to consider quantiles. Maybe we should indeed wait for numpy to take a decision on the matter and go with their default choice so we remain consistent with the ecosystem? 2018-03-16 5:36 GMT+01:00 Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp>: > Steven D'Aprano writes: > > > Indeed. I've been considering quantiles and quartiles for a long time, > > and I've found at least ten different definitions for quantiles and > > sixteen for quartiles. > > My take is "don't let the perfect be the enemy of the good." > Quantiles are used a lot for "government work". Pick a definition > that's good enough for that. > > Pick one that is commonly used and has nice invariance properties if > any are applicable (e.g., quartile1(list) == quartile3(reversed(list), > although I'm not even sure that is appropriate!), document it > carefully, and give (some) examples of the edge cases that affect > comparability of the statistics module's computation to alternative > formulae. > > I'd like to see your list written up. To my mind, it would be of > enough general interest that you could probably publish the results of > your research in a political science or maybe psychology journal, or a > statistics journal oriented to practitioners and/or educators[1]. I'm > not suggesting you should do the work involved in actually submitting > unless there's benefit to you in it, I'm just saying I think it's that > interesting. > > Regards, > > Steve > > > Footnotes: > [1] I'd find it useful in talking to my students about trusting > computers, for one thing. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From marcidy at gmail.com Fri Mar 16 11:47:19 2018 From: marcidy at gmail.com (Matt Arcidy) Date: Fri, 16 Mar 2018 08:47:19 -0700 Subject: [Python-ideas] Descouraging the implicit string concatenation In-Reply-To: References: <20180314131552.GX18868@ando.pearwood.info> Message-ID: On Thu, Mar 15, 2018 at 8:58 PM, Rob Cliffe via Python-ideas wrote: > > > On 14/03/2018 17:57, Chris Angelico wrote: > > On Thu, Mar 15, 2018 at 12:40 AM, S?ren Pilg?rd > wrote: > > Of course you can always make error, even in a single letter. > But I think there is a big difference between mixing up +-/* and ** > where the operator is in "focus" and the implicit concatenation where > there is no operator. > A common problem is that you have something like > foo(["a", > "b", > "c" > ]).bar() > but then you remember that there also needs to be a "d" so you just add > foo(["a", > "b", > "c" > "d" > ]).bar() > Causing an error, not with the "d" expression you are working on but > due to what you thought was the previous expression but python turns > it into one. > The , is seen as a delimiter by the programmer not as part of the > operation (or the lack of the ,). > > You're creating a list. Put a comma at the end of every line; problem > solved. Your edit would be from this: > > foo(["a", > "b", > "c", > ]).bar() > > to this: > > foo(["a", > "b", > "c", > "d", > ]).bar() > > and there is no bug. In fact, EVERY time you lay out a list display > vertically (or dict or set or any equivalent), ALWAYS put commas. That > way, you can reorder the lines freely (you don't special-case the last > one), you can append a line without changing the previous one (no > noise in the diff), etc, etc, etc. > > > My thoughts exactly. I make it a personal rule to ALWAYS add a comma to > every line, including the last one, in this kind of list (/dict/set etc.). > Python allows it - take advantage of it! (A perhaps minor-seeming feature > of the language which actually is a big benefit if you use it.) Preferably > with all the commas vertically aligned to highlight the structure (I'm a > great believer BTW in using vertical alignment, even if it means violating > Python near-taboos such as more that one statement per line). Also I would > automatically put the first string (as well as the last) on a line by > itself: > foo([ > "short string" , > "extremely looooooooooooong string" , > "some other string" , > ]) > Then as Chris says (sorry to keep banging the drum), the lines can trivially > be reordered, and adding more lines never causes a problem as long as I > stick to the rule. Which I do automatically because I think my code looks > prettier that way. > > From a purist angle, implicit string concatenation is somewhat inelegant > (where else in Python can you have two adjacent operands not separated by an > operator/comma/whatever? We don't use reverse Polish notation). And I > could live without it. But I have found it useful from time to time: > constructing SQL queries or error messages or other long strings that I > needed for some reason, where triple quotes would be more awkward (and I > find line continuation backslashes ugly, *especially* in mid-string). I > guess my attitude is: "If you want to read my Python code, 90%+ of the time > it will be *obvious* that these strings are *meant* to be concatenated. But > if it isn't, then you need to learn some Python basics first (Sorry!).". > > I have never as far as I can remember had a bug caused by string > concatenation. However, it is possible that I am guilty of selective > memory. > > +1, though, for linters to point out possible errors from possible > accidental omission of a comma (if they don't already). It never hurts to > have our code checked. The linters I know use parsed ast nodes, so if it's not valid grammar, it won't parse. The _linters_ don't check for cases like f(a b) because that's not valid grammar and already caught by the parser. I think that's what you were noting when you said "if they don't already"? As for the issue in general, this is my understanding of linters: The code is loaded immediate into the ast parser. There's a no post-hoc way to know why the node is a string literal. Specifically the nodes for "ab" and "a""b" are identical Str nodes. Reversing from the Str node is impossible (unless a flag/attribute/context/whatever gets added), as the information is destroyed by then. The following might be possible: 1) The line number for a node n1 provides a way select the code between nodes n1 and the next node n2, which contains the offending string. This code needs to be retrieved from somewhere (easily kept from the beginning if not already) 2) A quick reparse of just that chunk will confirm that it contains the target node, so the code retrieval for the target node can be sanity checked (it's not exact retrieval of just the code we want, it's guaranteed to overshoot as resolution is on line numbers.) 3) ? (note below) 4) A quick check shows the tokenizer can differentiate the styles of literal. It gives 2 STRING for 'a''b' and 1 for 'ab'. This is a reliable test if the right code can be found which matches the target string. Hopefully there are others better ways but at least one exists. (better in the sense that the linters i know do not including tokenization currently) 3) The biggest problem for me (hopefully someone just knows the answer) is that some other very reliable parsing is required to know that the literal being _tested_ is the literal we _mean_ to test. I can use ast parser in 2) because im confirming a different piece of information. The problem is that the literal can exist in a very complex section, and coincidentally with 'a''b' and 'ab' in the same expression. The ast node won't tell is if we are looking for 0, 1 or 2 cases of either syntax, we only get 2 Str nodes and 1 line number. I think the parser chosen would pretty much have to be a rewrite of the ast parser due to nesting. Or someone needs to root around in the internals of the ast parser to see if the information can be extracted at the right time. I hope this helps in some way. I don't think it's impossible, but the above will introduce annoying bits into existing linters for this one issue. Of course, a working example would certainly make any case a lot easier. Given there is no technical reason, I agree there's no reason to change anything. This just "feels" ugly to me, so i can tilt at windmills all day on it, but I can see no technical reason. Always amazes me that string are so weird. They are literals and lists, but also multi white space jumping zebras. Are they "multi" "white space" "jumping zebras"? "multi" "white space jumping" "zebras"? "multi white" "space jumping" "zebras? We'll never know! -Matt > > Best wishes > Rob Cliffe > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Mar 17 02:18:27 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 17 Mar 2018 15:18:27 +0900 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> Message-ID: <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> Joonas Liik writes: > then it might be an acceptable compromise to have yet another... "There should be one-- and preferably only one -- obvious way to do it." The obvious way is to use the existing stdlib modules. So.... > package that just imports os, pathlib, shutil etc and re-exports > all relevant functions. Anybody wanting this can easily do a better job than the stdlib ever can do -- by writing a package including all the modules they frequently use and only re-exporting those names that they use, perhaps with shorter or personally mnemonic aliases (Windows vs. Unix nomenclature for many shell utilities, for example -- but watch those builtins like "dir"!) Of course, this is horrible programming practice, making for burdens on reviewers and maintainers. I see the convenience for writing one-off scripts and perhaps a personal library, but I don't see it as a core function of the language and its standard library. So, -1 in the stdlib, and +0 for a base module on PyPI that demonstrates the principle and individual users could modify to personal taste. Steve From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Mar 17 02:19:04 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sat, 17 Mar 2018 15:19:04 +0900 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: References: <20180315234644.GD16661@ando.pearwood.info> <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> Message-ID: <23212.45912.852632.751692@turnbull.sk.tsukuba.ac.jp> PLIQUE Guillaume writes: > That's really interesting. I did not know there were so many way to > consider quantiles. Maybe we should indeed wait for numpy to take a > decision on the matter and go with their default choice so we remain > consistent with the ecosystem? The example of R with 9 variants baked into one function suggests that numpy is unlikely to come up with a single "good" choice. If R's default is to Steven's taste, I would say go with that for cross- language consistency, and hope that numpy makes the same decision. In fact, I would argue that numpy might very well make a decision for a default that has nice mathematical properties, while the stdlib module might very well prefer consistency with R's default since defaults will be used in the same kind of "good enough for government work" contexts in both languages. The main thing is that they're all going to give similar results and in most applications the data will be fuzzy (eg, a sample or subjective), so as long as the same version is accurately documented and used consistently across analyses that should be comparable, results will be sufficiently accurate, perfectly reproducible, and comparable. For my purposes, there's no reason to wait. It's up to Steven, and I trust his taste. Steve From njs at pobox.com Sat Mar 17 02:48:14 2018 From: njs at pobox.com (Nathaniel Smith) Date: Fri, 16 Mar 2018 23:48:14 -0700 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: <23212.45912.852632.751692@turnbull.sk.tsukuba.ac.jp> References: <20180315234644.GD16661@ando.pearwood.info> <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> <23212.45912.852632.751692@turnbull.sk.tsukuba.ac.jp> Message-ID: On Fri, Mar 16, 2018 at 11:19 PM, Stephen J. Turnbull wrote: > PLIQUE Guillaume writes: > > > That's really interesting. I did not know there were so many way to > > consider quantiles. Maybe we should indeed wait for numpy to take a > > decision on the matter and go with their default choice so we remain > > consistent with the ecosystem? > > The example of R with 9 variants baked into one function suggests that > numpy is unlikely to come up with a single "good" choice. If R's > default is to Steven's taste, I would say go with that for cross- > language consistency, and hope that numpy makes the same decision. In > fact, I would argue that numpy might very well make a decision for a > default that has nice mathematical properties, while the stdlib module > might very well prefer consistency with R's default since defaults > will be used in the same kind of "good enough for government work" > contexts in both languages. NumPy already has a default and supports a number of variants. I'd have to go digging to figure out which languages/tools use which methods and how those match to theoretical properties, but IIRC numpy, R, and matlab all have different defaults. The 9 types that R supports come from a well-known review article (Hyndman & Fan, 1996). Their docs note that Hyndman & Fan's recommendation is different from the default, because the default was chosen to match a previous package (S) before they read Hyndman & Fan. It's all a bit messy. None of this is to say that Python shouldn't have some way to compute quantiles, but unfortunately you're not going to find TOOWTDI. -n -- Nathaniel J. Smith -- https://vorpus.org From davidfstr at gmail.com Sat Mar 17 02:49:14 2018 From: davidfstr at gmail.com (David Foster) Date: Fri, 16 Mar 2018 23:49:14 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: (1) I am concerned with the proposal's ability to introduce variables with a new broader kind of multi-line scope not seen anywhere else in Python. It is difficult to reason about, particularly in constructs like lambdas and inline def functions. Limiting scope to the very same line is great: > stuff = [[(f(x) as y), x/y] for x in range(5)] However I advocate that named expressions should ONLY be usable in the header of statements that enclose other statements. Thus I would expect the following to be an error: > if (re.match(...) as m): > print(m.groups(0)) # NameError: name 'm' is not defined > while (sock.read() as data): > print("Received data:", data) # NameError: name 'data' is not defined Using the named expression in other parts of the statement header is still fine: > if (re.match(...) as m) is not None and m.groups(1) == 'begin': > ... In summary, -1 from me for introducing a new kind of sublocal scope. +0 from me for keeping the scope limited to the header of the statement. Predictable conservative semantics. (2) If there WAS a desire to allow a named expression inside the substatements of a compound statement, I would advocate that the named expression just be treated as a regular (usually-)local variable assignment, with scope that extends to the entire function. This would imply that the variable would appear in locals(). Under this interpretation, the following would be okay: > while (sock.read() as data): > print("Received data:", data) > print("Last received data:", data) # ok; data is a regular local I would be -0 for using a regular local scope for a named expression. Predictable semantics but leaks the expression to the wider scope of the function. (3a) With a header-limited scope (in proposal #1 above), I advocate that a named expression should NOT be able to shadow other variables, giving a SyntaxError. I can't think of a reasonable reason why such shadowing should be allowed, and preventing shadowing may eliminate unintentional errors. (3b) With a local scope (in proposal #2 above), a named expression simply replaces any preexisting local with the same name. (Or it would replace the global/nonlocal if a global/nonlocal statement was in use.) There is no shadowing per-se. Cheers, -- David Foster | Seattle, WA, USA On Fri, Mar 2, 2018 at 3:43 AM, Chris Angelico wrote: > After dozens of posts and a wide variety of useful opinions and > concerns being raised, here is the newest version of PEP 572 for your > debating pleasure. > > Formatted version: > > https://www.python.org/dev/peps/pep-0572/ > > There are now several more examples, greater clarity in edge cases, > and improved wording of the actual proposal and specifications. Also, > the reference implementation has been significantly enhanced, for > those who wish to try this themselves. > > ChrisA > > PEP: 572 > Title: Syntax for Statement-Local Name Bindings > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 28-Feb-2018 > Python-Version: 3.8 > Post-History: 28-Feb-2018, 02-Mar-2018 > > > Abstract > ======== > > Programming is all about reusing code rather than duplicating it. When > an expression needs to be used twice in quick succession but never again, > it is convenient to assign it to a temporary name with small scope. > By permitting name bindings to exist within a single statement only, we > make this both convenient and safe against name collisions. > > > Rationale > ========= > > When a subexpression is used multiple times in a list comprehension, there > are currently several ways to spell this, none of which is universally > accepted as ideal. A statement-local name allows any subexpression to be > temporarily captured and then used multiple times. > > Additionally, this syntax can in places be used to remove the need to > write an > infinite loop with a ``break`` in it. Capturing part of a ``while`` loop's > condition can improve the clarity of the loop header while still making the > actual value available within the loop body. > > > Syntax and semantics > ==================== > > In any context where arbitrary Python expressions can be used, a named > expression can appear. This must be parenthesized for clarity, and is of > the form ``(expr as NAME)`` where ``expr`` is any valid Python expression, > and ``NAME`` is a simple name. > > The value of such a named expression is the same as the incorporated > expression, with the additional side-effect that NAME is bound to that > value in all retrievals for the remainder of the current statement. > > Just as function-local names shadow global names for the scope of the > function, statement-local names shadow other names for that statement. > They can also shadow each other, though actually doing this should be > strongly discouraged in style guides. > > Assignment to statement-local names is ONLY through this syntax. Regular > assignment to the same name will remove the statement-local name and > affect the name in the surrounding scope (function, class, or module). > > Statement-local names never appear in locals() or globals(), and cannot be > closed over by nested functions. > > > Execution order and its consequences > ------------------------------------ > > Since the statement-local name binding lasts from its point of execution > to the end of the current statement, this can potentially cause confusion > when the actual order of execution does not match the programmer's > expectations. Some examples:: > > # A simple statement ends at the newline or semicolon. > a = (1 as y) > print(y) # NameError > > # The assignment ignores the SLNB - this adds one to 'a' > a = (a + 1 as a) > > # Compound statements usually enclose everything... > if (re.match(...) as m): > print(m.groups(0)) > print(m) # NameError > > # ... except when function bodies are involved... > if (input("> ") as cmd): > def run_cmd(): > print("Running command", cmd) # NameError > > # ... but function *headers* are executed immediately > if (input("> ") as cmd): > def run_cmd(cmd=cmd): # Capture the value in the default arg > print("Running command", cmd) # Works > > Some of these examples should be considered *bad code* and rejected by code > review and/or linters; they are not, however, illegal. > > > Example usage > ============= > > These list comprehensions are all approximately equivalent:: > > # Calling the function twice > stuff = [[f(x), x/f(x)] for x in range(5)] > > # External helper function > def pair(x, value): return [value, x/value] > stuff = [pair(x, f(x)) for x in range(5)] > > # Inline helper function > stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)] > > # Extra 'for' loop - see also Serhiy's optimization > stuff = [[y, x/y] for x in range(5) for y in [f(x)]] > > # Iterating over a genexp > stuff = [[y, x/y] for x, y in ((x, f(x)) for x in range(5))] > > # Expanding the comprehension into a loop > stuff = [] > for x in range(5): > y = f(x) > stuff.append([y, x/y]) > > # Wrapping the loop in a generator function > def g(): > for x in range(5): > y = f(x) > yield [y, x/y] > stuff = list(g) > > # Using a statement-local name > stuff = [[(f(x) as y), x/y] for x in range(5)] > > If calling ``f(x)`` is expensive or has side effects, the clean operation > of > the list comprehension gets muddled. Using a short-duration name binding > retains the simplicity; while the extra ``for`` loop does achieve this, it > does so at the cost of dividing the expression visually, putting the named > part at the end of the comprehension instead of the beginning. > > Statement-local name bindings can be used in any context, but should be > avoided where regular assignment can be used, just as ``lambda`` should be > avoided when ``def`` is an option. As the name's scope extends to the full > current statement, even a block statement, this can be used to good effect > in the header of an ``if`` or ``while`` statement:: > > # Current Python, not caring about function return value > while input("> ") != "quit": > print("You entered a command.") > > # Current Python, capturing return value - four-line loop header > while True: > command = input("> "); > if command == "quit": > break > print("You entered:", command) > > # Proposed alternative to the above > while (input("> ") as command) != "quit": > print("You entered:", command) > > # See, for instance, Lib/pydoc.py > if (re.search(pat, text) as match): > print("Found:", match.group(0)) > > while (sock.read() as data): > print("Received data:", data) > > Particularly with the ``while`` loop, this can remove the need to have an > infinite loop, an assignment, and a condition. It also creates a smooth > parallel between a loop which simply uses a function call as its condition, > and one which uses that as its condition but also uses the actual value. > > > Performance costs > ================= > > The cost of SLNBs must be kept to a minimum, particularly when they are not > used; the normal case MUST NOT be measurably penalized. SLNBs are expected > to be uncommon, and using many of them in a single function should > definitely > be discouraged. Thus the current implementation uses a linked list of SLNB > cells, with the absence of such a list being the normal case. This list is > used for code compilation only; once a function's bytecode has been baked > in, > execution of that bytecode has no performance cost compared to regular > assignment. > > Other Python implementations may choose to do things differently, but a > zero > run-time cost is strongly recommended, as is a minimal compile-time cost in > the case where no SLNBs are used. > > > Open questions > ============== > > 1. What happens if the name has already been used? ``(x, (1 as x), x)`` > Currently, prior usage functions as if the named expression did not > exist (following the usual lookup rules); the new name binding will > shadow the other name from the point where it is evaluated until the > end of the statement. Is this acceptable? Should it raise a syntax > error or warning? > > 2. Syntactic confusion in ``except`` statements. While technically > unambiguous, it is potentially confusing to humans. In Python 3.7, > parenthesizing ``except (Exception as e):`` is illegal, and there is no > reason to capture the exception type (as opposed to the exception > instance, as is done by the regular syntax). Should this be made > outright illegal, to prevent confusion? Can it be left to linters? > It may also (and independently) be of value to use a subscope for the > normal except clause binding, such that ``except Exception as e:`` will > no longer unbind a previous use of the name ``e``. > > 3. Similar confusion in ``with`` statements, with the difference that there > is good reason to capture the result of an expression, and it is also > very common for ``__enter__`` methods to return ``self``. In many > cases, > ``with expr as name:`` will do the same thing as ``with (expr as > name):``, > adding to the confusion. > > 4. Should closures be able to refer to statement-local names? Either way, > there will be edge cases that make no sense. Assigning to a name will > "push through" the SLNB and bind to the regular name; this means that a > statement ``x = x`` will promote the SLNB to full name, and thus has an > impact. Closing over statement-local names, however, introduces scope > and lifetime confusions, as it then becomes possible to have two > functions > in almost the same context, closing over the same name, referring to two > different cells. > > > Alternative proposals > ===================== > > Proposals of this nature have come up frequently on python-ideas. Below are > a number of alternative syntaxes, some of them specific to comprehensions, > which have been rejected in favour of the one given above. > > 1. ``where``, ``let``, ``given``:: > > stuff = [(y, x/y) where y = f(x) for x in range(5)] > stuff = [(y, x/y) let y = f(x) for x in range(5)] > stuff = [(y, x/y) given y = f(x) for x in range(5)] > > This brings the subexpression to a location in between the 'for' loop > and > the expression. It introduces an additional language keyword, which > creates > conflicts. Of the three, ``where`` reads the most cleanly, but also has > the > greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` > methods, as does ``tkinter.dnd.Icon`` in the standard library). > > 2. ``with``:: > > stuff = [(y, x/y) with y = f(x) for x in range(5)] > > As above, but reusing the `with` keyword. Doesn't read too badly, and > needs > no additional language keyword. Is restricted to comprehensions, though, > and cannot as easily be transformed into "longhand" for-loop syntax. Has > the C problem that an equals sign in an expression can now create a name > binding, rather than performing a comparison. > > 3. ``with... as``:: > > stuff = [(y, x/y) with f(x) as y for x in range(5)] > > As per option 2, but using ``as`` in place of the equals sign. Aligns > syntactically with other uses of ``as`` for name binding, but a simple > transformation to for-loop longhand would create drastically different > semantics; the meaning of ``with`` inside a comprehension would be > completely different from the meaning as a stand-alone statement. > > 4. ``EXPR as NAME`` without parentheses:: > > stuff = [[f(x) as y, x/y] for x in range(5)] > > Omitting the parentheses from this PEP's proposed syntax introduces many > syntactic ambiguities. > > 5. Adorning statement-local names with a leading dot:: > > stuff = [[(f(x) as .y), x/.y] for x in range(5)] > > This has the advantage that leaked usage can be readily detected, > removing > some forms of syntactic ambiguity. However, this would be the only > place > in Python where a variable's scope is encoded into its name, making > refactoring harder. This syntax is quite viable, and could be promoted > to > become the current recommendation if its advantages are found to > outweigh > its cost. > > 6. Allowing ``(EXPR as NAME)`` to assign to any form of name. > > This is exactly the same as the promoted proposal, save that the name is > bound in the same scope that it would otherwise have. Any expression can > assign to any name, just as it would if the ``=`` operator had been > used. > > > Discrepancies in the current implementation > =========================================== > > 1. SLNBs are implemented using a special (and mostly-invisible) name > mangling. They may sometimes appear in globals() and/or locals() with > their simple or mangled names (but buggily and unreliably). They should > be suppressed as though they were guinea pigs. > > > References > ========== > > .. [1] Proof of concept / reference implementation > (https://github.com/Rosuav/cpython/tree/statement-local-variables) > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- David Foster -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Mar 17 03:13:18 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 17 Mar 2018 18:13:18 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On Sat, Mar 17, 2018 at 5:49 PM, David Foster wrote: > (3a) With a header-limited scope (in proposal #1 above), I advocate that a > named expression should NOT be able to shadow other variables, giving a > SyntaxError. I can't think of a reasonable reason why such shadowing should > be allowed, and preventing shadowing may eliminate unintentional errors. Header-limited scope is hard to define. Do you mean expression-local? (Also hard to define.) Do you mean local to one line of source code? Definitely not. And what happens with a 'for' loop - part of its header gets run after each loop iteration but still kinda references stuff that was done once-only before the loop started. ChrisA From george at fischhof.hu Sat Mar 17 06:33:34 2018 From: george at fischhof.hu (George Fischhof) Date: Sat, 17 Mar 2018 11:33:34 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> Message-ID: 2018-03-17 7:18 GMT+01:00 Stephen J. Turnbull < turnbull.stephen.fw at u.tsukuba.ac.jp>: > Joonas Liik writes: > > > then it might be an acceptable compromise to have yet another... > > "There should be one-- and preferably only one -- obvious way to do it." > > The obvious way is to use the existing stdlib modules. So.... > > > package that just imports os, pathlib, shutil etc and re-exports > > all relevant functions. > > Anybody wanting this can easily do a better job than the stdlib ever > can do -- by writing a package including all the modules they > frequently use and only re-exporting those names that they use, > perhaps with shorter or personally mnemonic aliases (Windows vs. Unix > nomenclature for many shell utilities, for example -- but watch those > builtins like "dir"!) > > Of course, this is horrible programming practice, making for burdens > on reviewers and maintainers. I see the convenience for writing > one-off scripts and perhaps a personal library, but I don't see it as > a core function of the language and its standard library. So, -1 in > the stdlib, and +0 for a base module on PyPI that demonstrates the > principle and individual users could modify to personal taste. > > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > Hi Steve, as I wrote previuosly pathlib already contains 17 functions from os and shutil. It seems that the original idea was something like for my idea. Just it not finished yet, With adding the remaining functions it would become a nice and whole thing George -------------- next part -------------- An HTML attachment was scrubbed... URL: From george at fischhof.hu Sat Mar 17 06:42:55 2018 From: george at fischhof.hu (George Fischhof) Date: Sat, 17 Mar 2018 11:42:55 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 Message-ID: Hi folks, I added the list of functions to the proposal, here is the new version. George PEP: 9999 Title: Pathlib Module Should Contain All File Operations Author: George Fischhof Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Mar-2018 Python-Version: 3.8 Post-History: 12-Mar-2018, 17-Mar-2018 Abstract ======== This PEP proposes pathlib module to be a centralized place for all file-system related operations. Rationale ========= Right now we have several modules that contain functions related to file-system operations mainly the os, pathlib and shutil. For beginners it is quite hard to remember where can he / she find a function (copy resides in shutil, but the remove function can be found in the os module. (And sometimes developers with moderate experience have to check the documentation as well.) After the release of version 3.6 several methods became aware of path-like object. There are only a few ones which does not support the path-like object. After making these methods path-like object aware, these functions could be added to pathlib. With functions in pathlib the developers should not have to think on which method (function) can be found in which module. Makes the life easier. Implementation ============== For compatibility reasons the pathlib should contain wrappers to the original functions. The original functions should remain at their original place. (Or if pathlib contains the function, the original modules should have a wrapper to it.) List of mentioned functions =========================== * os.link => path.hardlink_to (should be named similarly to path.softlink_to) * os.mkfifo * os.readlink * os.remove * os.removedirs (shutil.rmtree has the same functionalaty) * os.truncate * shutil.copyfileobj * shutil.copyfile * shutil.copymode * shutil.copystat * shutil.copy * shutil.copy2 * shutil.copytree with shutil.ignore_patterns * shutil.move * shutil.disk_usage * shutil.chown All functions from os module accept path-like objects, and none of the shutil functions. Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sat Mar 17 11:56:17 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 17 Mar 2018 08:56:17 -0700 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: References: <20180315234644.GD16661@ando.pearwood.info> <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> <23212.45912.852632.751692@turnbull.sk.tsukuba.ac.jp> Message-ID: Since Python is not held to backwards compatibility with S, and for most datasets (and users) it doesn't matter much, why not ho with the default recommended by Hyndman & Fan? On Fri, Mar 16, 2018 at 11:48 PM, Nathaniel Smith wrote: > On Fri, Mar 16, 2018 at 11:19 PM, Stephen J. Turnbull > wrote: > > PLIQUE Guillaume writes: > > > > > That's really interesting. I did not know there were so many way to > > > consider quantiles. Maybe we should indeed wait for numpy to take a > > > decision on the matter and go with their default choice so we remain > > > consistent with the ecosystem? > > > > The example of R with 9 variants baked into one function suggests that > > numpy is unlikely to come up with a single "good" choice. If R's > > default is to Steven's taste, I would say go with that for cross- > > language consistency, and hope that numpy makes the same decision. In > > fact, I would argue that numpy might very well make a decision for a > > default that has nice mathematical properties, while the stdlib module > > might very well prefer consistency with R's default since defaults > > will be used in the same kind of "good enough for government work" > > contexts in both languages. > > NumPy already has a default and supports a number of variants. I'd > have to go digging to figure out which languages/tools use which > methods and how those match to theoretical properties, but IIRC numpy, > R, and matlab all have different defaults. > > The 9 types that R supports come from a well-known review article > (Hyndman & Fan, 1996). Their docs note that Hyndman & Fan's > recommendation is different from the default, because the default was > chosen to match a previous package (S) before they read Hyndman & Fan. > It's all a bit messy. > > None of this is to say that Python shouldn't have some way to compute > quantiles, but unfortunately you're not going to find TOOWTDI. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From davidfstr at gmail.com Sat Mar 17 12:10:32 2018 From: davidfstr at gmail.com (David Foster) Date: Sat, 17 Mar 2018 09:10:32 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: I mean approximately local to one line of source code. Perhaps the unpopular opinion based on your reaction. :) More specifically, for a simple statement (with no trailing colon), there is one scope enclosing everything in the statement. For a compound statement, composed of multiple clauses, where each clause has a header (ending with a colon) and a suite, there are N non-overlapping scopes, one scope for each of the N clause headers. The scope is limited to the header only and does not include the suite. In considering a 'for' loop, I'd advocate for keeping the scope of the expression_list separate from the target_list, since I can't think of a reasonable case where the target_list would want to reference something from the expression_list. So the following code would have a NameError for magic_index in the target_list: > # NameError: name 'magic index' is not defined > for my_array[magic_index] in list_of_lists[(f(...) as magic_index)]: > ... That's pretty bizarre code, using a fixed index of an array as an iteration variable. The only other type of 'for' loop target that might try to use a named expression from the expression_list is a slice expression, which would be even more bizarre code. Best to make bizarre cases into errors. Cheers, -- David Foster | Seattle, WA, USA On Sat, Mar 17, 2018 at 12:13 AM, Chris Angelico wrote: > On Sat, Mar 17, 2018 at 5:49 PM, David Foster wrote: > > (3a) With a header-limited scope (in proposal #1 above), I advocate that > a > > named expression should NOT be able to shadow other variables, giving a > > SyntaxError. I can't think of a reasonable reason why such shadowing > should > > be allowed, and preventing shadowing may eliminate unintentional errors. > > Header-limited scope is hard to define. Do you mean expression-local? > (Also hard to define.) Do you mean local to one line of source code? > Definitely not. And what happens with a 'for' loop - part of its > header gets run after each loop iteration but still kinda references > stuff that was done once-only before the loop started. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- David Foster -------------- next part -------------- An HTML attachment was scrubbed... URL: From tim.peters at gmail.com Sat Mar 17 12:28:49 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 17 Mar 2018 11:28:49 -0500 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: References: <20180315234644.GD16661@ando.pearwood.info> <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> <23212.45912.852632.751692@turnbull.sk.tsukuba.ac.jp> Message-ID: [Guido] > Since Python is not held to backwards compatibility with S, and for most > datasets (and users) it doesn't matter much, why not ho with the default > recommended by Hyndman & Fan? Here's Hyndman in 2016[1]: """ The main point of our paper was that statistical software should standardize the definition of a sample quantile for consistency. We listed 9 different methods that we found in various software packages, and argued for one of them (type 8). In that sense, the paper was a complete failure. No major software uses type 8 by default, and the diversity of definitions continues 20 years later. In fact, the paper may have had the opposite effect to what was intended. We drew attention to the many approaches to computing sample quantiles and several software products added them all as options. Our own quantile function for R allows all 9 to be computed, and has type 7 as default (for backwards consistency ? the price we had to pay to get R core to agree to include our function). """ Familiar & hilarious ;-) [1] https://robjhyndman.com/hyndsight/sample-quantiles-20-years-later/ From turnbull.stephen.fw at u.tsukuba.ac.jp Sat Mar 17 13:15:23 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Sun, 18 Mar 2018 02:15:23 +0900 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> Message-ID: <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> George Fischhof writes: > It seems that the original idea was something like for my idea. > Just it not finished yet, Antoine (author and maintainer of pathlib) is not the kind of developer who leaves things unfinished. In PEP 428, there's a hint that some shutil functionality could be added, but I really don't think he meant anything as broad as in your PEP. As far as I can recall, pathlib is intended from the beginning to (1) represent paths in hierarchical local filesystems as Paths, (2) manipulate individual Paths in various ways consistent with the semantics of a hierarchal filesystem, and (3) offer various ways to access the object denoted by a single Path. Its functionality is very complete as far as that goes. It does not contain methods to (4) operate on directories as collections (with the exception of the iterdir, glob, and rglob methods, which expose directory contents as iterators of Paths), (5) perform operations on several objects denoted by Paths at once (copy and its multiple operand variants), (6) perform process control or access process characteristics, (7) perform operations (eg, mounting partitions and flow control on TTYs) on devices (block or character), even if they can be accessed via paths in some filesystem as in POSIX, or (8) deal with users and other specialized OS objects. I conclude there never was any intention to overlap with os or shutil, except to the extent that they provide for any kind of path manipulation. Rather, I suppose the intent was to provide a substitute for os.path with a more convenient, complete, object-oriented API and consistent semantics, based on more than a decade of experience with os.path. Regards, Steve From mahmoud at hatnote.com Sat Mar 17 13:20:53 2018 From: mahmoud at hatnote.com (Mahmoud Hashemi) Date: Sat, 17 Mar 2018 10:20:53 -0700 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: References: <20180315234644.GD16661@ando.pearwood.info> <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> <23212.45912.852632.751692@turnbull.sk.tsukuba.ac.jp> Message-ID: Hahaha, that Hyndman story will never get old. FWIW, based on much informal polling, the most common intuition on the topic stems from elementary education: a median of an even-numbered set is the mean of the two central values. So, linear-weighted average on discontinuities seems to be least surprising. Whichever type is chosen, quantiles are often computed in sets. For instance, min/max/median, quartiles (+ interquartile range), and percentiles. Quantiles was one of the main reasons statsutils uses a class[1] to wrap datasets. Otherwise, there's a lot of work in resorting. All the galloping in the world isn't going to beat sorting once. :) Other calculations benefit from this cached approach, too. Variance is faster to calculate after calculating stddev, for instance, but if memory serves, quantiles are the most expensive for mid-sized datasets that don't call for pandas/numpy. [1]: http://boltons.readthedocs.io/en/latest/statsutils.html#boltons.statsutils.Stats On Sat, Mar 17, 2018 at 9:28 AM, Tim Peters wrote: > [Guido] > > Since Python is not held to backwards compatibility with S, and for most > > datasets (and users) it doesn't matter much, why not ho with the default > > recommended by Hyndman & Fan? > > Here's Hyndman in 2016[1]: > > """ > The main point of our paper was that statistical software should > standardize the definition of a sample quantile for consistency. We > listed 9 different methods that we found in various software packages, > and argued for one of them (type 8). In that sense, the paper was a > complete failure. No major software uses type 8 by default, and the > diversity of definitions continues 20 years later. In fact, the paper > may have had the opposite effect to what was intended. We drew > attention to the many approaches to computing sample quantiles and > several software products added them all as options. Our own quantile > function for R allows all 9 to be computed, and has type 7 as default > (for backwards consistency ? the price we had to pay to get R core to > agree to include our function). > """ > > Familiar & hilarious ;-) > > [1] https://robjhyndman.com/hyndsight/sample-quantiles-20-years-later/ > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Sat Mar 17 16:34:23 2018 From: barry at barrys-emacs.org (Barry) Date: Sat, 17 Mar 2018 20:34:23 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 In-Reply-To: References: Message-ID: > On 17 Mar 2018, at 10:42, George Fischhof wrote: > > Hi folks, > > I added the list of functions to the proposal, here is the new version. > > George > > > > > PEP: 9999 > Title: Pathlib Module Should Contain All File Operations > Author: George Fischhof > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 15-Mar-2018 > Python-Version: 3.8 > Post-History: 12-Mar-2018, 17-Mar-2018 > > > Abstract > ======== > > This PEP proposes pathlib module to be a centralized place for all > file-system related operations. > > > Rationale > ========= > > Right now we have several modules that contain functions related > to file-system operations mainly the os, pathlib and shutil. > For beginners it is quite hard to remember where can he / she find > a function (copy resides in shutil, but the remove function can be > found in the os module. (And sometimes developers with moderate > experience have to check the documentation as well.) > > After the release of version 3.6 several methods became aware of > path-like object. There are only a few ones which does not support > the path-like object. After making these methods path-like object > aware, these functions could be added to pathlib. > > With functions in pathlib the developers should not have to think > on which method (function) can be found in which module. > > Makes the life easier. > > > Implementation > ============== > > For compatibility reasons the pathlib should contain wrappers to > the original functions. The original functions should remain > at their original place. (Or if pathlib contains the function, the > original modules should have a wrapper to it.) > > > List of mentioned functions > =========================== > > * os.link => path.hardlink_to > (should be named similarly to path.softlink_to) > > * os.mkfifo > > * os.readlink > > * os.remove > > * os.removedirs > (shutil.rmtree has the same functionalaty) I think you will find these to are a lot different. > > * os.truncate > > * shutil.copyfileobj This function does not take paths as arguments. I guess it does not belong here. > > * shutil.copyfile > > * shutil.copymode > > * shutil.copystat > > * shutil.copy > > * shutil.copy2 > > * shutil.copytree with shutil.ignore_patterns > > * shutil.move > > * shutil.disk_usage > > * shutil.chown Is yout plan to also expose all the constants need to use chmod etc from pathlib? > > All functions from os module accept path-like objects, > and none of the shutil functions. Should shutils be improved to accept path-like first? Barry > > > Copyright > ========= > > This document has been placed in the public domain. > > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From george at fischhof.hu Sat Mar 17 18:49:00 2018 From: george at fischhof.hu (George Fischhof) Date: Sat, 17 Mar 2018 23:49:00 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 In-Reply-To: References: Message-ID: 2018. m?rc. 17. 21:34 ezt ?rta ("Barry" ): On 17 Mar 2018, at 10:42, George Fischhof wrote: Hi folks, I added the list of functions to the proposal, here is the new version. George PEP: 9999 Title: Pathlib Module Should Contain All File Operations Author: George Fischhof Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 15-Mar-2018 Python-Version: 3.8 Post-History: 12-Mar-2018, 17-Mar-2018 Abstract ======== This PEP proposes pathlib module to be a centralized place for all file-system related operations. Rationale ========= Right now we have several modules that contain functions related to file-system operations mainly the os, pathlib and shutil. For beginners it is quite hard to remember where can he / she find a function (copy resides in shutil, but the remove function can be found in the os module. (And sometimes developers with moderate experience have to check the documentation as well.) After the release of version 3.6 several methods became aware of path-like object. There are only a few ones which does not support the path-like object. After making these methods path-like object aware, these functions could be added to pathlib. With functions in pathlib the developers should not have to think on which method (function) can be found in which module. Makes the life easier. Implementation ============== For compatibility reasons the pathlib should contain wrappers to the original functions. The original functions should remain at their original place. (Or if pathlib contains the function, the original modules should have a wrapper to it.) List of mentioned functions =========================== * os.link => path.hardlink_to (should be named similarly to path.softlink_to) * os.mkfifo * os.readlink * os.remove * os.removedirs (shutil.rmtree has the same functionalaty) I think you will find these to are a lot different. >From high level, it is only question of implementation: the main goal is to remove a chain of directories * os.truncate * shutil.copyfileobj This function does not take paths as arguments. I guess it does not belong here. No path, it is true (right now), but it has sense to have this function among the others. * shutil.copyfile * shutil.copymode * shutil.copystat * shutil.copy * shutil.copy2 * shutil.copytree with shutil.ignore_patterns * shutil.move * shutil.disk_usage * shutil.chown Is yout plan to also expose all the constants need to use chmod etc from pathlib? Of course, because this PEP is about conveniece and easyness. All functions from os module accept path-like objects, and none of the shutil functions. Should shutils be improved to accept path-like first? Maybe. The PEP should show the goal. This is just a task among others to achieve the goal. But it can be done simultanuosly. George Barry Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From mikhailwas at gmail.com Sat Mar 17 21:04:43 2018 From: mikhailwas at gmail.com (Mikhail V) Date: Sun, 18 Mar 2018 02:04:43 +0100 Subject: [Python-ideas] Syntax idea for 2D lists\arrays In-Reply-To: <20180315051525.GB16661@ando.pearwood.info> References: <20180315051525.GB16661@ando.pearwood.info> Message-ID: On Thu, Mar 15, 2018 at 6:15 AM, Steven D'Aprano wrote: > On Thu, Mar 15, 2018 at 01:32:35AM +0100, Mikhail V wrote: > > > Using spaces to separate items has the fatal flaw that it cannot > distinguish > > x - y 0 # two items, the expression `x - y` and the integer 0 > > from: > > x - y 0 # three items, `x`, `-y`, and 0 > > Ah, sorry, my bad. Now I remember preciser what was my formatting idea originally. That would make things clearer indeed - the separator would be TAB character only. (I wrote tabs or spaces in proposal). The root of the idea was finding a cleaner way for defining tables and data plus some experiments with nesting syntax, etc. Back then I did not think much about supportin expressions inside elements, so this important issue slipped through. So with the TAB separator, just think of replacement TAB->comma, this should support all Python expressions automatically. At least seems to me so, but if I am delusional - please correct me. Of course the reality is still, that sadly, most editors cannot handle tabulaton adequately. But I am a believer and hope for better future. (Heck, people building space ships and what not, so maybe tabulation in code editors comes next?) From steve at pearwood.info Sat Mar 17 21:33:17 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 18 Mar 2018 12:33:17 +1100 Subject: [Python-ideas] Syntax idea for 2D lists\arrays In-Reply-To: References: <20180315051525.GB16661@ando.pearwood.info> Message-ID: <20180318013315.GI16661@ando.pearwood.info> On Sun, Mar 18, 2018 at 02:04:43AM +0100, Mikhail V wrote: > So with the TAB separator, just think of replacement TAB->comma, > this should support all Python expressions automatically. > At least seems to me so, but if I am delusional - please correct me. It is still ambiguous: py> eval("10\t-2") 8 -- Steve From tim.peters at gmail.com Sat Mar 17 23:07:19 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sat, 17 Mar 2018 22:07:19 -0500 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: References: <20180315234644.GD16661@ando.pearwood.info> <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> <23212.45912.852632.751692@turnbull.sk.tsukuba.ac.jp> Message-ID: [Guido] > Since Python is not held to backwards compatibility with S, and for most > datasets (and users) it doesn't matter much, why not ho with the default > recommended by Hyndman & Fan? BTW, I should clarify that I agree! H&F didn't invent "method 8", or any of the other methods their paper named, they just evaluated 9 methods with a keen eye. Their case for method 8 being "the best" seems pretty clear: it satisfies as least as many desirable formal properties as the other serious candidates, and is apparently optimal in some technical senses (related to avoiding bias) among methods that can't assume anything about the underlying distribution. For the median, looks like method 8 reduces to the usual "return the mean of the two middle values" for an even number of data points, which is the only "intuition" anyone brings to this ;-) So I'd make it the default, and add others later as options if there's enough screaming. It's the Right(est) Thing to Do. From njs at pobox.com Sun Mar 18 00:41:10 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sat, 17 Mar 2018 21:41:10 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> Message-ID: On Sat, Mar 17, 2018 at 10:15 AM, Stephen J. Turnbull wrote: > (5) perform operations on several objects denoted by Paths at once > (copy and its multiple operand variants), Sure it does: Path.rename and Path.replace. I know why rename and copy have historically been in separate modules, but the distinction is pretty arcane and matters a lot more to implementers than it does to users. Similarly, it's hard to explain why we have Path.mkdir but not Path.makedirs -- and these have historically both lived in the 'os' module, so we can't blame it on Path being a mirror of os.path. It's also not obvious why we should have Path.rmdir, but not Path.rmtree. My understanding is that the point of Path is to be a convenient, pleasant-to-use mechanism for accessing common filesystem operations. And it does a pretty excellent job of that. But it seems obvious to me that it's still missing a number of fairly basic operations that people need all the time. I don't think the PEP is there yet, and we can quibble over the details -- just copying over all the historical decisions in shutil isn't obviously the right move (maybe it should be Path.mkdir(include_parents=True) and Path.unlink(recursive=True) instead of Path.makedirs and Path.rmtree?), but there's definitely room for improvement. -n -- Nathaniel J. Smith -- https://vorpus.org From ncoghlan at gmail.com Sun Mar 18 04:05:58 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 18 Mar 2018 18:05:58 +1000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: On 16 March 2018 at 03:15, Chris Angelico wrote: > On Fri, Mar 16, 2018 at 12:38 AM, George Fischhof > wrote: > > > > > > " if new file functions are added, they will go only in pathlib, > > which makes pathlib effectively mandatory;" > > Yes but I think this part of the evolution: slowly everyone will shift to > > pathlib, > > and being mandatory is true for the current status as well: if you need a > > function, you need the module. > > Right now if you wan to execute some file operations, you need os plus > > shutil, because the half of the > > functions are in one of them, the other half is in the other module > > The os module is cheap; pathlib has a definite cost. If every file > operation goes through pathlib Keep in mind that the `os` layer will never go away: `pathlib` still needs a lower level API to call to *do the work* of actually interacting with the underlying operating system APIs (e.g. this is why we added os.scandir). A similar situation applies when it comes to glob, fnmatch, etc. Even `shutil` will likely retain its place as a lower level procedural API behind pathlib's object-oriented facade, since raw strings are still frequently going to be easier to work with when mixing and matching Python code and native operating system shell code. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Sun Mar 18 04:30:01 2018 From: barry at barrys-emacs.org (Barry) Date: Sun, 18 Mar 2018 08:30:01 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 In-Reply-To: References: Message-ID: <4FE81DB4-E8A1-4F93-8EEE-89331F7F2D9D@barrys-emacs.org> > On 17 Mar 2018, at 22:49, George Fischhof wrote: > > > > 2018. m?rc. 17. 21:34 ezt ?rta ("Barry" ): > > >> On 17 Mar 2018, at 10:42, George Fischhof wrote: >> >> Hi folks, >> >> I added the list of functions to the proposal, here is the new version. >> >> George >> >> >> >> >> PEP: 9999 >> Title: Pathlib Module Should Contain All File Operations >> Author: George Fischhof >> Status: Draft >> Type: Standards Track >> Content-Type: text/x-rst >> Created: 15-Mar-2018 >> Python-Version: 3.8 >> Post-History: 12-Mar-2018, 17-Mar-2018 >> >> >> Abstract >> ======== >> >> This PEP proposes pathlib module to be a centralized place for all >> file-system related operations. >> >> >> Rationale >> ========= >> >> Right now we have several modules that contain functions related >> to file-system operations mainly the os, pathlib and shutil. >> For beginners it is quite hard to remember where can he / she find >> a function (copy resides in shutil, but the remove function can be >> found in the os module. (And sometimes developers with moderate >> experience have to check the documentation as well.) >> >> After the release of version 3.6 several methods became aware of >> path-like object. There are only a few ones which does not support >> the path-like object. After making these methods path-like object >> aware, these functions could be added to pathlib. >> >> With functions in pathlib the developers should not have to think >> on which method (function) can be found in which module. >> >> Makes the life easier. >> >> >> Implementation >> ============== >> >> For compatibility reasons the pathlib should contain wrappers to >> the original functions. The original functions should remain >> at their original place. (Or if pathlib contains the function, the >> original modules should have a wrapper to it.) >> >> >> List of mentioned functions >> =========================== >> >> * os.link => path.hardlink_to >> (should be named similarly to path.softlink_to) >> >> * os.mkfifo >> >> * os.readlink >> >> * os.remove >> >> * os.removedirs >> (shutil.rmtree has the same functionalaty) > > I think you will find these to are a lot different. > > > From high level, it is only question of implementation: the main goal is to remove a chain of directories This is a PEP you and proposing details matter and this is one of the details tha you need be clear on. > >> >> * os.truncate >> >> * shutil.copyfileobj > > This function does not take paths as arguments. I guess it does not belong here. > > > No path, it is true (right now), but it has sense to have this function among the others. What do you mena right now? > > >> >> * shutil.copyfile >> >> * shutil.copymode >> >> * shutil.copystat >> >> * shutil.copy >> >> * shutil.copy2 >> >> * shutil.copytree with shutil.ignore_patterns >> >> * shutil.move >> >> * shutil.disk_usage >> >> * shutil.chown > > Is yout plan to also expose all the constants need to use chmod etc from pathlib? > > Of course, because this PEP is about conveniece and easyness. Can you say so then and not leave it in doubt. > >> >> All functions from os module accept path-like objects, >> and none of the shutil functions. > > Should shutils be improved to accept path-like first? > > Maybe. The PEP should show the goal. This is just a task among others to achieve the goal. But it can be done simultanuosly. > > George > > Barry > >> >> >> Copyright >> ========= >> >> This document has been placed in the public domain. >> >> >> >> .. >> Local Variables: >> mode: indented-text >> indent-tabs-mode: nil >> sentence-end-double-space: t >> fill-column: 70 >> coding: utf-8 >> End: >> >> >> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From george at fischhof.hu Sun Mar 18 07:13:28 2018 From: george at fischhof.hu (George Fischhof) Date: Sun, 18 Mar 2018 12:13:28 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> Message-ID: 2018-03-18 5:41 GMT+01:00 Nathaniel Smith : > On Sat, Mar 17, 2018 at 10:15 AM, Stephen J. Turnbull > wrote: > > (5) perform operations on several objects denoted by Paths at once > > (copy and its multiple operand variants), > > Sure it does: Path.rename and Path.replace. I know why rename and copy > have historically been in separate modules, but the distinction is > pretty arcane and matters a lot more to implementers than it does to > users. > > Similarly, it's hard to explain why we have Path.mkdir but not > Path.makedirs -- and these have historically both lived in the 'os' > module, so we can't blame it on Path being a mirror of os.path. It's > also not obvious why we should have Path.rmdir, but not Path.rmtree. > > My understanding is that the point of Path is to be a convenient, > pleasant-to-use mechanism for accessing common filesystem operations. > And it does a pretty excellent job of that. But it seems obvious to me > that it's still missing a number of fairly basic operations that > people need all the time. I don't think the PEP is there yet, and we > can quibble over the details -- just copying over all the historical > decisions in shutil isn't obviously the right move (maybe it should be > Path.mkdir(include_parents=True) and Path.unlink(recursive=True) > instead of Path.makedirs and Path.rmtree?), but there's definitely > room for improvement. > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > :-) + 1 George -------------- next part -------------- An HTML attachment was scrubbed... URL: From george at fischhof.hu Sun Mar 18 07:14:04 2018 From: george at fischhof.hu (George Fischhof) Date: Sun, 18 Mar 2018 12:14:04 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: 2018-03-18 9:05 GMT+01:00 Nick Coghlan : > On 16 March 2018 at 03:15, Chris Angelico wrote: > >> On Fri, Mar 16, 2018 at 12:38 AM, George Fischhof >> wrote: >> > >> > >> > " if new file functions are added, they will go only in pathlib, >> > which makes pathlib effectively mandatory;" >> > Yes but I think this part of the evolution: slowly everyone will shift >> to >> > pathlib, >> > and being mandatory is true for the current status as well: if you need >> a >> > function, you need the module. >> > Right now if you wan to execute some file operations, you need os plus >> > shutil, because the half of the >> > functions are in one of them, the other half is in the other module >> >> The os module is cheap; pathlib has a definite cost. If every file >> operation goes through pathlib > > > Keep in mind that the `os` layer will never go away: `pathlib` still needs > a lower level API to call to *do the work* of actually interacting with the > underlying operating system APIs (e.g. this is why we added os.scandir). > > A similar situation applies when it comes to glob, fnmatch, etc. > > Even `shutil` will likely retain its place as a lower level procedural API > behind pathlib's object-oriented facade, since raw strings are still > frequently going to be easier to work with when mixing and matching Python > code and native operating system shell code. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > :-) + 1 George -------------- next part -------------- An HTML attachment was scrubbed... URL: From storchaka at gmail.com Sun Mar 18 07:45:46 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Sun, 18 Mar 2018 13:45:46 +0200 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> Message-ID: 17.03.18 19:15, Stephen J. Turnbull ????: > As far as I can recall, pathlib is intended from the beginning to > (1) represent paths in hierarchical local filesystems as Paths, > (2) manipulate individual Paths in various ways consistent with the > semantics of a hierarchal filesystem, and > (3) offer various ways to access the object denoted by a single Path. > Its functionality is very complete as far as that goes. > > It does not contain methods to > (4) operate on directories as collections (with the exception of the > iterdir, glob, and rglob methods, which expose directory contents as > iterators of Paths), > (5) perform operations on several objects denoted by Paths at once > (copy and its multiple operand variants), > (6) perform process control or access process characteristics, > (7) perform operations (eg, mounting partitions and flow control on > TTYs) on devices (block or character), even if they can be > accessed via paths in some filesystem as in POSIX, or > (8) deal with users and other specialized OS objects. > > I conclude there never was any intention to overlap with os or shutil, > except to the extent that they provide for any kind of path > manipulation. Rather, I suppose the intent was to provide a > substitute for os.path with a more convenient, complete, > object-oriented API and consistent semantics, based on more than a > decade of experience with os.path. I concur with Stephen. Let's not move a half of the stdlib that have any relations with files to the pathlib module. From george at fischhof.hu Sun Mar 18 07:58:45 2018 From: george at fischhof.hu (George Fischhof) Date: Sun, 18 Mar 2018 12:58:45 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: Hi Folks, it seems for me that the welcoming of this proposal is rather positive than not. Of course several details could be put into it, but I think it would better to let the developers decide the details, because they know the environment and the possibilities. The name of the functions and the method they solve the problem (for example rmdir(tree=True9 instead of removedirs()) is all the same. The (main) goal would be that file and directory operations reside in one module. And currently the pathlib seems to be the best candidate. (we could put then into a very new module, but it would be just another duplicataion) So what do You think, this proposal IS PEPable or should I do something with this to achieve the PEPable state? BR, George 2018-03-18 9:05 GMT+01:00 Nick Coghlan : > On 16 March 2018 at 03:15, Chris Angelico wrote: > >> On Fri, Mar 16, 2018 at 12:38 AM, George Fischhof >> wrote: >> > >> > >> > " if new file functions are added, they will go only in pathlib, >> > which makes pathlib effectively mandatory;" >> > Yes but I think this part of the evolution: slowly everyone will shift >> to >> > pathlib, >> > and being mandatory is true for the current status as well: if you need >> a >> > function, you need the module. >> > Right now if you wan to execute some file operations, you need os plus >> > shutil, because the half of the >> > functions are in one of them, the other half is in the other module >> >> The os module is cheap; pathlib has a definite cost. If every file >> operation goes through pathlib > > > Keep in mind that the `os` layer will never go away: `pathlib` still needs > a lower level API to call to *do the work* of actually interacting with the > underlying operating system APIs (e.g. this is why we added os.scandir). > > A similar situation applies when it comes to glob, fnmatch, etc. > > Even `shutil` will likely retain its place as a lower level procedural API > behind pathlib's object-oriented facade, since raw strings are still > frequently going to be easier to work with when mixing and matching Python > code and native operating system shell code. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Mar 18 10:34:35 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 18 Mar 2018 14:34:35 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 In-Reply-To: References: Message-ID: On 17 March 2018 at 22:49, George Fischhof wrote: >>> >>> * os.truncate >>> >>> * shutil.copyfileobj >>> >>> >>> This function does not take paths as arguments. I guess it does not belong >>> here. > > No path, it is true (right now), but it has sense to have this function > among the others. So you're proposing to include these how? As standalone functions in pathlib? Strong -1 on that. I thought the intention here was to add methods to path objects (which I still have reservations about). But I'm definitely against adding standalone functions to pathlib - I see pathlib as where Path objects are defined, not as "the place where all path-related and other file operations should live"). Paul From p.f.moore at gmail.com Sun Mar 18 10:46:44 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 18 Mar 2018 14:46:44 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> Message-ID: On 18 March 2018 at 04:41, Nathaniel Smith wrote: > My understanding is that the point of Path is to be a convenient, > pleasant-to-use mechanism for accessing common filesystem operations. > And it does a pretty excellent job of that. But it seems obvious to me > that it's still missing a number of fairly basic operations that > people need all the time. IMO, the pathlib module (just) defines Path. So I'm -1 on adding anything to pathlib that isn't a method of a Path object. Beyond that, I agree with you that Path should be a convenient interface for filesystem path objects. I haven't personally found that there's much missing that I've needed, but I agree that there are some gaps from a theoretical point of view, and adding methods to fill those gaps could be justifiable. OTOH, the fspath protocol was explicitly designed so that standalone functions (such as the ones in os and shutil) can work cleanly with Path objects - so there's a strong argument that "not everything needs to be a method" applies here. For example, while there isn't a Path.makedirs(), what's so bad about os.makedirs(Path)? (There's consistency and discoverability arguments, but they are not what I'd call compelling on their own). > I don't think the PEP is there yet, and we > can quibble over the details -- just copying over all the historical > decisions in shutil isn't obviously the right move (maybe it should be > Path.mkdir(include_parents=True) and Path.unlink(recursive=True) > instead of Path.makedirs and Path.rmtree?), but there's definitely > room for improvement. I agree that there are some potential candidates for "useful additional methods for Path objects", but I'd like to see these discussed on a case by case basis, much like you do here, rather than as a blanket "if it's in some other module and it works on paths, it should be in pathlib. My biggest problem with the proposal as it stands is that it makes no attempt to justify the suggestions on a case by case basis (the first version wasn't even explicit in the functions it was proposing!) but argues from a pure "lump everything together" standpoint. Paul From barry at barrys-emacs.org Sun Mar 18 10:44:19 2018 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 18 Mar 2018 14:44:19 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 In-Reply-To: References: Message-ID: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> It seems that the main quest to answer first is this. Should Path() have methods to access all file operations? Maybe it was a mistake to add the ones that are already there. Especially in light of the fspath protocol that now exists. If yes proceed to details the methods and their semantics. If no then concentrate on making shutil etc easy to use this Path() args. Maybe do this anyway. Barry From barry at barrys-emacs.org Sun Mar 18 13:51:16 2018 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 18 Mar 2018 17:51:16 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: <69571784-B513-4689-A59D-6B9BCE2C07D9@barrys-emacs.org> > On 18 Mar 2018, at 11:58, George Fischhof wrote: > > Hi Folks, > > it seems for me that the welcoming of this proposal is rather positive than not. I think that is up for debate. > > Of course several details could be put into it, but I think it would better to let the developers decide the details, because they know the environment and the possibilities. You mean you have no intention of doing the implementation? If not you who is willing to the go the, not inconsiderable, work. > > The name of the functions and the method they solve the problem (for example rmdir(tree=True9 instead of removedirs()) is all the same. But it is not the same. os.removedirs only removes dirs, where as shutil.rmdir will remove files and dirs. > > The (main) goal would be that file and directory operations reside in one module. As I said earlier the question is should that module be pathlib? There are good arguments on both side. Barry > And currently the pathlib seems to be the best candidate. > (we could put then into a very new module, but it would be just another duplicataion) > > > So what do You think, this proposal IS PEPable or should I do something with this to achieve the PEPable state? > > > BR, > George > > 2018-03-18 9:05 GMT+01:00 Nick Coghlan >: > On 16 March 2018 at 03:15, Chris Angelico > wrote: > On Fri, Mar 16, 2018 at 12:38 AM, George Fischhof > wrote: > > > > > > " if new file functions are added, they will go only in pathlib, > > which makes pathlib effectively mandatory;" > > Yes but I think this part of the evolution: slowly everyone will shift to > > pathlib, > > and being mandatory is true for the current status as well: if you need a > > function, you need the module. > > Right now if you wan to execute some file operations, you need os plus > > shutil, because the half of the > > functions are in one of them, the other half is in the other module > > The os module is cheap; pathlib has a definite cost. If every file > operation goes through pathlib > > Keep in mind that the `os` layer will never go away: `pathlib` still needs a lower level API to call to *do the work* of actually interacting with the underlying operating system APIs (e.g. this is why we added os.scandir). > > A similar situation applies when it comes to glob, fnmatch, etc. > > Even `shutil` will likely retain its place as a lower level procedural API behind pathlib's object-oriented facade, since raw strings are still frequently going to be easier to work with when mixing and matching Python code and native operating system shell code. > > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjmaldonis at gmail.com Sun Mar 18 13:50:58 2018 From: jjmaldonis at gmail.com (Jason Maldonis) Date: Sun, 18 Mar 2018 12:50:58 -0500 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> Message-ID: Maybe this is obvious or I am missing something crucial, but I'm surprised that this hasn't been discussed yet: >From a user perspective, imo the problem is that users currently need three modules (pathlib, os, and shutil) to have a nice interface for working with, copying, and removing files. In general, it seems to me that there should be two modules for file-like operations: a fast, low-level module, and a (slower) high-level module that provides a more natural interface than the low-level module. It seems natural that the high-level module should simply use the low-level module to do the file operators, and just provide a nice (probably object-oriented) interface for those methods. In python, `os` and `shutil` are currently the low-level modules, and it stands to reason that we might consider combining these somehow (although I'm assuming that there was a good reason not to in the first place, which is why they both exist, but I haven't looked into it). And `pathlib` is currently the "high-level" module. I see two problems currently: 1) the low-level module is split in half (for example, operations for copying are contained in `shutil` and operations for removing are contained in `os`). This is a bit annoying for the user, but it's not game-breaking. It does, however, make python feel a bit unnatural in this context, and that's even more unusual because normally python feels very natural. So this becomes sort of a "huh this feels weird" situation. 2) The `pathlib` modules only provides a high-level interface for working with _single_ Path objects. There isn't really functionality to work with multiple Path objects (as pointed out by someone previously if I am understanding correctly). I don't think the current PEP under consideration adequately solves either of these problems. Currently, it seems like it's trying to make `pathlib` both a high- and low-level module, which imo doesn't make sense. But I do think we need, if not a single low-level module, at least a high-level module that makes it unnecessary to use the low-level modules. That means that `pathlib` needs more functionality added to it, which is similar in spirit to the current PEP proposal. - Jason, a reader On Sun, Mar 18, 2018 at 9:46 AM, Paul Moore wrote: > On 18 March 2018 at 04:41, Nathaniel Smith wrote: > > My understanding is that the point of Path is to be a convenient, > > pleasant-to-use mechanism for accessing common filesystem operations. > > And it does a pretty excellent job of that. But it seems obvious to me > > that it's still missing a number of fairly basic operations that > > people need all the time. > > IMO, the pathlib module (just) defines Path. So I'm -1 on adding > anything to pathlib that isn't a method of a Path object. Beyond that, > I agree with you that Path should be a convenient interface for > filesystem path objects. I haven't personally found that there's much > missing that I've needed, but I agree that there are some gaps from a > theoretical point of view, and adding methods to fill those gaps could > be justifiable. OTOH, the fspath protocol was explicitly designed so > that standalone functions (such as the ones in os and shutil) can work > cleanly with Path objects - so there's a strong argument that "not > everything needs to be a method" applies here. For example, while > there isn't a Path.makedirs(), what's so bad about os.makedirs(Path)? > (There's consistency and discoverability arguments, but they are not > what I'd call compelling on their own). > > > I don't think the PEP is there yet, and we > > can quibble over the details -- just copying over all the historical > > decisions in shutil isn't obviously the right move (maybe it should be > > Path.mkdir(include_parents=True) and Path.unlink(recursive=True) > > instead of Path.makedirs and Path.rmtree?), but there's definitely > > room for improvement. > > I agree that there are some potential candidates for "useful > additional methods for Path objects", but I'd like to see these > discussed on a case by case basis, much like you do here, rather than > as a blanket "if it's in some other module and it works on paths, it > should be in pathlib. > > My biggest problem with the proposal as it stands is that it makes no > attempt to justify the suggestions on a case by case basis (the first > version wasn't even explicit in the functions it was proposing!) but > argues from a pure "lump everything together" standpoint. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Sun Mar 18 13:55:32 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sun, 18 Mar 2018 17:55:32 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 In-Reply-To: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> Message-ID: On 18 March 2018 at 14:44, Barry Scott wrote: > It seems that the main quest to answer first is this. > > Should Path() have methods to access all file operations? No, (Counterexample, having a Path operation to set Windows ACLs for a path). > Maybe it was a mistake to add the ones that are already there. Seems unlikely, as Path objects are pretty useful in their current form. Even if it were, in some cases, backward compatibility dictates keeping them so who cares? > Especially in light of the fspath protocol that now exists. Again, *maybe* the arguments for some of the existing methods were weakened by the existence of the fspath protocol, but not enough to override backward compatibility, so who cares? > If yes proceed to details the methods and their semantics. Not applicable, see above. > If no then concentrate on making shutil etc easy to use this Path() args. Wrong conclusion - just because it shouldn't have methods for *all* doesn't imply that *nothing* new warrants being a Path method. Just that cases should be considered on their individual merits. > Maybe do this anyway. Certainly this is a reasonable thing to propose, Probably doesn't even need much discussion, an issue on bpo should be sufficient. Paul From barry at barrys-emacs.org Sun Mar 18 14:29:56 2018 From: barry at barrys-emacs.org (Barry Scott) Date: Sun, 18 Mar 2018 18:29:56 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> Message-ID: <5959B700-AFE4-4CCF-9F96-3DB79C5EF5A5@barrys-emacs.org> > On 18 Mar 2018, at 17:50, Jason Maldonis wrote: > > Maybe this is obvious or I am missing something crucial, but I'm surprised that this hasn't been discussed yet: > > From a user perspective, imo the problem is that users currently need three modules (pathlib, os, and shutil) to have a nice interface for working with, copying, and removing files. In general, it seems to me that there should be two modules for file-like operations: a fast, low-level module, and a (slower) high-level module that provides a more natural interface than the low-level module. It seems natural that the high-level module should simply use the low-level module to do the file operators, and just provide a nice (probably object-oriented) interface for those methods. > > In python, `os` and `shutil` are currently the low-level modules, and it stands to reason that we might consider combining these somehow (although I'm assuming that there was a good reason not to in the first place, which is why they both exist, but I haven't looked into it). And `pathlib` is currently the "high-level" module. > > I see two problems currently: > > 1) the low-level module is split in half (for example, operations for copying are contained in `shutil` and operations for removing are contained in `os`). This is a bit annoying for the user, but it's not game-breaking. It does, however, make python feel a bit unnatural in this context, and that's even more unusual because normally python feels very natural. So this becomes sort of a "huh this feels weird" situation. > > 2) The `pathlib` modules only provides a high-level interface for working with _single_ Path objects. There isn't really functionality to work with multiple Path objects (as pointed out by someone previously if I am understanding correctly). > > I don't think the current PEP under consideration adequately solves either of these problems. Currently, it seems like it's trying to make `pathlib` both a high- and low-level module, which imo doesn't make sense. > > But I do think we need, if not a single low-level module, at least a high-level module that makes it unnecessary to use the low-level modules. That means that `pathlib` needs more functionality added to it, which is similar in spirit to the current PEP proposal Surely shutil is a *high* level modules. os is a low level module that shutil builds on. Adding the missing pieces to shutil would make it the place to go to do file operations. Pity its not called filelib. Barry > > - Jason, a reader > > > > > On Sun, Mar 18, 2018 at 9:46 AM, Paul Moore > wrote: > On 18 March 2018 at 04:41, Nathaniel Smith > wrote: > > My understanding is that the point of Path is to be a convenient, > > pleasant-to-use mechanism for accessing common filesystem operations. > > And it does a pretty excellent job of that. But it seems obvious to me > > that it's still missing a number of fairly basic operations that > > people need all the time. > > IMO, the pathlib module (just) defines Path. So I'm -1 on adding > anything to pathlib that isn't a method of a Path object. Beyond that, > I agree with you that Path should be a convenient interface for > filesystem path objects. I haven't personally found that there's much > missing that I've needed, but I agree that there are some gaps from a > theoretical point of view, and adding methods to fill those gaps could > be justifiable. OTOH, the fspath protocol was explicitly designed so > that standalone functions (such as the ones in os and shutil) can work > cleanly with Path objects - so there's a strong argument that "not > everything needs to be a method" applies here. For example, while > there isn't a Path.makedirs(), what's so bad about os.makedirs(Path)? > (There's consistency and discoverability arguments, but they are not > what I'd call compelling on their own). > > > I don't think the PEP is there yet, and we > > can quibble over the details -- just copying over all the historical > > decisions in shutil isn't obviously the right move (maybe it should be > > Path.mkdir(include_parents=True) and Path.unlink(recursive=True) > > instead of Path.makedirs and Path.rmtree?), but there's definitely > > room for improvement. > > I agree that there are some potential candidates for "useful > additional methods for Path objects", but I'd like to see these > discussed on a case by case basis, much like you do here, rather than > as a blanket "if it's in some other module and it works on paths, it > should be in pathlib. > > My biggest problem with the proposal as it stands is that it makes no > attempt to justify the suggestions on a case by case basis (the first > version wasn't even explicit in the functions it was proposing!) but > argues from a pure "lump everything together" standpoint. > > Paul > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjmaldonis at gmail.com Sun Mar 18 14:43:52 2018 From: jjmaldonis at gmail.com (Jason Maldonis) Date: Sun, 18 Mar 2018 13:43:52 -0500 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: <5959B700-AFE4-4CCF-9F96-3DB79C5EF5A5@barrys-emacs.org> References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <5959B700-AFE4-4CCF-9F96-3DB79C5EF5A5@barrys-emacs.org> Message-ID: > > Surely shutil is a *high* level modules. > os is a low level module that shutil builds on. > Adding the missing pieces to shutil would make it the place to go to do > file operations. > Pity its not called filelib. > Gotcha, thank you! shutil being a high level library complicates things... So we have two "high-level" libraries (pathlib and shutil) and both of them provide different pieces of useful functionality. Maybe I am starting to see why this is complicated. Thanks for reading my above reply and taking the time to respond. -------------- next part -------------- An HTML attachment was scrubbed... URL: From george at fischhof.hu Sun Mar 18 17:16:31 2018 From: george at fischhof.hu (George Fischhof) Date: Sun, 18 Mar 2018 22:16:31 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <5959B700-AFE4-4CCF-9F96-3DB79C5EF5A5@barrys-emacs.org> Message-ID: Hi Jason, the status of os and shutil became this because of C functions in implementation (I got something similar answer before) ... What do you think, what would be a good way to solve this - add stuff from os to shutil - add stuff from os and shutil to pathlib - create a new module on top of os, shutil and pathlib (we could name it for example filelib George 2018-03-18 19:43 GMT+01:00 Jason Maldonis : > Surely shutil is a *high* level modules. >> os is a low level module that shutil builds on. >> Adding the missing pieces to shutil would make it the place to go to do >> file operations. >> Pity its not called filelib. >> > > Gotcha, thank you! shutil being a high level library complicates things... > So we have two "high-level" libraries (pathlib and shutil) and both of them > provide different pieces of useful functionality. Maybe I am starting to > see why this is complicated. Thanks for reading my above reply and taking > the time to respond. > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sun Mar 18 21:08:58 2018 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sun, 18 Mar 2018 20:08:58 -0500 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <5959B700-AFE4-4CCF-9F96-3DB79C5EF5A5@barrys-emacs.org> Message-ID: On Sun, Mar 18, 2018 at 4:16 PM, George Fischhof wrote: > Hi Jason, > > the status of os and shutil became this because of C functions in > implementation (I got something similar answer before) > ... > > What do you think, what would be a good way to solve this > - add stuff from os to shutil > - add stuff from os and shutil to pathlib > - create a new module on top of os, shutil and pathlib (we could name it > for example filelib > > But...why? It's not clear to me as to how this would actually, you know, *solve* anything. > George > > 2018-03-18 19:43 GMT+01:00 Jason Maldonis : > >> Surely shutil is a *high* level modules. >>> os is a low level module that shutil builds on. >>> Adding the missing pieces to shutil would make it the place to go to do >>> file operations. >>> Pity its not called filelib. >>> >> >> Gotcha, thank you! shutil being a high level library complicates >> things... So we have two "high-level" libraries (pathlib and shutil) and >> both of them provide different pieces of useful functionality. Maybe I am >> starting to see why this is complicated. Thanks for reading my above reply >> and taking the time to respond. >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -- Ryan (????) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Sun Mar 18 18:53:20 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Mon, 19 Mar 2018 11:53:20 +1300 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> Message-ID: <5AAEEDE0.30602@canterbury.ac.nz> Jason Maldonis wrote: > In python, `os` and `shutil` are currently the low-level modules, and it > stands to reason that we might consider combining these somehow > (although I'm assuming that there was a good reason not to in the first > place The functions in os are thin wrappers around system calls, whereas those in shutil require substantial amounts of code (either python or C) to implement and are used relatively rarely. -- Greg From jjmaldonis at gmail.com Sun Mar 18 22:48:50 2018 From: jjmaldonis at gmail.com (Jason Maldonis) Date: Sun, 18 Mar 2018 21:48:50 -0500 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <5959B700-AFE4-4CCF-9F96-3DB79C5EF5A5@barrys-emacs.org> Message-ID: On Sun, Mar 18, 2018 at 4:16 PM, George Fischhof wrote: > Hi Jason, > > the status of os and shutil became this because of C functions in > implementation (I got something similar answer before) > ... > > What do you think, what would be a good way to solve this > - add stuff from os to shutil > - add stuff from os and shutil to pathlib > - create a new module on top of os, shutil and pathlib (we could name it > for example filelib > I'm pretty ignorant of the situation so my opinion probably isn't worth very much, but I'll give it a try ;) If shutil is functioning as a high-level library, it makes sense to me to add the stuff from os to shutil. Then users can simply import shutil and know that they have all the functionality they need to work with files. Someone said that pathlib's Paths work with the os module but not with shutil -- if os is the low-level module and shutil is the high-level, that seems a bit strange but I'm sure it made sense to do that. It seems to me that adding os functionality to shutil to make shutil "complete", and also modifying shutil functions to work with Path objects makes the most sense. Once that's done, shutil will be _the_ high-level library for using file operations in python, with the option to use Path objects if you want (side note: I'm a big fan of the Path class!). After that, it may make sense to add some (maybe all) of the methods in shutil to the pathlib module similar to how Path already implements some file operations (e.g. Path.glob and Path.unlink). All of that seems like a three step process to me: 1) Add the missing methods from os to shutil 2) Make the shutil methods work nicely with Path objects 3) Reevaluate and see what methods make sense to add to the pathlib module As a user, I'd be thrilled with that setup. -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Sun Mar 18 23:32:42 2018 From: njs at pobox.com (Nathaniel Smith) Date: Sun, 18 Mar 2018 20:32:42 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: On Sun, Mar 18, 2018 at 4:58 AM, George Fischhof wrote: > Of course several details could be put into it, but I think it would better > to let the developers decide the details, because they know the environment > and the possibilities. That's not how PEPs work :-). Someone has to do the work of collating contradictory feedback and making opinionated design proposals, and the person who does that is called the PEP author. In this case, I'd also suggest framing the PEP as a list of specific things that should be added to pathlib.Path, with justifications for each. If your argument is "X should be in pathlib because it's in some other module", then that's not very compelling -- by definition it already means we have an X, so why do we need another? I think for a number of these cases there actually is a good answer to that question, but your PEP has to actually provide that answer :-). -n -- Nathaniel J. Smith -- https://vorpus.org From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Mar 19 04:17:54 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 19 Mar 2018 17:17:54 +0900 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> Message-ID: <23215.29234.147843.69581@turnbull.sk.tsukuba.ac.jp> Nathaniel Smith writes: > On Sat, Mar 17, 2018 at 10:15 AM, Stephen J. Turnbull > wrote: > > (5) perform operations on several objects denoted by Paths at once > > (copy and its multiple operand variants), > > Sure it does: Path.rename and Path.replace. I was very careful about the semantics. Those are *single objects* denoted by multiple Paths (at different times). You could argue multiple objects for Path.replace, but I consider maybe removing the reference to the original target of the new Path to be an edge case that you need to address if you have Path.rename. > Similarly, it's hard to explain why we have Path.mkdir but not > Path.makedirs So what? Let's fix that. As you propose: > (maybe it should be Path.mkdir(include_parents=True) is fine, although that default seems a little risky vs. typos. I know I have confused myself with mkdir -p that way a few (though very few) times. Perhaps Guido would prefer Path.makedirs for this functionality. > and Path.unlink(recursive=True) I dislike that API, to be honest (at least two interpretations: rmtree and remove_empty_directories=True). I would definitely call the more destructive operation Path.rmtree. > but there's definitely room for improvement. I didn't deny that. All I argued was that, no, it really seems unlikely to me that Antoine intended pathlib to become Emacs. And I am against the PEP in its current form where it clearly intends to incorporate practically everything in os (dealing with filesystem objects) and shutil. Those APIs are not clean. I also feel that before we do anything but the minor filling-in exercises discussed explicitly above, we should see if we can add URIPath conforming to RFC 3986 and RFC 3987. Echoing Antoine's misgivings, I'm dubious about that, though, because Antoine implemented the concrete "realpath" semantics (resolve links before ..) in pathlib, while RFC 3986 specifies formal path manipulation semantics to prevent traversal above DocumentRoot and similar exploits. In web programming URIPath to Path conversions, and vice versa, will be very common, but I suspect at least one direction will be fragile because conversion and canonicalize won't commute. I hope I'm wrong![1] Well we know where we're going But we don't know where we've been And we know what we're knowing But we can't say what we've seen We're on a Path to nowhere Come on inside Taking that ride to nowhere We'll take that ride :-) Footnotes: [1] Perhaps that can be fixed by recommending a single composed operation that is safe. Maybe the simple conversion operations themselves can be "private" methods. From turnbull.stephen.fw at u.tsukuba.ac.jp Mon Mar 19 04:20:22 2018 From: turnbull.stephen.fw at u.tsukuba.ac.jp (Stephen J. Turnbull) Date: Mon, 19 Mar 2018 17:20:22 +0900 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <5959B700-AFE4-4CCF-9F96-3DB79C5EF5A5@barrys-emacs.org> Message-ID: <23215.29382.637066.972361@turnbull.sk.tsukuba.ac.jp> Jason Maldonis writes: > So we have two "high-level" libraries (pathlib and shutil) pathlib is currently "low-level" as I understand the word. The only complex things it does are resolving and globbing, which are reasonable things to do with a Path's target object. Everything else is either a formal Path manipulation, or acts on a single object that is the target of a Path. From encukou at gmail.com Mon Mar 19 05:10:40 2018 From: encukou at gmail.com (Petr Viktorin) Date: Mon, 19 Mar 2018 10:10:40 +0100 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: <23215.29234.147843.69581@turnbull.sk.tsukuba.ac.jp> References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <23215.29234.147843.69581@turnbull.sk.tsukuba.ac.jp> Message-ID: On 03/19/18 09:17, Stephen J. Turnbull wrote: > Nathaniel Smith writes: > > Similarly, it's hard to explain why we have Path.mkdir but not > > Path.makedirs > > So what? Let's fix that. As you propose: > > > (maybe it should be Path.mkdir(include_parents=True) > > is fine, although that default seems a little risky vs. typos. I know > I have confused myself with mkdir -p that way a few (though very few) > times. Perhaps Guido would prefer Path.makedirs for this functionality. This is already there as Path.mkdir(parents=True). From wes.turner at gmail.com Mon Mar 19 05:23:35 2018 From: wes.turner at gmail.com (Wes Turner) Date: Mon, 19 Mar 2018 05:23:35 -0400 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: Here's a comparison table of os, os.path, shutil, pathlib, and path.py. The full version is at https://github.com/westurner/pyfilemods (README.rst) and at https://westurner.github.io/pyfilemods. I ran a few set intersections and went ahead and wrote a report to compare function/method signatures and sources. attr table ========== ================== == ======= ====== ======= ======= attr os os.path shutil pathlib path.py ================== == ======= ====== ======= ======= `__div__`_ X `__rdiv__`_ X `absolute`_ X `abspath`_ X X `access`_ X X `altsep`_ X X `anchor`_ X `as_posix`_ X `as_uri`_ X `atime`_ X `basename`_ X X `bytes`_ X `capitalize`_ X `casefold`_ X `cd`_ X `center`_ X `chdir`_ X X `chmod`_ X X X `chown`_ X X X `chroot`_ X X `chunks`_ X `commonpath`_ X `commonprefix`_ X `copy`_ X X `copy2`_ X X `copyfile`_ X X `copymode`_ X X `copystat`_ X X `copytree`_ X X `count`_ X `ctime`_ X `curdir`_ X X `cwd`_ X `defpath`_ X X `devnull`_ X X `dirname`_ X X `dirs`_ X `drive`_ X X `encode`_ X `endswith`_ X `exists`_ X X X `expand`_ X `expandtabs`_ X `expanduser`_ X X X `expandvars`_ X X `ext`_ X `extsep`_ X X `files`_ X `find`_ X `fnmatch`_ X X `format`_ X `format_map`_ X `get_owner`_ X `getatime`_ X X `getctime`_ X X `getcwd`_ X X `getmtime`_ X X `getsize`_ X X `glob`_ X X `group`_ X `home`_ X `in_place`_ X `index`_ X `is_absolute`_ X `is_block_device`_ X `is_char_device`_ X `is_dir`_ X `is_fifo`_ X `is_file`_ X `is_reserved`_ X `is_socket`_ X `is_symlink`_ X `isabs`_ X X `isalnum`_ X `isalpha`_ X `isdecimal`_ X `isdigit`_ X `isdir`_ X X `isfile`_ X X `isidentifier`_ X `islink`_ X X `islower`_ X `ismount`_ X X `isnumeric`_ X `isprintable`_ X `isspace`_ X `istitle`_ X `isupper`_ X `iterdir`_ X `join`_ X X `joinpath`_ X X `lchmod`_ X `lexists`_ X `lines`_ X `link`_ X X `listdir`_ X X `ljust`_ X `lower`_ X `lstat`_ X X X `lstrip`_ X `makedirs`_ X X `makedirs_p`_ X `maketrans`_ X `match`_ X `merge_tree`_ X `mkdir`_ X X X `mkdir_p`_ X `module`_ X `move`_ X X `mtime`_ X `name`_ X X X `namebase`_ X `normcase`_ X X `normpath`_ X X `open`_ X X X `os`_ X X `owner`_ X X `pardir`_ X X `parent`_ X X `parents`_ X `partition`_ X `parts`_ X `pathconf`_ X X `pathsep`_ X X `read_bytes`_ X `read_hash`_ X `read_hexhash`_ X `read_md5`_ X `read_text`_ X `readlink`_ X X `readlinkabs`_ X `realpath`_ X X `relative_to`_ X `relpath`_ X X `relpathto`_ X `remove`_ X X `remove_p`_ X `removedirs`_ X X `removedirs_p`_ X `rename`_ X X X `renames`_ X X `replace`_ X X X `resolve`_ X `rfind`_ X `rglob`_ X `rindex`_ X `rjust`_ X `rmdir`_ X X X `rmdir_p`_ X `rmtree`_ X X `rmtree_p`_ X `root`_ X `rpartition`_ X `rsplit`_ X `rstrip`_ X `samefile`_ X X X `sameopenfile`_ X `samestat`_ X `sep`_ X X `size`_ X `special`_ X `split`_ X X `splitall`_ X `splitdrive`_ X X `splitext`_ X X `splitlines`_ X `splitpath`_ X `splitunc`_ X `startswith`_ X `stat`_ X X X X X `statvfs`_ X X `stem`_ X X `strip`_ X `stripext`_ X `suffix`_ X `suffixes`_ X `swapcase`_ X `symlink`_ X X `symlink_to`_ X `text`_ X `title`_ X `touch`_ X X `translate`_ X `uncshare`_ X `unlink`_ X X X `unlink_p`_ X `upper`_ X `using_module`_ X `utime`_ X X `walk`_ X X `walkdirs`_ X `walkfiles`_ X `with_name`_ X `with_suffix`_ X X `write_bytes`_ X X `write_lines`_ X `write_text`_ X X `zfill`_ X ================== == ======= ====== ======= ======= On Wed, Mar 14, 2018 at 2:22 PM, Wes Turner wrote: > path.py conveniently defines very many (most?) file operations as methods: > https://github.com/jaraco/path.py/blob/master/path.py > > https://pathpy.readthedocs.io/en/latest/api.html > > The division operator is mapped to os.path.join: > > Path('a') / '/root' == Path('/root') > > > On Monday, March 12, 2018, George Fischhof wrote: > >> >> >> 2018-03-12 22:16 GMT+01:00 Paul Moore : >> >>> On 12 March 2018 at 20:57, George Fischhof wrote: >>> > Good day all, >>> > >>> > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to >>> > contain file operations. >>> > >>> > Here is the draft. What do you think about this? >>> >>> I don't know for certain what I think I feel about the idea - in >>> general, it seems plausible. But I think you'll need to get down to >>> specifics in the PEP, exactly what functions do you suggest get added >>> to pathlib? >>> >>> Paul >>> >> >> >> Basically file and directory operations: copy, rename, move, delete. >> With all variants. >> I will collect them, and put intot PEP >> >> George >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Mar 19 07:36:09 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 19 Mar 2018 22:36:09 +1100 Subject: [Python-ideas] Adding quantile to the statistics module In-Reply-To: <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> References: <20180315234644.GD16661@ando.pearwood.info> <23211.18895.270104.810022@turnbull.sk.tsukuba.ac.jp> Message-ID: <20180319113607.GK16661@ando.pearwood.info> On Fri, Mar 16, 2018 at 01:36:31PM +0900, Stephen J. Turnbull wrote: > Steven D'Aprano writes: > > > Indeed. I've been considering quantiles and quartiles for a long time, > > and I've found at least ten different definitions for quantiles and > > sixteen for quartiles. > I'd like to see your list written up. On checking my notes, there is considerable overlap in the numbers above (some calculatation methods are equivalent to others) but overall I find a total of 16 distinct methods in use. Some of those are only suitable for generating quartiles. This should not be considered an exhaustive list, I may have missed some. Additions and corrections will be welcomed :-) My major sources are Hyndman & Fan: https://www.amherst.edu/media/view/129116/original/Sample+Quantiles.pdf and Langford: https://ww2.amstat.org/publications/jse/v14n3/langford.html Langford concentrates on methods of calculating quartiles, while Hyndman & Fan consider more general quantile methods. Obviously if you have a general quantile method, you can use it to calculate quartiles. I have compiled a summary in the following table. Reading across the row are the (usually numeric) label or parameter used to specify a calculation method. Entries in the same column are the same calculation method regardless of the label. For example, what Hyndman & Fan call method 1, Langford calls method 15, and the SAS software uses a parameter of 3. The Excel QUARTILE function is equivalent to what H&F call method 7 and what Langford calls 12. You will need to use a monospaced font for the columns to line up. H&F 1 2 3 4 5 6 7 8 9 Langford 15 4 14 13 10 11 12 1 2 5 6 9 Excel Q Excel 2010+ QE QI JMP X Maple 1 2 3 4 5 6 7 8 Mathematica AQ MQ Minitab X R 1 2 3 4 5 6 7 8 9 S X SAS 3 5 2 1 4 SPSS X TI calc X Notes: X Only calculation method used by the software. Q Excel QUARTILE function (pre 2010) QE Excel QUARTILE.EXC function QI Excel QUARTILE and QUARTILE.INC functions AQ Mathematica AsymmetricQuartiles function MQ Mathematica Quartiles function Langford's 3 and 7 (not shown) is the same as his 1; his 8 (not shown) is the same as his 2. Hyndman & Fan recommend method 8 as the best method for general quantiles. Langford (who has certainly read H&F) recommends his method 4, which is H&F's method 2, as the standard quartile. That is the same as the default used by SAS. For what it's worth, the method taught in Australian high schools for calculating quartiles and interquartile range is Langford's method 2. That's the method that Texas Instruments calculators use. I haven't personally confirmed all of the software equivalences, in particular I'm a bit dubious about the Maple methods. If anyone has access to Maple and doesn't mind running a few sample calculations for me, please contact me off-list. -- Steve From rhodri at kynesim.co.uk Mon Mar 19 11:31:39 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 19 Mar 2018 15:31:39 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: On 18/03/18 11:58, George Fischhof wrote: > it seems for me that the welcoming of this proposal is rather positive than > not. I think you may have observer bias :-) As far as I am concerned you have yet to make a convincing case that there is a problem, never mind that your solution is appropriate. Your solution also isn't detailed enough, as several people have pointed out. -- Rhodri James *-* Kynesim Ltd From tjreedy at udel.edu Mon Mar 19 12:59:18 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Mon, 19 Mar 2018 12:59:18 -0400 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: On 3/19/2018 11:31 AM, Rhodri James wrote: > On 18/03/18 11:58, George Fischhof wrote: >> it seems for me that the welcoming of this proposal is rather positive >> than >> not. > > I think you may have observer bias :-) Or, as often happens, George is making too much of a biased sample of opinions -- those who care enough to respond, *given what has also been said.* Python-idea works best as a place to evaluate ideas and develop alternatives. Vote counting is premature. Proposers *must* be flexible and not defend their initial solution like the Alamo. > As far as I am concerned you have yet to make a convincing case that > there is a problem, never mind that your solution is appropriate.? Your > solution also isn't detailed enough, as several people have pointed out. I am convinced that *some* people, especially but not limited to newbies, find the current situation confusing and less than optimal. I am also pretty convinced that the idea of dumping a copy of everything into pathlib is the wrong solution. I most agree with the latest posts from Nathaniel Smith and Stephen Trunball. -- Terry Jan Reedy From python-ideas at mgmiller.net Mon Mar 19 13:39:23 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Mon, 19 Mar 2018 10:39:23 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 In-Reply-To: References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> Message-ID: On 2018-03-18 10:55, Paul Moore wrote: >> Should Path() have methods to access all file operations? > > No, (Counterexample, having a Path operation to set Windows ACLs for a path). Agreed, not a big fan of everything filesystem-related in pathlib, simply because it doesn't read well. Having them scattered isn't a great experience either. Perhaps it would be better to have a filesystem package instead, maybe named "fs" that included all this stuff in one easy to use location. File stuff from os, path stuff from os.path, pathlib, utils like stat, and shutil etc? From brett at python.org Mon Mar 19 15:08:56 2018 From: brett at python.org (Brett Cannon) Date: Mon, 19 Mar 2018 19:08:56 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: On Sun, 18 Mar 2018 at 20:37 Nathaniel Smith wrote: > On Sun, Mar 18, 2018 at 4:58 AM, George Fischhof > wrote: > > Of course several details could be put into it, but I think it would > better > > to let the developers decide the details, because they know the > environment > > and the possibilities. > > That's not how PEPs work :-). Someone has to do the work of collating > contradictory feedback and making opinionated design proposals, and > the person who does that is called the PEP author. > > In this case, I'd also suggest framing the PEP as a list of specific > things that should be added to pathlib.Path, with justifications for > each. If your argument is "X should be in pathlib because it's in some > other module", then that's not very compelling -- by definition it > already means we have an X, so why do we need another? I think for a > number of these cases there actually is a good answer to that > question, but your PEP has to actually provide that answer :-). > And just to make it super-clear, the advise Nathaniel has provided will be required to be met before any of the PEP editors accept this proposal. IOW a bullet list will not suffice and you will need clear justification for every change you make in order to explain why every book on Python will need to be updated due to this PEP. :) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfahlgren at gmail.com Mon Mar 19 19:11:01 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Mon, 19 Mar 2018 16:11:01 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> Message-ID: On Mon, Mar 19, 2018 at 9:59 AM, Terry Reedy wrote: > I am convinced that *some* people, especially but not limited to newbies, > find the current situation confusing and less than optimal. I am also > pretty convinced that the idea of dumping a copy of everything into pathlib > is the wrong solution. I most agree with the latest posts from Nathaniel > Smith and Stephen Trunball. > > I'm confused ? by the whole discussion? because I thought pathlib was a means for dealing with paths (i.e., names of things), not the objects to which they refer ?, and was simply a Pythonic modernization of os.path.? ? ? -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Mon Mar 19 21:07:02 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 Mar 2018 01:07:02 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: <5AAEEDE0.30602@canterbury.ac.nz> References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <5AAEEDE0.30602@canterbury.ac.nz> Message-ID: On Sun, Mar 18, 2018 at 10:53 PM, Greg Ewing wrote: > > The functions in os are thin wrappers around system calls, > exactly -- and this is a very old legacy from way back. Modern Python users should not have to concern themselves with whether something they want to do is essentially a system call or a higher-level process. Typical users, and certainly newbies, think "I want to do this or that with the filesystem", and it would be really nice if there was one way, and one place to do that. The old os vs shutil was annoying enough, then we got pathlib with very little support in the stlib, which was really annoying. Now we finally have pathlib support in most of the stdlib, so I can really tell people that they can use Paths, rather than strings for paths -- great! But yes, the job is not yet finished, because we still have to go find _some_ functionality in os or shutil Yes, it seems like duplication, but that decision was made when pathlib as added. I do think we should not simply move everything, but rather work out each case -- and I like Nathanial's idea of simplifying / cleaning up the API a bit while we are at it. (please don't have an "unlink"!). -CHB PS: does shutil really still not work with Path objects? aarrgg! PPS: someone made a comment about "having to update every book about python" -- so I"ll repeat: that decision was made when pathlib was added. -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg at krypto.org Tue Mar 20 02:25:23 2018 From: greg at krypto.org (Gregory P. Smith) Date: Tue, 20 Mar 2018 06:25:23 +0000 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: Message-ID: On Thu, Mar 15, 2018 at 3:26 AM Nick Coghlan wrote: > On 14 March 2018 at 15:20, Chris Billington > wrote: >> >> I wonder if there's any reason something like this shouldn't be built >> into Python's default import system. >> > > There are two main challenges with enforcing such a check, one affecting > end users in general, one affecting standard library maintainers in > particular: > > * the user facing problem is a backwards compatibility one: while > double-imports usually aren't what people wanted, they're also typically > fairly harmless. As a result, elevating them from "sometimes a source of > obscure bugs" to "categorically prohibited" risks breaking currently > working code. While a conventional deprecation cycle should be possible, it > isn't clear whether or not the problem occurs frequently enough to warrant > that effort. > I call all such code "working"... It is a bad design. They are not harmless for any module that initializes global state or defines exceptions or types and relies on catching those exceptions or types via isinstance checks when things created in one part of the program wind up used in another that refers to a different multiply imported copy of the module. This is even more painful for extension modules. Debugging these issues is hard. A cache by os.path.abspath would be a good thing. I'd personally prefer it to be an ImportError with a message explaining the problem. Including a pointer to the original successful import when the import under another sys.modules name is attempted. It'd lead to better code health. But others likely disagree and prefer to silently return the existing module. > * the maintainer level problem is that we actually do this on purpose in > the standard library's test suite in order to test both pure Python and C > accelerated variants of various modules. That could be handled by being > careful about exactly where the reverse lookup cache from filenames back to > module names is checked and updated, but it does make the problem a bit > trickier than just "maintain a reverse lookup table from filesystem paths > to module names and complain if an import gets a hit in that table" > > I'm definitely sympathetic to the idea, though. > > If we did head in this direction, then we'd also need to accept & > implement PEP 499 [1] (which proposes aliasing __main__ as > __main__.__spec__.name in sys.modules when executed with "-m") to avoid > causing problems. > I don't doubt that doing this would require a lot of code cleanup. :) > > Cheers, > Nick. > > [1] https://www.python.org/dev/peps/pep-0499/ > > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From wes.turner at gmail.com Tue Mar 20 04:03:56 2018 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 20 Mar 2018 04:03:56 -0400 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: I added trio to the comparison table (Things are mostly just async-wrapped, though pathlib_not_trio does show a few missing methods?). https://github.com/westurner/pyfilemods/issues/2 https://github.com/westurner/pyfilemods/blob/master/README.rst#attr-table ================== == ======= ====== ======= ======= ==== attr os os.path shutil pathlib path.py trio ================== == ======= ====== ======= ======= ==== `__div__`_ X `__rdiv__`_ X `absolute`_ X X `abspath`_ X X `access`_ X X `altsep`_ X X `anchor`_ X `as_posix`_ X X `as_uri`_ X X `atime`_ X `basename`_ X X `bytes`_ X `capitalize`_ X `casefold`_ X `cd`_ X `center`_ X `chdir`_ X X `chmod`_ X X X X `chown`_ X X X `chroot`_ X X `chunks`_ X `commonpath`_ X `commonprefix`_ X `copy`_ X X `copy2`_ X X `copyfile`_ X X `copymode`_ X X `copystat`_ X X `copytree`_ X X `count`_ X `ctime`_ X `curdir`_ X X `cwd`_ X X `defpath`_ X X `devnull`_ X X `dirname`_ X X `dirs`_ X `drive`_ X X `encode`_ X `endswith`_ X `exists`_ X X X X `expand`_ X `expandtabs`_ X `expanduser`_ X X X X `expandvars`_ X X `ext`_ X `extsep`_ X X `files`_ X `find`_ X `fnmatch`_ X X `format`_ X `format_map`_ X `get_owner`_ X `getatime`_ X X `getctime`_ X X `getcwd`_ X X `getmtime`_ X X `getsize`_ X X `glob`_ X X X `group`_ X X `home`_ X X `in_place`_ X `index`_ X `is_absolute`_ X X `is_block_device`_ X X `is_char_device`_ X X `is_dir`_ X X `is_fifo`_ X X `is_file`_ X X `is_reserved`_ X X `is_socket`_ X X `is_symlink`_ X X `isabs`_ X X `isalnum`_ X `isalpha`_ X `isdecimal`_ X `isdigit`_ X `isdir`_ X X `isfile`_ X X `isidentifier`_ X `islink`_ X X `islower`_ X `ismount`_ X X `isnumeric`_ X `isprintable`_ X `isspace`_ X `istitle`_ X `isupper`_ X `iterdir`_ X X `join`_ X X `joinpath`_ X X X `lchmod`_ X X `lexists`_ X `lines`_ X `link`_ X X `listdir`_ X X `ljust`_ X `lower`_ X `lstat`_ X X X X `lstrip`_ X `makedirs`_ X X `makedirs_p`_ X `maketrans`_ X `match`_ X X `merge_tree`_ X `mkdir`_ X X X X `mkdir_p`_ X `module`_ X `move`_ X X `mtime`_ X `name`_ X X X `namebase`_ X `normcase`_ X X `normpath`_ X X `open`_ X X X X `os`_ X X `owner`_ X X X `pardir`_ X X `parent`_ X X `parents`_ X `partition`_ X `parts`_ X `pathconf`_ X X `pathsep`_ X X `read_bytes`_ X X `read_hash`_ X `read_hexhash`_ X `read_md5`_ X `read_text`_ X X `readlink`_ X X `readlinkabs`_ X `realpath`_ X X `relative_to`_ X X `relpath`_ X X `relpathto`_ X `remove`_ X X `remove_p`_ X `removedirs`_ X X `removedirs_p`_ X `rename`_ X X X X `renames`_ X X `replace`_ X X X X `resolve`_ X X `rfind`_ X `rglob`_ X X `rindex`_ X `rjust`_ X `rmdir`_ X X X X `rmdir_p`_ X `rmtree`_ X X `rmtree_p`_ X `root`_ X `rpartition`_ X `rsplit`_ X `rstrip`_ X `samefile`_ X X X X `sameopenfile`_ X `samestat`_ X `sep`_ X X `size`_ X `special`_ X `split`_ X X `splitall`_ X `splitdrive`_ X X `splitext`_ X X `splitlines`_ X `splitpath`_ X `splitunc`_ X `startswith`_ X `stat`_ X X X X X X `statvfs`_ X X `stem`_ X X `strip`_ X `stripext`_ X `suffix`_ X `suffixes`_ X `swapcase`_ X `symlink`_ X X `symlink_to`_ X X `text`_ X `title`_ X `touch`_ X X X `translate`_ X `uncshare`_ X `unlink`_ X X X X `unlink_p`_ X `upper`_ X `using_module`_ X `utime`_ X X `walk`_ X X `walkdirs`_ X `walkfiles`_ X `with_name`_ X X `with_suffix`_ X X X `write_bytes`_ X X X `write_lines`_ X `write_text`_ X X X `zfill`_ X ================== == ======= ====== ======= ======= ==== On Mon, Mar 19, 2018 at 5:23 AM, Wes Turner wrote: > Here's a comparison table of os, os.path, shutil, pathlib, and path.py. > > The full version is at > https://github.com/westurner/pyfilemods (README.rst) > and at > https://westurner.github.io/pyfilemods. > > I ran a few set intersections and went ahead and > wrote a report to compare function/method signatures and sources. > > attr table > ========== > > ================== == ======= ====== ======= ======= > attr os os.path shutil pathlib path.py > ================== == ======= ====== ======= ======= > `__div__`_ X > `__rdiv__`_ X > `absolute`_ X > `abspath`_ X X > `access`_ X X > `altsep`_ X X > `anchor`_ X > `as_posix`_ X > `as_uri`_ X > `atime`_ X > `basename`_ X X > `bytes`_ X > `capitalize`_ X > `casefold`_ X > `cd`_ X > `center`_ X > `chdir`_ X X > `chmod`_ X X X > `chown`_ X X X > `chroot`_ X X > `chunks`_ X > `commonpath`_ X > `commonprefix`_ X > `copy`_ X X > `copy2`_ X X > `copyfile`_ X X > `copymode`_ X X > `copystat`_ X X > `copytree`_ X X > `count`_ X > `ctime`_ X > `curdir`_ X X > `cwd`_ X > `defpath`_ X X > `devnull`_ X X > `dirname`_ X X > `dirs`_ X > `drive`_ X X > `encode`_ X > `endswith`_ X > `exists`_ X X X > `expand`_ X > `expandtabs`_ X > `expanduser`_ X X X > `expandvars`_ X X > `ext`_ X > `extsep`_ X X > `files`_ X > `find`_ X > `fnmatch`_ X X > `format`_ X > `format_map`_ X > `get_owner`_ X > `getatime`_ X X > `getctime`_ X X > `getcwd`_ X X > `getmtime`_ X X > `getsize`_ X X > `glob`_ X X > `group`_ X > `home`_ X > `in_place`_ X > `index`_ X > `is_absolute`_ X > `is_block_device`_ X > `is_char_device`_ X > `is_dir`_ X > `is_fifo`_ X > `is_file`_ X > `is_reserved`_ X > `is_socket`_ X > `is_symlink`_ X > `isabs`_ X X > `isalnum`_ X > `isalpha`_ X > `isdecimal`_ X > `isdigit`_ X > `isdir`_ X X > `isfile`_ X X > `isidentifier`_ X > `islink`_ X X > `islower`_ X > `ismount`_ X X > `isnumeric`_ X > `isprintable`_ X > `isspace`_ X > `istitle`_ X > `isupper`_ X > `iterdir`_ X > `join`_ X X > `joinpath`_ X X > `lchmod`_ X > `lexists`_ X > `lines`_ X > `link`_ X X > `listdir`_ X X > `ljust`_ X > `lower`_ X > `lstat`_ X X X > `lstrip`_ X > `makedirs`_ X X > `makedirs_p`_ X > `maketrans`_ X > `match`_ X > `merge_tree`_ X > `mkdir`_ X X X > `mkdir_p`_ X > `module`_ X > `move`_ X X > `mtime`_ X > `name`_ X X X > `namebase`_ X > `normcase`_ X X > `normpath`_ X X > `open`_ X X X > `os`_ X X > `owner`_ X X > `pardir`_ X X > `parent`_ X X > `parents`_ X > `partition`_ X > `parts`_ X > `pathconf`_ X X > `pathsep`_ X X > `read_bytes`_ X > `read_hash`_ X > `read_hexhash`_ X > `read_md5`_ X > `read_text`_ X > `readlink`_ X X > `readlinkabs`_ X > `realpath`_ X X > `relative_to`_ X > `relpath`_ X X > `relpathto`_ X > `remove`_ X X > `remove_p`_ X > `removedirs`_ X X > `removedirs_p`_ X > `rename`_ X X X > `renames`_ X X > `replace`_ X X X > `resolve`_ X > `rfind`_ X > `rglob`_ X > `rindex`_ X > `rjust`_ X > `rmdir`_ X X X > `rmdir_p`_ X > `rmtree`_ X X > `rmtree_p`_ X > `root`_ X > `rpartition`_ X > `rsplit`_ X > `rstrip`_ X > `samefile`_ X X X > `sameopenfile`_ X > `samestat`_ X > `sep`_ X X > `size`_ X > `special`_ X > `split`_ X X > `splitall`_ X > `splitdrive`_ X X > `splitext`_ X X > `splitlines`_ X > `splitpath`_ X > `splitunc`_ X > `startswith`_ X > `stat`_ X X X X X > `statvfs`_ X X > `stem`_ X X > `strip`_ X > `stripext`_ X > `suffix`_ X > `suffixes`_ X > `swapcase`_ X > `symlink`_ X X > `symlink_to`_ X > `text`_ X > `title`_ X > `touch`_ X X > `translate`_ X > `uncshare`_ X > `unlink`_ X X X > `unlink_p`_ X > `upper`_ X > `using_module`_ X > `utime`_ X X > `walk`_ X X > `walkdirs`_ X > `walkfiles`_ X > `with_name`_ X > `with_suffix`_ X X > `write_bytes`_ X X > `write_lines`_ X > `write_text`_ X X > `zfill`_ X > ================== == ======= ====== ======= ======= > > > > On Wed, Mar 14, 2018 at 2:22 PM, Wes Turner wrote: > >> path.py conveniently defines very many (most?) file operations as methods: >> https://github.com/jaraco/path.py/blob/master/path.py >> >> https://pathpy.readthedocs.io/en/latest/api.html >> >> The division operator is mapped to os.path.join: >> >> Path('a') / '/root' == Path('/root') >> >> >> On Monday, March 12, 2018, George Fischhof wrote: >> >>> >>> >>> 2018-03-12 22:16 GMT+01:00 Paul Moore : >>> >>>> On 12 March 2018 at 20:57, George Fischhof wrote: >>>> > Good day all, >>>> > >>>> > as it seemed to be a good idea, I wrote a PEP proposal for pathlib to >>>> > contain file operations. >>>> > >>>> > Here is the draft. What do you think about this? >>>> >>>> I don't know for certain what I think I feel about the idea - in >>>> general, it seems plausible. But I think you'll need to get down to >>>> specifics in the PEP, exactly what functions do you suggest get added >>>> to pathlib? >>>> >>>> Paul >>>> >>> >>> >>> Basically file and directory operations: copy, rename, move, delete. >>> With all variants. >>> I will collect them, and put intot PEP >>> >>> George >>> >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From njs at pobox.com Tue Mar 20 04:25:22 2018 From: njs at pobox.com (Nathaniel Smith) Date: Tue, 20 Mar 2018 01:25:22 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: On Tue, Mar 20, 2018 at 1:03 AM, Wes Turner wrote: > I added trio to the comparison table > (Things are mostly just async-wrapped, > though pathlib_not_trio does show a few missing methods?). trio.Path is an automatically generated, exact mirror of pathlib.Path, so I don't think it's very useful to have in your table? Also the missing attributes are actually handled via __getattr__, so they aren't actually missing, they're just invisible to your detection mechanism :-) In [21]: trio.Path("/a/b").anchor Out[21]: '/' In [22]: trio.Path("/a/b").name Out[22]: 'b' -n -- Nathaniel J. Smith -- https://vorpus.org From wes.turner at gmail.com Tue Mar 20 04:33:52 2018 From: wes.turner at gmail.com (Wes Turner) Date: Tue, 20 Mar 2018 04:33:52 -0400 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: On Tue, Mar 20, 2018 at 4:25 AM, Nathaniel Smith wrote: > On Tue, Mar 20, 2018 at 1:03 AM, Wes Turner wrote: > > I added trio to the comparison table > > (Things are mostly just async-wrapped, > > though pathlib_not_trio does show a few missing methods?). > > trio.Path is an automatically generated, exact mirror of pathlib.Path, > so I don't think it's very useful to have in your table? Also the > missing attributes are actually handled via __getattr__, so they > aren't actually missing, they're just invisible to your detection > mechanism :-) > > In [21]: trio.Path("/a/b").anchor > Out[21]: '/' > > In [22]: trio.Path("/a/b").name > Out[22]: 'b' > Ah, good call. Didn't even look. Is there a reason that the pathlib methods shouldn't just be similarly async-wrapped? > > -n > > -- > Nathaniel J. Smith -- https://vorpus.org > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Mar 20 05:06:00 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 20 Mar 2018 20:06:00 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> Message-ID: <20180320090559.GO16661@ando.pearwood.info> On Wed, Mar 14, 2018 at 12:09:55PM -0700, Guido van Rossum wrote: > Yeah, one should never add a module to sys.path that has a __init__.py file. Should import raise a warning in that case? I wouldn't want an outright error. I've cd'ed into a package directory in the shell, then run a python module from that directory too many times to want an exception, but I wouldn't mind a warning. -- Steve From chrisjbillington at gmail.com Tue Mar 20 06:23:12 2018 From: chrisjbillington at gmail.com (Chris Billington) Date: Tue, 20 Mar 2018 21:23:12 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: <20180320090559.GO16661@ando.pearwood.info> References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> <20180320090559.GO16661@ando.pearwood.info> Message-ID: On Tue, Mar 20, 2018 at 8:06 PM, Steven D'Aprano wrote: > On Wed, Mar 14, 2018 at 12:09:55PM -0700, Guido van Rossum wrote: > > > Yeah, one should never add a module to sys.path that has a __init__.py > file. > > Should import raise a warning in that case? > > I wouldn't want an outright error. I've cd'ed into a package directory > in the shell, then run a python module from that directory too many > times to want an exception, but I wouldn't mind a warning. > > Not to mention that plenty of programs are designed to run in whatever working directory they find themselves in, and that working directory may contain __init__.py files. For example, I wonder how mercurial gets around the fact that its own imports might be shadowed by whatever's in the current working directory. The mercurial project uses itself for version control, so it is presumably running with its working directory somewhere in its own source tree all the time. I wonder if mercurial removes the current working directory from sys.path to avoid any problems. A lot of programming tools no doubt often find themselves in working directories that are python packages. A warning would be pretty good! Especially if you could flip a switch to turn it into an error. Not if there is merely an __init__.py in view, but if you actually do an import twice, since a lot of code (with fully qualified imports, no submodules with names shadowing stdlib modules, etc) would never hit a problem with running from the package directory. It seems like running from within a package directory is bad news mostly *because* of the double import problem, and would be somewhat less of a bad idea if you could be confident you didn't have any accidental double imports (still something of a bad idea though because you can't know that your submodule isn't shadowing some other 3rd party module indirectly imported by your code, but that's about the only remaining issue with it I can think of). -Chris > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From brett at python.org Tue Mar 20 12:22:57 2018 From: brett at python.org (Brett Cannon) Date: Tue, 20 Mar 2018 16:22:57 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <5AAEEDE0.30602@canterbury.ac.nz> Message-ID: On Mon, 19 Mar 2018 at 18:08 Chris Barker wrote: > [SNIP] > PS: does shutil really still not work with Path objects? aarrgg! > Did you verify this or are you just guessing? If this is true then file a bug and optionally submit a patch. Saying "aarrgg" doesn't fix the situation nor motivate people to help out, especially when it sounds like you're not even sure yourself that it's even a problem. > > PPS: someone made a comment about "having to update every book about > python" -- so I"ll repeat: that decision was made when pathlib was added. > That was me and yes, we had to have updates made because we thought pathlib was worth it. At this point the PEP as proposed has not made the case that what it wants to add is worth it because right now it's just a huge bullet list of functions from two modules with no specific motivation behind the individual changes. The whole point of my comment is to say "we can't make changes just because; changes have to meet a certain bar of improvement" and this point the PEP has not proven there is such an improvement based on what has been proposed. IOW the core devs I have seen comment on this have pretty much all said "justify the individual methods" and yet no one has done that yet, so any discussion other than trying to meet that need is not helping to move anything forward. -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Tue Mar 20 13:50:01 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Tue, 20 Mar 2018 17:50:01 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: <20180313121736.GT18868@ando.pearwood.info> <20180315184822.3b50fb48@fsol> <23212.45875.337364.790795@turnbull.sk.tsukuba.ac.jp> <23213.19755.601512.793232@turnbull.sk.tsukuba.ac.jp> <5AAEEDE0.30602@canterbury.ac.nz> Message-ID: On Tue, Mar 20, 2018 at 4:23 PM Brett Cannon wrote: > > > "justify the individual methods" and yet no one has done that yet, so > any discussion other than trying to meet that need is not helping to move > anything forward. > My intent was, and still is, to encourage just that. And the rest of my message did say that ( I think). We all have only so many roundtoits to spend on this ? so I won?t be writing that PEP, but I do think it?s worthwhile to encourage the OP ( or anyone else ) to do so. Writing a PEP is a lot of work, one wants to know it has a chance of being accepted. Python really could be improved in this regard ? it?s had a confusing API for file system manipulations forever. Recent changes have helped, but it would be nice to get all the way there. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Mar 20 19:51:50 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 21 Mar 2018 12:51:50 +1300 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> <20180320090559.GO16661@ando.pearwood.info> Message-ID: <5AB19E96.5040405@canterbury.ac.nz> Chris Billington wrote: > I wonder how mercurial gets > around the fact that its own imports might be shadowed by whatever's in > the current working directory. The cwd is only added to sys.path in the interactive interpreter, not when you run "python something.py". So it's not usually a problem for applications implemented in Python. -- Greg From chrisjbillington at gmail.com Tue Mar 20 19:58:16 2018 From: chrisjbillington at gmail.com (Chris Billington) Date: Wed, 21 Mar 2018 10:58:16 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: <5AB19E96.5040405@canterbury.ac.nz> References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> <20180320090559.GO16661@ando.pearwood.info> <5AB19E96.5040405@canterbury.ac.nz> Message-ID: I don't think that's true: On Wed, Mar 21, 2018 at 10:51 AM, Greg Ewing wrote: > Chris Billington wrote: > >> I wonder how mercurial gets around the fact that its own imports might be >> shadowed by whatever's in the current working directory. >> > > The cwd is only added to sys.path in the interactive interpreter, > not when you run "python something.py". So it's not usually a > problem for applications implemented in Python. > > I don't think that's true: $ cd /tmp $ echo 'import bar' > foo.py $ echo 'print("this is bar")' > bar.py $ python foo.py this is bar (/tmp is not in the python path) -Chris > -- > Greg > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From chrisjbillington at gmail.com Tue Mar 20 20:01:59 2018 From: chrisjbillington at gmail.com (Chris Billington) Date: Wed, 21 Mar 2018 11:01:59 +1100 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> <20180320090559.GO16661@ando.pearwood.info> <5AB19E96.5040405@canterbury.ac.nz> Message-ID: On Wed, Mar 21, 2018 at 10:58 AM, Chris Billington < chrisjbillington at gmail.com> wrote: > I don't think that's true: > > > > On Wed, Mar 21, 2018 at 10:51 AM, Greg Ewing > wrote: > >> Chris Billington wrote: >> >>> I wonder how mercurial gets around the fact that its own imports might >>> be shadowed by whatever's in the current working directory. >>> >> >> The cwd is only added to sys.path in the interactive interpreter, >> not when you run "python something.py". So it's not usually a >> problem for applications implemented in Python. >> >> > I don't think that's true: > > $ cd /tmp > $ echo 'import bar' > foo.py > $ echo 'print("this is bar")' > bar.py > $ python foo.py > this is bar > > (/tmp is not in the python path) > > Actually I'm mistaken. That's the directory of the script being added to the path, not the current working directory. -Chris > >> -- >> Greg >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From danilo.bellini at gmail.com Tue Mar 20 21:49:20 2018 From: danilo.bellini at gmail.com (Danilo J. S. Bellini) Date: Tue, 20 Mar 2018 22:49:20 -0300 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> <20180320090559.GO16661@ando.pearwood.info> Message-ID: On 20 March 2018 at 07:23, Chris Billington wrote: > It seems like running from within a package directory is bad news mostly > *because* of the double import problem, [...] > I think the main issue regarding loading a module from within a package directory is about the paths, non-absolute non-relative import names, and [mainly] a desire to keep a module+script behavior in a single-file, not really about double importing. If you have a package "a" with a subpackage "b" that has a module "c", its file "a/b/c.py" would be: - Externally/internally imported as "from a.b import c" - Internally imported as "from .b import c", for modules in the "a" package level - Internally imported as "from . import c", for modules in the "b" subpackage level - Internally imported as "from ..b import c", for modules in a subpackage level sibling to "b" IMHO, both internally and externally to the package, you should never do "from b import c". The 3 issues I see: 1. [sys.path trick] Doing "from b import c" from within "a" wouldn't load the "a/b/c.py" file unless you do some nonstandard sys.path trick to search for modules/packages in both the root directory (to import "a") and the "a" package-level directory (to import "b"). This scenario makes both "a.b" and "b" imports available, distinct import names to the same file. 2. [non-absolute non-relative name] Using "from b import c" would be the way to [externally] load this module from the "a" package-level directory as the current working directory if the root directory isn't in the sys.path. But, in such a scenario, the "from a.b import c" simply don't work. That's the misleading point: you can import some package internals with alternative names, while the package itself can't be imported. The current working directory or a patched sys.path is the "culprit" that enforces a distinct/specific naming for the imports, which are otherwise invalid. It's not relative (there's no leading dot), but I'd not say it's absolute (as there's no leading "a." where "a" is a package), it's perhaps a relative-importable-from-external-scripts naming style (or absolute-from-internals), though it has the "absolute import" syntax. 3. [module+script in a single internal file] One might wish to load a module as a script, using a `if __name__ == "__main__":` block to segregate the twofold behavior: one as a module, another as a script. That's fine when dealing with a file that doesn't belong to a package. But, inside a package structure, loading a single internal module as a script breaks the paths (and the import names). To avoid that, I [almost] always use relative imports, so the file can't be loaded as a script, and I write the script in another file, either in the root directory (level that has the package directory) or as a "package_name/__main__.py" file, importing the stuff it needs from any module using the same names one would use otherwere. That is, the solution is to split the file in two (an internal module file, and an external script file). Nevertheless, if the same file can be imported from two valid paths/addresses, should them be a single module? I mean, should "file" and "module" be 1-to-1 concepts, or should "address/name" and "module" be 1-to-1 concepts (or neither)? How about symbolic links? I'm not sure, but linking "absolute file name" to "module" sounds like endorsing the relative-importable-from-external-scripts naming style, and IMHO that's not the main issue. As packages with modules that internally imports themselves has to choose between relative and absolute import names, I don't see [2] as a real issue. -- Danilo J. S. Bellini --------------- "*It is not our business to set up prohibitions, but to arrive at conventions.*" (R. Carnap) -------------- next part -------------- An HTML attachment was scrubbed... URL: From barry at barrys-emacs.org Wed Mar 21 05:09:10 2018 From: barry at barrys-emacs.org (Barry Scott) Date: Wed, 21 Mar 2018 09:09:10 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations In-Reply-To: References: Message-ID: <4715065.Htbq7em0lt@varric.chelsea.private> On Tuesday, 20 March 2018 16:22:57 GMT Brett Cannon wrote: > On Mon, 19 Mar 2018 at 18:08 Chris Barker wrote: > > [SNIP] > > PS: does shutil really still not work with Path objects? aarrgg! > > Did you verify this or are you just guessing? If this is true then file a > bug and optionally submit a patch. Saying "aarrgg" doesn't fix the > situation nor motivate people to help out, especially when it sounds like > you're not even sure yourself that it's even a problem. Checking this onm 3.6.4 on Fedora it looks like the shutil module works well with Path(). There was one bit of code that I thought might not work but it is not a problem as you cannot make a Path('b'filename'). Maybe the simplest change would be to add to the pathlib docs a notice at the top that says look in shutil and os for the file operations. Also show some examples of Path() working with shutil to do some typical operations. Barry From ncoghlan at gmail.com Wed Mar 21 07:32:39 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Mar 2018 21:32:39 +1000 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: Message-ID: On 20 March 2018 at 16:25, Gregory P. Smith wrote: > On Thu, Mar 15, 2018 at 3:26 AM Nick Coghlan wrote: > >> If we did head in this direction, then we'd also need to accept & >> implement PEP 499 [1] (which proposes aliasing __main__ as >> __main__.__spec__.name in sys.modules when executed with "-m") to avoid >> causing problems. >> > > I don't doubt that doing this would require a lot of code cleanup. :) > I was recently reminded of a "fun" edge case for PEP 499: "python -m site", which reruns a module that gets implicitly imported at startup (so "site" is already in sys.modules by the time __main__ runs). That said, the way that currently works (re-running sitecustomize and usercustomize) isn't particularly wonderful, so proposing changing it would be reasonable. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Wed Mar 21 07:36:39 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 21 Mar 2018 21:36:39 +1000 Subject: [Python-ideas] Disallow importing the same module under multiple names In-Reply-To: References: <20180314055828.GV18868@ando.pearwood.info> <20180314111808.GW18868@ando.pearwood.info> <5AA96918.40605@brenbarn.net> <20180320090559.GO16661@ando.pearwood.info> <5AB19E96.5040405@canterbury.ac.nz> Message-ID: On 21 March 2018 at 10:01, Chris Billington wrote: > > > On Wed, Mar 21, 2018 at 10:58 AM, Chris Billington < > chrisjbillington at gmail.com> wrote: > >> I don't think that's true: >> >> >> >> On Wed, Mar 21, 2018 at 10:51 AM, Greg Ewing > > wrote: >> >>> Chris Billington wrote: >>> >>>> I wonder how mercurial gets around the fact that its own imports might >>>> be shadowed by whatever's in the current working directory. >>>> >>> >>> The cwd is only added to sys.path in the interactive interpreter, >>> not when you run "python something.py". So it's not usually a >>> problem for applications implemented in Python. >>> >>> >> I don't think that's true: >> >> $ cd /tmp >> $ echo 'import bar' > foo.py >> $ echo 'print("this is bar")' > bar.py >> $ python foo.py >> this is bar >> >> (/tmp is not in the python path) >> >> > Actually I'm mistaken. That's the directory of the script being added to > the path, not the current working directory. > Scripts add the directory of the script, but the "-m" switch adds the current directory in order to locate modules and packages it can find there (although it's possible we'll attempt to figure out a way to change that in the future and require folks to explicitly opt-in to cwd relative main module imports: https://bugs.python.org/issue33053). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Thu Mar 22 16:34:44 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Thu, 22 Mar 2018 21:34:44 +0100 Subject: [Python-ideas] PEP proposal: unifying function/method classes Message-ID: <5AB41364.8010500@UGent.be> Dear python-ideas, I would like to draft a PEP to change the implementation of functions and methods in CPython. The idea for this PEP come from a discussion on https://bugs.python.org/issue30071 The core idea of the PEP is to reorganize classes for functions and methods with the goal of removing the difference between functions/methods implemented in C and functions/methods implemented in Python. Currently, CPython has two different function classes: the first is Python functions, which is what you get when defining a function with def or lambda. The second is built-in functions such as len, isinstance or numpy.dot. These are implemented in C. These two classes are completely independent with different functionality. For example, built-in functions do not have __get__ and therefore cannot be used as methods. And only Python functions support introspection like inspect.getargspec or inspect.getsourcefile. This is a problem for projects like Cython that want to implement functions in C but still want to offer functionality close to Python functions. In Cython, this was solved by inventing a new function class called cyfunction. Unfortunately, this new function class creates problems, for example the inspect module does not recognize such functions as being functions (this is the bug mentioned above) and the performance is worse (CPython has specific optimizations for calling built-in functions). When you look at methods, the inconsistency increases: On the one hand, the implementation of built-in functions and bound methods is very similar but built-in unbound methods are different. On the other hand, Python functions and unbound methods are identical but bound methods are different. Do you think that it makes sense to propose a PEP for this? Jeroen. From solipsis at pitrou.net Thu Mar 22 19:36:26 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Mar 2018 00:36:26 +0100 Subject: [Python-ideas] PEP proposal: unifying function/method classes References: <5AB41364.8010500@UGent.be> Message-ID: <20180323003626.318a6c34@fsol> On Thu, 22 Mar 2018 21:34:44 +0100 Jeroen Demeyer wrote: > Dear python-ideas, > > I would like to draft a PEP to change the implementation of functions > and methods in CPython. The idea for this PEP come from a discussion on > https://bugs.python.org/issue30071 > > The core idea of the PEP is to reorganize classes for functions and > methods with the goal of removing the difference between > functions/methods implemented in C and functions/methods implemented in > Python. > > Currently, CPython has two different function classes: the first is > Python functions, which is what you get when defining a function with > def or lambda. The second is built-in functions such as len, isinstance > or numpy.dot. These are implemented in C. > > These two classes are completely independent with different > functionality. For example, built-in functions do not have __get__ and > therefore cannot be used as methods. And only Python functions support > introspection like inspect.getargspec or inspect.getsourcefile. > > This is a problem for projects like Cython that want to implement > functions in C but still want to offer functionality close to Python > functions. In Cython, this was solved by inventing a new function class > called cyfunction. Unfortunately, this new function class creates > problems, for example the inspect module does not recognize such > functions as being functions (this is the bug mentioned above) and the > performance is worse (CPython has specific optimizations for calling > built-in functions). > > When you look at methods, the inconsistency increases: On the one hand, > the implementation of built-in functions and bound methods is very > similar but built-in unbound methods are different. On the other hand, > Python functions and unbound methods are identical but bound methods are > different. > > Do you think that it makes sense to propose a PEP for this? It does make sense, since the proposal sounds ambitious (and perhaps impossible without breaking compatibility). Regards Antoine. From J.Demeyer at UGent.be Fri Mar 23 02:25:33 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Fri, 23 Mar 2018 07:25:33 +0100 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: <9f7967f961e041ed9458543be77c0338@xmail102.UGent.be> References: <5AB41364.8010500@UGent.be> <9f7967f961e041ed9458543be77c0338@xmail102.UGent.be> Message-ID: <5AB49DDD.8040405@UGent.be> On 2018-03-23 00:36, Antoine Pitrou wrote: > It does make sense, since the proposal sounds ambitious (and perhaps > impossible without breaking compatibility). Well, *some* breakage of backwards compatibility will be unavoidable. My plan (just a plan for now!) is to preserve backwards compatibility in the following ways: * Existing Python attributes of functions/methods should continue to exist and behave the same * The inspect module should give the same results as now (by changing the implementation of some of the functions in inspect to match the new classes) * Everything from the documented Python/C API. This means that I might break compatibility in the following ways: * Changing the classes of functions/methods (this is the whole point of this PEP). So anything involving isinstance() checks might break. * The undocumented parts of the Python/C API, in particular the C structure. From rosuav at gmail.com Fri Mar 23 06:01:01 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 21:01:01 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! Message-ID: Apologies for letting this languish; life has an annoying habit of getting in the way now and then. Feedback from the previous rounds has been incorporated. From here, the most important concern and question is: Is there any other syntax or related proposal that ought to be mentioned here? If this proposal is rejected, it should be rejected with a full set of alternatives. Text of PEP is below; formatted version will be live shortly (if it isn't already) at: https://www.python.org/dev/peps/pep-0572/ ChrisA PEP: 572 Title: Syntax for Statement-Local Name Bindings Author: Chris Angelico Status: Draft Type: Standards Track Content-Type: text/x-rst Created: 28-Feb-2018 Python-Version: 3.8 Post-History: 28-Feb-2018, 02-Mar-2018, 23-Mar-2018 Abstract ======== Programming is all about reusing code rather than duplicating it. When an expression needs to be used twice in quick succession but never again, it is convenient to assign it to a temporary name with small scope. By permitting name bindings to exist within a single statement only, we make this both convenient and safe against name collisions. Rationale ========= When a subexpression is used multiple times in a list comprehension, there are currently several ways to spell this, none of which is universally accepted as ideal. A statement-local name allows any subexpression to be temporarily captured and then used multiple times. Additionally, this syntax can in places be used to remove the need to write an infinite loop with a ``break`` in it. Capturing part of a ``while`` loop's condition can improve the clarity of the loop header while still making the actual value available within the loop body. Syntax and semantics ==================== In any context where arbitrary Python expressions can be used, a **named expression** can appear. This must be parenthesized for clarity, and is of the form ``(expr as NAME)`` where ``expr`` is any valid Python expression, and ``NAME`` is a simple name. The value of such a named expression is the same as the incorporated expression, with the additional side-effect that NAME is bound to that value for the remainder of the current statement. Just as function-local names shadow global names for the scope of the function, statement-local names shadow other names for that statement. (They can technically also shadow each other, though actually doing this should not be encouraged.) Assignment to statement-local names is ONLY through this syntax. Regular assignment to the same name will remove the statement-local name and affect the name in the surrounding scope (function, class, or module). Statement-local names never appear in locals() or globals(), and cannot be closed over by nested functions. Execution order and its consequences ------------------------------------ Since the statement-local name binding lasts from its point of execution to the end of the current statement, this can potentially cause confusion when the actual order of execution does not match the programmer's expectations. Some examples:: # A simple statement ends at the newline or semicolon. a = (1 as y) print(y) # NameError # The assignment ignores the SLNB - this adds one to 'a' a = (a + 1 as a) # Compound statements usually enclose everything... if (re.match(...) as m): print(m.groups(0)) print(m) # NameError # ... except when function bodies are involved... if (input("> ") as cmd): def run_cmd(): print("Running command", cmd) # NameError # ... but function *headers* are executed immediately if (input("> ") as cmd): def run_cmd(cmd=cmd): # Capture the value in the default arg print("Running command", cmd) # Works Function bodies, in this respect, behave the same way they do in class scope; assigned names are not closed over by method definitions. Defining a function inside a loop already has potentially-confusing consequences, and SLNBs do not materially worsen the existing situation. Differences from regular assignment statements ---------------------------------------------- Using ``(EXPR as NAME)`` is similar to ``NAME = EXPR``, but has a number of important distinctions. * Assignment is a statement; an SLNB is an expression whose value is the same as the object bound to the new name. * SLNBs disappear at the end of their enclosing statement, at which point the name again refers to whatever it previously would have. SLNBs can thus shadow other names without conflict (although deliberately doing so will often be a sign of bad code). * SLNBs cannot be closed over by nested functions, and are completely ignored for this purpose. * SLNBs do not appear in ``locals()`` or ``globals()``. * An SLNB cannot be the target of any form of assignment, including augmented. Attempting to do so will remove the SLNB and assign to the fully-scoped name. In many respects, an SLNB is akin to a local variable in an imaginary nested function, except that the overhead of creating and calling a function is bypassed. As with names bound by ``for`` loops inside list comprehensions, SLNBs cannot "leak" into their surrounding scope. Example usage ============= These list comprehensions are all approximately equivalent:: # Calling the function twice stuff = [[f(x), x/f(x)] for x in range(5)] # External helper function def pair(x, value): return [value, x/value] stuff = [pair(x, f(x)) for x in range(5)] # Inline helper function stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)] # Extra 'for' loop - potentially could be optimized internally stuff = [[y, x/y] for x in range(5) for y in [f(x)]] # Iterating over a genexp stuff = [[y, x/y] for x, y in ((x, f(x)) for x in range(5))] # Expanding the comprehension into a loop stuff = [] for x in range(5): y = f(x) stuff.append([y, x/y]) # Wrapping the loop in a generator function def g(): for x in range(5): y = f(x) yield [y, x/y] stuff = list(g()) # Using a statement-local name stuff = [[(f(x) as y), x/y] for x in range(5)] If calling ``f(x)`` is expensive or has side effects, the clean operation of the list comprehension gets muddled. Using a short-duration name binding retains the simplicity; while the extra ``for`` loop does achieve this, it does so at the cost of dividing the expression visually, putting the named part at the end of the comprehension instead of the beginning. Statement-local name bindings can be used in any context, but should be avoided where regular assignment can be used, just as ``lambda`` should be avoided when ``def`` is an option. As the name's scope extends to the full current statement, even a block statement, this can be used to good effect in the header of an ``if`` or ``while`` statement:: # Current Python, not caring about function return value while input("> ") != "quit": print("You entered a command.") # Current Python, capturing return value - four-line loop header while True: command = input("> "); if command == "quit": break print("You entered:", command) # Proposed alternative to the above while (input("> ") as command) != "quit": print("You entered:", command) # See, for instance, Lib/pydoc.py if (re.search(pat, text) as match): print("Found:", match.group(0)) while (sock.read() as data): print("Received data:", data) Particularly with the ``while`` loop, this can remove the need to have an infinite loop, an assignment, and a condition. It also creates a smooth parallel between a loop which simply uses a function call as its condition, and one which uses that as its condition but also uses the actual value. Performance costs ================= The cost of SLNBs must be kept to a minimum, particularly when they are not used; the normal case MUST NOT be measurably penalized. SLNBs are expected to be uncommon, and using many of them in a single function should definitely be discouraged. Thus the current implementation uses a linked list of SLNB cells, with the absence of such a list being the normal case. This list is used for code compilation only; once a function's bytecode has been baked in, execution of that bytecode has no performance cost compared to regular assignment. Other Python implementations may choose to do things differently, but a zero run-time cost is strongly recommended, as is a minimal compile-time cost in the case where no SLNBs are used. Forbidden special cases ======================= In two situations, the use of SLNBs makes no sense, and could be confusing due to the ``as`` keyword already having a different meaning in the same context. 1. Exception catching:: try: ... except (Exception as e1) as e2: ... The expression ``(Exception as e1)`` has the value ``Exception``, and creates an SLNB ``e1 = Exception``. This is generally useless, and creates the potential confusion in that these two statements do quite different things: except (Exception as e1): except Exception as e2: The latter captures the exception **instance**, while the former captures the ``Exception`` **type** (not the type of the raised exception). 2. Context managers:: lock = threading.Lock() with (lock as l) as m: ... This captures the original Lock object as ``l``, and the result of calling its ``__enter__`` method as ``m``. As with ``except`` statements, this creates a situation in which parenthesizing an expression subtly changes its semantics, with the additional pitfall that this will frequently work (when ``x.__enter__()`` returns x, eg with file objects). Both of these are forbidden; creating SLNBs in the headers of these statements will result in a SyntaxError. Alternative proposals ===================== Proposals broadly similar to this one have come up frequently on python-ideas. Below are a number of alternative syntaxes, some of them specific to comprehensions, which have been rejected in favour of the one given above. 1. ``where``, ``let``, ``given``:: stuff = [(y, x/y) where y = f(x) for x in range(5)] stuff = [(y, x/y) let y = f(x) for x in range(5)] stuff = [(y, x/y) given y = f(x) for x in range(5)] This brings the subexpression to a location in between the 'for' loop and the expression. It introduces an additional language keyword, which creates conflicts. Of the three, ``where`` reads the most cleanly, but also has the greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` methods, as does ``tkinter.dnd.Icon`` in the standard library). 2. ``with NAME = EXPR``:: stuff = [(y, x/y) with y = f(x) for x in range(5)] As above, but reusing the `with` keyword. Doesn't read too badly, and needs no additional language keyword. Is restricted to comprehensions, though, and cannot as easily be transformed into "longhand" for-loop syntax. Has the C problem that an equals sign in an expression can now create a name binding, rather than performing a comparison. Would raise the question of why "with NAME = EXPR:" cannot be used as a statement on its own. 3. ``with EXPR as NAME``:: stuff = [(y, x/y) with f(x) as y for x in range(5)] As per option 2, but using ``as`` rather than an equals sign. Aligns syntactically with other uses of ``as`` for name binding, but a simple transformation to for-loop longhand would create drastically different semantics; the meaning of ``with`` inside a comprehension would be completely different from the meaning as a stand-alone statement, while retaining identical syntax. 4. ``EXPR as NAME`` without parentheses:: stuff = [[f(x) as y, x/y] for x in range(5)] Omitting the parentheses from this PEP's proposed syntax introduces many syntactic ambiguities. Requiring them in all contexts leaves open the option to make them optional in specific situations where the syntax is unambiguous (cf generator expressions as sole parameters in function calls), but there is no plausible way to make them optional everywhere. 5. Adorning statement-local names with a leading dot:: stuff = [[(f(x) as .y), x/.y] for x in range(5)] This has the advantage that leaked usage can be readily detected, removing some forms of syntactic ambiguity. However, this would be the only place in Python where a variable's scope is encoded into its name, making refactoring harder. This syntax is quite viable, and could be promoted to become the current recommendation if its advantages are found to outweigh its cost. 6. Allowing ``(EXPR as NAME)`` to assign to any form of name. This is exactly the same as the promoted proposal, save that the name is bound in the same scope that it would otherwise have. Any expression can assign to any name, just as it would if the ``=`` operator had been used. Such variables would leak out of the statement into the enclosing function, subject to the regular behaviour of comprehensions (since they implicitly create a nested function, the name binding would be restricted to the comprehension itself, just as with the names bound by ``for`` loops). 7. Enhancing ``if`` and ``while`` syntax to permit the capture of their conditions:: if re.search(pat, text) as match: print("Found:", match.group(0)) This works beautifully if and ONLY if the desired condition is based on the truthiness of the captured value. It is thus effective for specific use-cases (regex matches, socket reads that return `''` when done), and completely useless in more complicated cases (eg where the condition is ``f(x) < 0`` and you want to capture the value of ``f(x)``). It also has no benefit to list comprehensions. Discrepancies in the current implementation =========================================== 1. SLNBs are implemented using a special (and mostly-invisible) name mangling. They may sometimes appear in globals() and/or locals() with their simple or mangled names (but buggily and unreliably). They should be suppressed as though they were guinea pigs. 2. The forbidden special cases do not yet raise SyntaxError. References ========== .. [1] Proof of concept / reference implementation (https://github.com/Rosuav/cpython/tree/statement-local-variables) Copyright ========= This document has been placed in the public domain. .. Local Variables: mode: indented-text indent-tabs-mode: nil sentence-end-double-space: t fill-column: 70 coding: utf-8 End: From solipsis at pitrou.net Fri Mar 23 06:25:54 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Fri, 23 Mar 2018 11:25:54 +0100 Subject: [Python-ideas] PEP proposal: unifying function/method classes References: <5AB41364.8010500@UGent.be> <9f7967f961e041ed9458543be77c0338@xmail102.UGent.be> <5AB49DDD.8040405@UGent.be> Message-ID: <20180323112554.3957d0b1@fsol> On Fri, 23 Mar 2018 07:25:33 +0100 Jeroen Demeyer wrote: > On 2018-03-23 00:36, Antoine Pitrou wrote: > > It does make sense, since the proposal sounds ambitious (and perhaps > > impossible without breaking compatibility). > > Well, *some* breakage of backwards compatibility will be unavoidable. > > > My plan (just a plan for now!) is to preserve backwards compatibility in > the following ways: > > * Existing Python attributes of functions/methods should continue to > exist and behave the same > > * The inspect module should give the same results as now (by changing > the implementation of some of the functions in inspect to match the new > classes) > > * Everything from the documented Python/C API. > > > This means that I might break compatibility in the following ways: > > * Changing the classes of functions/methods (this is the whole point of > this PEP). So anything involving isinstance() checks might break. > > * The undocumented parts of the Python/C API, in particular the C structure. One breaking change would be to add __get__ to C functions. This means e.g. the following: class MyClass: my_open = open would make my_open a MyClass method, therefore you would need to spell it: class MyClass: my_open = staticmethod(open) ... if you wanted MyClass().my_open('some file') to continue to work. Of course that might be considered a minor annoyance. Regards Antoine. From p.f.moore at gmail.com Fri Mar 23 06:38:19 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Mar 2018 10:38:19 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: Message-ID: On 23 March 2018 at 10:01, Chris Angelico wrote: > # ... except when function bodies are involved... > if (input("> ") as cmd): > def run_cmd(): > print("Running command", cmd) # NameError > > # ... but function *headers* are executed immediately > if (input("> ") as cmd): > def run_cmd(cmd=cmd): # Capture the value in the default arg > print("Running command", cmd) # Works What about cmd = "Something else" if (input("> ") as cmd): def run_cmd(): print("Running command", cmd) # Closes over the "outer" cmd, not the statement-local one? Did I get that right? I don't really like it if so (I think it's confusing) but I guess I could live with "well, don't do that then" as an answer. And I don't have a better interpretation. I'm still not convinced I like the proposal, but it's a lot cleaner than previous versions, so thanks for that. Far fewer places where I said "hmm, I don't understand the implications". Paul From rosuav at gmail.com Fri Mar 23 07:03:57 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 23 Mar 2018 22:03:57 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: Message-ID: On Fri, Mar 23, 2018 at 9:38 PM, Paul Moore wrote: > On 23 March 2018 at 10:01, Chris Angelico wrote: >> # ... except when function bodies are involved... >> if (input("> ") as cmd): >> def run_cmd(): >> print("Running command", cmd) # NameError >> >> # ... but function *headers* are executed immediately >> if (input("> ") as cmd): >> def run_cmd(cmd=cmd): # Capture the value in the default arg >> print("Running command", cmd) # Works > > What about > > cmd = "Something else" > if (input("> ") as cmd): > def run_cmd(): > print("Running command", cmd) # Closes over the "outer" > cmd, not the statement-local one? > > Did I get that right? I don't really like it if so (I think it's > confusing) but I guess I could live with "well, don't do that then" as > an answer. And I don't have a better interpretation. Yes, that would be it. And I agree: Don't do that. It's the same sort of confusion you'd get here: def f(): spam = 1 class C: spam = 2 def g(x=spam): print(spam) # prints 1 print(x) # prints 2 C.g() A class creates a scope that function bodies inside it don't close over, but their headers are still executed in that scope. So default argument values "see" those inner variables, but the body of the function doesn't. It's the same with SLNBs. > I'm still not convinced I like the proposal, but it's a lot cleaner > than previous versions, so thanks for that. Far fewer places where I > said "hmm, I don't understand the implications". Cool, thanks. That's the idea here. ChrisA From rhodri at kynesim.co.uk Fri Mar 23 06:52:00 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Fri, 23 Mar 2018 10:52:00 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: Message-ID: On 23/03/18 10:01, Chris Angelico wrote: > Apologies for letting this languish; life has an annoying habit of > getting in the way now and then. > > Feedback from the previous rounds has been incorporated. From here, > the most important concern and question is: Is there any other syntax > or related proposal that ought to be mentioned here? If this proposal > is rejected, it should be rejected with a full set of alternatives. Thank you very much, Chris. I think you've won me over on most points, though I'm not sure whether I'm overall +0 or -0 on the whole PEP :-) -- Rhodri James *-* Kynesim Ltd From steve.dower at python.org Fri Mar 23 10:26:41 2018 From: steve.dower at python.org (Steve Dower) Date: Fri, 23 Mar 2018 07:26:41 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> Message-ID: I had a colleague complaining to me the other day about having to search multiple packages for the right function to move a file (implying: with the same semantics as drag-drop). If there isn?t a pathtools library on PyPI yet, this would certainly be valuable for newer developers. My view on Path is to either have everything on it or nothing on it (without removing what?s already there, of course), and since everything is so popular we should at least put everything in the one place. Top-posted from my Windows phone From: Mike Miller Sent: Monday, March 19, 2018 10:51 To: python-ideas at python.org Subject: Re: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 On 2018-03-18 10:55, Paul Moore wrote: >> Should Path() have methods to access all file operations? > > No, (Counterexample, having a Path operation to set Windows ACLs for a path). Agreed, not a big fan of everything filesystem-related in pathlib, simply because it doesn't read well. Having them scattered isn't a great experience either. Perhaps it would be better to have a filesystem package instead, maybe named "fs" that included all this stuff in one easy to use location. File stuff from os, path stuff from os.path, pathlib, utils like stat, and shutil etc? _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Fri Mar 23 11:00:58 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2018 02:00:58 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: Message-ID: <20180323150058.GU16661@ando.pearwood.info> On Fri, Mar 23, 2018 at 09:01:01PM +1100, Chris Angelico wrote: > PEP: 572 > Title: Syntax for Statement-Local Name Bindings [...] > Abstract > ======== > > Programming is all about reusing code rather than duplicating it. I don't think that editorial comment belongs here, or at least, it is way too strong. I'm pretty sure that programming is not ALL about reusing code, and code duplication is not always wrong. Rather, we can say that *often* we want to avoid code duplication, and this proposal is way way to do so. And this should go into the Rationale, not the Abstract. The abstract should describe what this proposal *does*, not why, for example: This is a proposal for permitting temporary name bindings which are limited to a single statement. What the proposal *is* goes in the Abstract; reasons *why* we want it go in the Rationale. I see you haven't mentioned anything about Nick Coglan's (long ago) concept of a "where" block. If memory serves, it would be something like: value = x**2 + 2*x where: x = some expression These are not necessarily competing, but they are relevant. Nor have you done a review of any other languages, to see what similar features they already offer. Not even the C's form of "assignment as an expression" -- you should refer to that, and explain why this would not similarly be a bug magnet. > Rationale > ========= > > When a subexpression is used multiple times in a list comprehension, I think that list comps are merely a single concrete example of a more general concept that we sometimes want or need to apply the DRY principle to a single expression. This is (usually) a violation of DRY whether it is inside or outside of a list comp: result = (func(x), func(x)+1, func(x)*2) > Syntax and semantics > ==================== > > In any context where arbitrary Python expressions can be used, a **named > expression** can appear. This must be parenthesized for clarity, and is of > the form ``(expr as NAME)`` where ``expr`` is any valid Python expression, > and ``NAME`` is a simple name. > > The value of such a named expression is the same as the incorporated > expression, with the additional side-effect that NAME is bound to that > value for the remainder of the current statement. Examples should go with the description. Such as: x = None if (spam().ham as eggs) is None else eggs y = ((spam() as eggs), (eggs.method() as cheese), cheese[eggs]) > Just as function-local names shadow global names for the scope of the > function, statement-local names shadow other names for that statement. > (They can technically also shadow each other, though actually doing this > should not be encouraged.) That seems weird. > Assignment to statement-local names is ONLY through this syntax. Regular > assignment to the same name will remove the statement-local name and > affect the name in the surrounding scope (function, class, or module). That seems unnecessary. Since the scope only applies to a single statement, not a block, there can be no other assignment to that name. Correction: I see further in that this isn't the case. But that's deeply confusing, to have the same name refer to two (or more!) scopes in the same block. I think that's going to lead to some really confusing scoping problems. > Statement-local names never appear in locals() or globals(), and cannot be > closed over by nested functions. Why can they not be used in closures? I expect that's going to cause a lot of frustration. > Execution order and its consequences > ------------------------------------ > > Since the statement-local name binding lasts from its point of execution > to the end of the current statement, this can potentially cause confusion > when the actual order of execution does not match the programmer's > expectations. Some examples:: > > # A simple statement ends at the newline or semicolon. > a = (1 as y) > print(y) # NameError That error surprises me. Every other use of "as" binds to the current local namespace. (Or global, if you use the global declaration first.) I think there's going to be a lot of confusion about which uses of "as" bind to a new local and which don't. I think this proposal is conflating two unrelated concepts: - introducing new variables in order to meet DRY requirements; - introducing a new scope. Why can't we do the first without the second? a = (1 as y) print(y) # prints 1, as other uses of "as" would do That would avoid the unnecessary (IMO) restriction that these variables cannot be used in closures. > # The assignment ignores the SLNB - this adds one to 'a' > a = (a + 1 as a) "SLNB"? Undefined acronym. What is it? I presume it has something to do with the single-statement variable. I know it would be legal, but why would you write something like that? Surely your examples must at least have a pretence of being useful (even if the examples are only toy examples rather than realistic). I think that having "a" be both local and single-statement in the same expression is an awful idea. Lua has the (mis-)features that variables are global by default, locals need to be declared, and the same variable name can refer to both local and global simultaneously. Thus we have: print(x) # prints the global x local x = x + 1 # sets local x to the global x plus 1 print(x) # prints the local x https://www.lua.org/pil/4.2.html This idea of local + single-statement names in the same expression strikes me as similar. Having that same sort of thing happening within a single statement gives me a headache: spam = (spam, ((spam + spam as spam) + spam as spam), spam) Explain that, if you will. > # Compound statements usually enclose everything... > if (re.match(...) as m): > print(m.groups(0)) > print(m) # NameError Ah, how surprising -- given the tone of this PEP, I honestly thought that it only applied to a single statement, not compound statements. You should mention this much earlier. > # ... except when function bodies are involved... > if (input("> ") as cmd): > def run_cmd(): > print("Running command", cmd) # NameError Such a special case is a violation of the Principle of Least Surprise. > # ... but function *headers* are executed immediately > if (input("> ") as cmd): > def run_cmd(cmd=cmd): # Capture the value in the default arg > print("Running command", cmd) # Works > > Function bodies, in this respect, behave the same way they do in class scope; > assigned names are not closed over by method definitions. Defining a function > inside a loop already has potentially-confusing consequences, and SLNBs do not > materially worsen the existing situation. Except by adding more complications to make it even harder to understand the scoping rules. > Differences from regular assignment statements > ---------------------------------------------- > > Using ``(EXPR as NAME)`` is similar to ``NAME = EXPR``, but has a number of > important distinctions. > > * Assignment is a statement; an SLNB is an expression whose value is the same > as the object bound to the new name. > * SLNBs disappear at the end of their enclosing statement, at which point the > name again refers to whatever it previously would have. SLNBs can thus > shadow other names without conflict (although deliberately doing so will > often be a sign of bad code). Why choose this design over binding to a local variable? What benefit is there to using yet another scope? > * SLNBs cannot be closed over by nested functions, and are completely ignored > for this purpose. What's the justification for this limitation? > * SLNBs do not appear in ``locals()`` or ``globals()``. That is like non-locals, so I suppose that's not unprecedented. Will there be a function slnbs() to retrieve these? > * An SLNB cannot be the target of any form of assignment, including augmented. > Attempting to do so will remove the SLNB and assign to the fully-scoped name. What's the justification for this limitation? > Example usage > ============= > > These list comprehensions are all approximately equivalent:: [...] I don't think you need to give an exhaustive list of every way to write a list comp. List comps are only a single use-case for this feature. > # See, for instance, Lib/pydoc.py > if (re.search(pat, text) as match): > print("Found:", match.group(0)) I do not believe that is actually code found in Lib/pydoc.py, since that will be a syntax error. What are you trying to say here? > while (sock.read() as data): > print("Received data:", data) Looking at that example, I wonder why we need to include the parens when there is no ambiguity. # okay while sock.read() as data: print("Received data:", data) # needs parentheses while (spam.method() as eggs) is None or eggs.count() < 100: print("something") > Performance costs > ================= > > The cost of SLNBs must be kept to a minimum, particularly when they are not > used; the normal case MUST NOT be measurably penalized. What is the "normal case"? It takes time, even if only a nanosecond, to bind a value to a name, as opposed to *not* binding it to a name. x = (spam as eggs) has to be more expensive than x = spam because the first performs two name bindings rather than one. So "MUST NOT" already implies this proposal *must* be rejected. Perhaps you mean that there SHOULD NOT be a SIGNIFICANT performance penalty. > SLNBs are expected to be uncommon, On what basis do you expect this? Me, I'm cynical about my fellow coders, because I've worked with them and read their code *wink* and I expect they'll use this everywhere "just in case" and "to avoid namespace pollution". But putting aside such (potential) abuse of the feature, I think you're under-cutting your own proposal. If this is really going to be uncommon, why bother complicating the language with a whole extra scope that hardly anyone is going to use but will be cryptic and mysterious on the rare occasion that they bump into it? Especially using a keyword that is already used elsewhere: "import as", "with as" and "except as" are going to dominate the search results. If this really will be uncommon, it's not worth it, but I don't think it would be uncommon. For good or ill, I think people will use this. Besides, I think that the while loop example is a really nice one. I'd use that, I think. I *almost* think that it alone justifies the exercise. > and using many of them in a single function should definitely > be discouraged. Do you mean a single statement? I don't see why it should be discouraged from using this many times in a single function. > Forbidden special cases > ======================= > > In two situations, the use of SLNBs makes no sense, and could be confusing due > to the ``as`` keyword already having a different meaning in the same context. I'm pretty sure there are many more than just two situations where the use of this makes no sense. Many of your examples perform an unnecessary name binding that is then never used. I think that's going to encourage programmers to do the same, especially when they read this PEP and think your examples are "Best Practice". Besides, in principle they could be useful (at least in contrived examples). Emember that exceptions are not necessarily constants. They can be computed at runtime: try: ... except (Errors[key], spam(Errors[key]): ... Since we have a DRY-violation in Errors[key] twice, it is conceivable that we could write: try: ... except ((Errors[key] as my_error), spam(my_error)): ... Contrived? Sure. But I think it makes sense. Perhaps a better argument is that it may be ambiguous with existing syntax, in which case the ambiguous cases should be banned. > Alternative proposals > ===================== > > Proposals broadly similar to this one have come up frequently on python-ideas. > Below are a number of alternative syntaxes, some of them specific to > comprehensions, which have been rejected in favour of the one given above. > > 1. ``where``, ``let``, ``given``:: > > stuff = [(y, x/y) where y = f(x) for x in range(5)] > stuff = [(y, x/y) let y = f(x) for x in range(5)] > stuff = [(y, x/y) given y = f(x) for x in range(5)] > > This brings the subexpression to a location in between the 'for' loop and > the expression. It introduces an additional language keyword, which creates > conflicts. Of the three, ``where`` reads the most cleanly, but also has the > greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` > methods, as does ``tkinter.dnd.Icon`` in the standard library). > > 2. ``with NAME = EXPR``:: > > stuff = [(y, x/y) with y = f(x) for x in range(5)] This is the same proposal as above, just using a different keyword. > 3. ``with EXPR as NAME``:: > > stuff = [(y, x/y) with f(x) as y for x in range(5)] Again, this isn't an alternative proposal, this is the same as 1. above just with different syntax. Likewise for 4. and 5. So you don't really have five different proposals, but only 1, with slight variations of syntax or semantics. They should be grouped together. "We have five different lunches available. Spam, spam and spam, spam deluxe, spam with eggs and spam, and chicken surprise." "What's the chicken surprise?" "It's actually made of spam." > 6. Allowing ``(EXPR as NAME)`` to assign to any form of name. And this would be a second proposal. > This is exactly the same as the promoted proposal, save that the name is > bound in the same scope that it would otherwise have. Any expression can > assign to any name, just as it would if the ``=`` operator had been used. > Such variables would leak out of the statement into the enclosing function, > subject to the regular behaviour of comprehensions (since they implicitly > create a nested function, the name binding would be restricted to the > comprehension itself, just as with the names bound by ``for`` loops). Indeed. Why are you rejecting this in favour of combining name-binding + new scope into a single syntax? -- Steve From christoph at grothesque.org Fri Mar 23 12:34:09 2018 From: christoph at grothesque.org (Christoph Groth) Date: Fri, 23 Mar 2018 17:34:09 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: Message-ID: <87370qn43i.fsf@grothesque.org> Disclaimer: I skimmed/searched through the PEP 572 threads (or should I say "literature"?) and did not find a discussion of the following point. If it has been discussed already, I'd be glad to be pointed to it. I am aware that Python, in contrast to C-like languages, has chosen not to treat assignments (=) as an expression with a value. The motivation for this is to avoid bugs due to confusion with the equality operator (==). But wouldn't it be a good alternative to PEP 572 to *add* an assignment operator that is an expression to Python, and is distinct from "=", for example ":="? Then, instead of PEP 572's: stuff = [[(f(x) as y), x/y] for x in range(5)] one could write stuff = [[(y := f(x)), x/y] for x in range(5)] In difference to PEP 572, the variable y would not be statement-local, which IMHO would be a welcome simplification. (PEP 572 introduces a third class of variables to Python.) Overall, it seems to me that introducing a new operator ":=" would serve the same purpose as PEP 572, but in a simpler and arguably cleaner way, while eliminating the risk of confusion with "==". The operator "=" would be left around, but could be deprecated in Python 5 and removed in Python 6. It would certainly suit a language that is widely used in education to sharpen the distinction between assignment and equality operators. Cheers, Christoph From guido at python.org Fri Mar 23 12:58:13 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 23 Mar 2018 09:58:13 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <87370qn43i.fsf@grothesque.org> References: <87370qn43i.fsf@grothesque.org> Message-ID: I also think it's fair to at least reconsider adding inline assignment, with the "traditional" semantics (possibly with mandatory parentheses). This would be easier to learn and understand for people who are familiar with it from other languages (C++, Java, JavaScript). On Fri, Mar 23, 2018 at 9:34 AM, Christoph Groth wrote: > Disclaimer: I skimmed/searched through the PEP 572 threads (or should I > say "literature"?) and did not find a discussion of the following point. > If it has been discussed already, I'd be glad to be pointed to it. > > I am aware that Python, in contrast to C-like languages, has chosen not > to treat assignments (=) as an expression with a value. The motivation > for this is to avoid bugs due to confusion with the equality operator > (==). > > But wouldn't it be a good alternative to PEP 572 to *add* an assignment > operator that is an expression to Python, and is distinct from "=", for > example ":="? Then, instead of PEP 572's: > > stuff = [[(f(x) as y), x/y] for x in range(5)] > > one could write > > stuff = [[(y := f(x)), x/y] for x in range(5)] > > In difference to PEP 572, the variable y would not be statement-local, > which IMHO would be a welcome simplification. (PEP 572 introduces a > third class of variables to Python.) > > Overall, it seems to me that introducing a new operator ":=" would serve > the same purpose as PEP 572, but in a simpler and arguably cleaner way, > while eliminating the risk of confusion with "==". The operator "=" > would be left around, but could be deprecated in Python 5 and removed in > Python 6. It would certainly suit a language that is widely used in > education to sharpen the distinction between assignment and equality > operators. > > Cheers, > Christoph > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 23 14:09:54 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 05:09:54 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180323150058.GU16661@ando.pearwood.info> References: <20180323150058.GU16661@ando.pearwood.info> Message-ID: On Sat, Mar 24, 2018 at 2:00 AM, Steven D'Aprano wrote: > On Fri, Mar 23, 2018 at 09:01:01PM +1100, Chris Angelico wrote: > >> PEP: 572 >> Title: Syntax for Statement-Local Name Bindings > [...] >> Abstract >> ======== >> >> Programming is all about reusing code rather than duplicating it. > > I don't think that editorial comment belongs here, or at least, it is > way too strong. I'm pretty sure that programming is not ALL about reusing code, > and code duplication is not always wrong. > > Rather, we can say that *often* we want to avoid code duplication, and > this proposal is way way to do so. And this should go into the > Rationale, not the Abstract. The abstract should describe what this > proposal *does*, not why, for example: > > This is a proposal for permitting temporary name bindings > which are limited to a single statement. > > What the proposal *is* goes in the Abstract; reasons *why* we want it > go in the Rationale. Thanks. I've never really been happy with my "Abstract" / "Rationale" split, as they're two sections both designed to give that initial 'sell', and I'm clearly not good at writing the distinction :) Unless you object, I'm just going to steal your Abstract wholesale. Seems like some good words there. > I see you haven't mentioned anything about Nick Coglan's (long ago) > concept of a "where" block. If memory serves, it would be something > like: > > value = x**2 + 2*x where: > x = some expression > > These are not necessarily competing, but they are relevant. Definitely relevant, thanks. This is exactly what I'm looking for - related proposals that got lost in the lengthy threads on the subject. I'll mention it as another proposal, but if anyone has an actual post for me to reference, that would be appreciated (just to make sure I'm correctly representing it). > Nor have you done a review of any other languages, to see what similar > features they already offer. Not even the C's form of "assignment as an > expression" -- you should refer to that, and explain why this would not > similarly be a bug magnet. No, I haven't yet. Sounds like a new section is needed. Thing is, there's a HUGE family of C-like and C-inspired languages that allow assignment expressions, and for the rest, I don't have any personal experience. So I need input from people: what languages do you know of that have small-scope name bindings like this? >> Rationale >> ========= >> >> When a subexpression is used multiple times in a list comprehension, > > I think that list comps are merely a single concrete example of a more > general concept that we sometimes want or need to apply the DRY > principle to a single expression. > > This is (usually) a violation of DRY whether it is inside or outside of > a list comp: > > result = (func(x), func(x)+1, func(x)*2) True, but outside of comprehensions, the most obvious response is "just add another assignment statement". You can't do that in a list comp (or equivalently in a genexp or dict comp). Syntactically you're right that they're just one example of a general concept; but they're one of the original motivating reasons. I've tweaked the rationale wording some; the idea is now "here's a general idea" followed by two paragraphs of specific use-cases (comprehensions and loops). Let me know if that works better. >> Syntax and semantics >> ==================== >> >> In any context where arbitrary Python expressions can be used, a **named >> expression** can appear. This must be parenthesized for clarity, and is of >> the form ``(expr as NAME)`` where ``expr`` is any valid Python expression, >> and ``NAME`` is a simple name. >> >> The value of such a named expression is the same as the incorporated >> expression, with the additional side-effect that NAME is bound to that >> value for the remainder of the current statement. > > > Examples should go with the description. Such as: > > x = None if (spam().ham as eggs) is None else eggs Not sure what you gain out of that :) Maybe a different first expression would help. > y = ((spam() as eggs), (eggs.method() as cheese), cheese[eggs]) Sure. I may need to get some simpler examples to kick things off though. >> Just as function-local names shadow global names for the scope of the >> function, statement-local names shadow other names for that statement. >> (They can technically also shadow each other, though actually doing this >> should not be encouraged.) > > That seems weird. Which part? That they shadow, or that they can shadow each other? Shadowing is the same as nested functions (including comprehensions, since they're implemented with functions); and if SLNBs are *not* to shadow each other, the only way is to straight-up disallow it. For the moment, I'm not forbidding it, as there's no particular advantage to popping a SyntaxError. >> Assignment to statement-local names is ONLY through this syntax. Regular >> assignment to the same name will remove the statement-local name and >> affect the name in the surrounding scope (function, class, or module). > > That seems unnecessary. Since the scope only applies to a single > statement, not a block, there can be no other assignment to that name. > > Correction: I see further in that this isn't the case. But that's deeply > confusing, to have the same name refer to two (or more!) scopes in the > same block. I think that's going to lead to some really confusing > scoping problems. For the current proposal, I prefer simpler definitions to outlawing the odd options. The rule is: An SLNB exists from the moment it's created to the end of that statement. Very simple, very straight-forward. Yes, that means you could use the same name earlier in the statement, but ideally, you just wouldn't do that. Python already has weirder behaviour in it. >>> def f(): ... e = 2.71828 ... try: ... 1/0 ... except Exception as e: ... print(e) ... print(e) ... >>> f() division by zero Traceback (most recent call last): File "", line 1, in File "", line 7, in f UnboundLocalError: local variable 'e' referenced before assignment Does this often cause problems? No, because most functions don't use the same name in two different ways. An SLNB should be basically the same. >> Statement-local names never appear in locals() or globals(), and cannot be >> closed over by nested functions. > > Why can they not be used in closures? I expect that's going to cause a > lot of frustration. Conceptually, the variable stops existing at the end of that statement. It makes for some oddities, but fewer oddities than every other variant that I toyed with. For example, does this create one single temporary or many different temporaries? def f(): x = "outer" funcs = {} for i in range(10): if (g(i) as x) > 0: def closure(): return x funcs[x] = closure Obviously the 'x' in funcs[x] is the current version of x as it runs through the loop. But what about the one closed over? If regular assignment is used ("x = g(i)"), the last value of x will be seen by every function. With a statement-local variable, should it be a single temporary all through the loop, or should each iteration create a brand new "slot" that gets closed over? If the latter, why is it different from regular assignment, and how would it be implemented anyway? Do we now need an infinite number of closure cells that all have the exact same name? >> Execution order and its consequences >> ------------------------------------ >> >> Since the statement-local name binding lasts from its point of execution >> to the end of the current statement, this can potentially cause confusion >> when the actual order of execution does not match the programmer's >> expectations. Some examples:: >> >> # A simple statement ends at the newline or semicolon. >> a = (1 as y) >> print(y) # NameError > > That error surprises me. Every other use of "as" binds to the > current local namespace. (Or global, if you use the global > declaration first.) > > I think there's going to be a lot of confusion about which uses of "as" > bind to a new local and which don't. That's the exact point of "statement-local" though. > I think this proposal is conflating two unrelated concepts: > > - introducing new variables in order to meet DRY requirements; > > - introducing a new scope. > > Why can't we do the first without the second? > > a = (1 as y) > print(y) # prints 1, as other uses of "as" would do > > > That would avoid the unnecessary (IMO) restriction that these variables > cannot be used in closures. You're talking about one of the alternate proposals there. (#6, currently.) I have talked about the possibility of splitting this into two separate proposals, but then I'd have to try to chair two separate concurrent discussions that would constantly interact and cross over :) >> # The assignment ignores the SLNB - this adds one to 'a' >> a = (a + 1 as a) > > "SLNB"? Undefined acronym. What is it? I presume it has something to do > with the single-statement variable. Statement-Local Name Binding, from the title of the PEP. (But people probably don't read titles.) > I know it would be legal, but why would you write something like that? > Surely your examples must at least have a pretence of being useful (even > if the examples are only toy examples rather than realistic). That section is about the edge cases, and one such edge case is assigning through an SLNB. > I think that having "a" be both local and single-statement in the same > expression is an awful idea. Lua has the (mis-)features that > variables are global by default, locals need to be declared, and the > same variable name can refer to both local and global simultaneously. > Thus we have: > > print(x) # prints the global x > local x = x + 1 # sets local x to the global x plus 1 > print(x) # prints the local x > > https://www.lua.org/pil/4.2.html IMO that's a *good* thing. JavaScript works the other way; either you say "var x = x + 1;" and the variable exists for the whole function, pre-initialized to the special value 'undefined', or you say "let x = x + 1;" and the variable is in limbo until you hit that statement, causing a ReferenceError (JS's version of NameError). Neither makes as much sense as evaluating the initializer before the variable starts to exist. That said, though, this is STILL an edge case. It's giving a somewhat-sane meaning to something you normally won't do. > This idea of local + single-statement names in the same expression > strikes me as similar. Having that same sort of thing happening within a > single statement gives me a headache: > > spam = (spam, ((spam + spam as spam) + spam as spam), spam) > > Explain that, if you will. Sure. First, eliminate all the name bindings: spam = (spam, ((spam + spam) + spam), spam) Okay. Now anyone with basic understanding of algebra can figure out the execution order. Then every time you have a construct with an 'as', you change the value of 'spam' from that point on. Which means we have: spam0 = (spam0, ((spam0 + spam0 as spam1) + spam1 as spam2), spam2) Execution order is strictly left-to-right here, so it's pretty straight-forward. Less clear if you have an if/else expression (since they're executed middle-out instead of left-to-right), but SLNBs are just like any other side effects in an expression, performed in a well-defined order. And just like with side effects, you don't want to have complex interactions between them, but there's nothing illegal in it. >> # Compound statements usually enclose everything... >> if (re.match(...) as m): >> print(m.groups(0)) >> print(m) # NameError > > Ah, how surprising -- given the tone of this PEP, I honestly thought > that it only applied to a single statement, not compound statements. > > You should mention this much earlier. Hmm. It's right up in the Rationale section, but without an example. Maybe an example would make it clearer? >> # ... except when function bodies are involved... >> if (input("> ") as cmd): >> def run_cmd(): >> print("Running command", cmd) # NameError > > Such a special case is a violation of the Principle of Least Surprise. Blame classes, which already do this. Exactly this. Being able to close over temporaries creates its own problems. >> # ... but function *headers* are executed immediately >> if (input("> ") as cmd): >> def run_cmd(cmd=cmd): # Capture the value in the default arg >> print("Running command", cmd) # Works >> >> Function bodies, in this respect, behave the same way they do in class scope; >> assigned names are not closed over by method definitions. Defining a function >> inside a loop already has potentially-confusing consequences, and SLNBs do not >> materially worsen the existing situation. > > Except by adding more complications to make it even harder to > understand the scoping rules. Except that I'm adding no complications. This is just the consequences of Python's *existing* scoping rules. >> Differences from regular assignment statements >> ---------------------------------------------- >> >> Using ``(EXPR as NAME)`` is similar to ``NAME = EXPR``, but has a number of >> important distinctions. >> >> * Assignment is a statement; an SLNB is an expression whose value is the same >> as the object bound to the new name. >> * SLNBs disappear at the end of their enclosing statement, at which point the >> name again refers to whatever it previously would have. SLNBs can thus >> shadow other names without conflict (although deliberately doing so will >> often be a sign of bad code). > > Why choose this design over binding to a local variable? What benefit is > there to using yet another scope? Mainly, I just know that there has been a lot of backlash against a generic "assignment as expression" syntax in the past. >> * SLNBs do not appear in ``locals()`` or ``globals()``. > > That is like non-locals, so I suppose that's not unprecedented. > > Will there be a function slnbs() to retrieve these? Not in the current proposal, no. Originally, I planned for them to appear in locals() while they were in scope, but that created its own problems; I'd be happy to return to that proposal if it were worthwhile. >> * An SLNB cannot be the target of any form of assignment, including augmented. >> Attempting to do so will remove the SLNB and assign to the fully-scoped name. > > What's the justification for this limitation? Not having that limitation creates worse problems, like that having "(1 as a)" somewhere can suddenly make an assignment fail. This is particularly notable with loop headers rather than simple statements. >> Example usage >> ============= >> >> These list comprehensions are all approximately equivalent:: > [...] > > I don't think you need to give an exhaustive list of every way to write > a list comp. List comps are only a single use-case for this feature. > > >> # See, for instance, Lib/pydoc.py >> if (re.search(pat, text) as match): >> print("Found:", match.group(0)) > > I do not believe that is actually code found in Lib/pydoc.py, since that > will be a syntax error. What are you trying to say here? Lib/pydoc.py has a more complicated version of the exact same functionality. This would be a simplification of a common idiom that can be found in the stdlib and elsewhere. >> while (sock.read() as data): >> print("Received data:", data) > > Looking at that example, I wonder why we need to include the parens when > there is no ambiguity. > > # okay > while sock.read() as data: > print("Received data:", data) > > # needs parentheses > while (spam.method() as eggs) is None or eggs.count() < 100: > print("something") I agree, but starting with them mandatory allows for future relaxation of requirements. The changes to the grammar are less intrusive if the parens are always required (for instance, the special case "f(x for x in y)" has its own entry in the grammar). >> Performance costs >> ================= >> >> The cost of SLNBs must be kept to a minimum, particularly when they are not >> used; the normal case MUST NOT be measurably penalized. > > What is the "normal case"? The case where you're not using any SLNBs. > It takes time, even if only a nanosecond, to bind a value to a > name, as opposed to *not* binding it to a name. > > x = (spam as eggs) > > has to be more expensive than > > x = spam > > because the first performs two name bindings rather than one. So "MUST > NOT" already implies this proposal *must* be rejected. Perhaps you mean > that there SHOULD NOT be a SIGNIFICANT performance penalty. The mere fact that this feature exists in the language MUST NOT measurably impact Python run-time performance. >> SLNBs are expected to be uncommon, > > On what basis do you expect this? > > Me, I'm cynical about my fellow coders, because I've worked with them > and read their code *wink* and I expect they'll use this everywhere > "just in case" and "to avoid namespace pollution". Compared to regular name bindings? Just look at the number of ways to assign that are NOT statement-local, and then add in the fact that SLNBs aren't going to be effective for anything that you need to mutate more than once, and I fully expect that regular name bindings will far exceed SLNBs. > Besides, I think that the while loop example is a really nice one. I'd > use that, I think. I *almost* think that it alone justifies the > exercise. Hmm, okay. I'll work on rewording that section later. >> Forbidden special cases >> ======================= >> >> In two situations, the use of SLNBs makes no sense, and could be confusing due >> to the ``as`` keyword already having a different meaning in the same context. > > I'm pretty sure there are many more than just two situations where the > use of this makes no sense. Many of your examples perform an unnecessary > name binding that is then never used. I think that's going to encourage > programmers to do the same, especially when they read this PEP and think > your examples are "Best Practice". Unnecessary, yes, but not downright problematic. The two specific cases mentioned are (a) evaluating expressions, and (b) using the 'as' keyword in a way that's incompatible with PEP 572. (There's no confusion in "import x as y", for instance, because "x" is not an expression.) > Besides, in principle they could be useful (at least in contrived > examples). Emember that exceptions are not necessarily constants. They > can be computed at runtime: > > try: > ... > except (Errors[key], spam(Errors[key]): > ... Sure they *can*. Have you ever seen something like that in production? I've seen simple examples (eg having a tuple of exception types that you care about, and that tuple not always being constant), but nothing where you could ever want an SLNB. > Since we have a DRY-violation in Errors[key] twice, it is conceivable > that we could write: > > try: > ... > except ((Errors[key] as my_error), spam(my_error)): > ... > > Contrived? Sure. But I think it makes sense. > > Perhaps a better argument is that it may be ambiguous with existing > syntax, in which case the ambiguous cases should be banned. It's not *technically* ambiguous, because PEP 572 demands parentheses and both 'except' and 'with' statements forbid parentheses. The compiler can, with 100% accuracy, pick between the two alternatives. But having "except X as Y:" mean something drastically different from "except (X as Y):" is confusing *to humans*. >> 2. ``with NAME = EXPR``:: >> >> stuff = [(y, x/y) with y = f(x) for x in range(5)] > > This is the same proposal as above, just using a different keyword. Yep. I've changed the heading to "Alternative proposals and variants" as some of them are merely variations on each other. They're given separate entries because I have separate commentary about them. >> 6. Allowing ``(EXPR as NAME)`` to assign to any form of name. > > And this would be a second proposal. > >> This is exactly the same as the promoted proposal, save that the name is >> bound in the same scope that it would otherwise have. Any expression can >> assign to any name, just as it would if the ``=`` operator had been used. >> Such variables would leak out of the statement into the enclosing function, >> subject to the regular behaviour of comprehensions (since they implicitly >> create a nested function, the name binding would be restricted to the >> comprehension itself, just as with the names bound by ``for`` loops). > > Indeed. Why are you rejecting this in favour of combining name-binding + > new scope into a single syntax? > Mainly because there's been a lot of backlash against regular assignment inside expressions. One thing I *have* learned from life is that you can't make everyone happy. Sometimes, "why isn't your proposal X instead of Y" is just "well, X is a valid proposal too, so you can go ahead and push for that one if you like". :) I had to pick something, and I picked that one. ChrisA From chris.barker at noaa.gov Fri Mar 23 16:09:22 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Fri, 23 Mar 2018 13:09:22 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: <4075V41VMzzFqVS@mail.python.org> References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> <4075V41VMzzFqVS@mail.python.org> Message-ID: On Mar 23, 2018, at 7:26 AM, Steve Dower wrote: I had a colleague complaining to me the other day about having to search multiple packages for the right function to move a file (implying: with the same semantics as drag-drop). Thanks Steve ? I know there isn?t any kind of consensus about exactly what should be part of a ?most file operations? package/class, but I?ve found it odd that there doesn?t even seem to be acknowledgment among longtime python users that the current hodgepodge is pretty dysfunctional for new users. One thing that?s been hammered home for me teaching newbies is that people get very far with computing these days without ever having touched a command line ? certainly not a *nix one. Even the concept of a working directory is foreign to many. So telling them that they can create and manipulate paths in nifty ways with the Path object, but then telling them to go to the os module to do simple things like rename or delete a file, and then looking at the docs for the os module, which starts with a bunch of references to other modules you need for not quite as simple things (shutil) or more complex things (tempfile). Then you look through the module and see a LOT of low level cryptic and semi-platform dependent functions ? not newbie friendly. Heck, I?ve been using Python for almost 20 years and I still have go look up whether it?s ?delete? or ?remove? or ?unlink?. And there is so much in os that ipython or IDE completion doesn?t help much. So as not to only write a critical rant ? here is a proposal of sorts: The python standard library should have one-stop shopping for the basic file system manipulations. ?Basic? could be guided by ?what users typically do with a GUI file manager? This could be yet another module, but given that navigating the file system ( I.e. path manipulation) is one of the things people typically do with a file manager, it makes sense to add it all to pathlib, maybe even all to the Path object. ( though maybe more advanced classes/functions could be added to pathlib as way to put it all in one place, while not expanding the Path namespace too much) Of course, this will require a substantial amount of work to work out the details in a PEP. I don?t have the time to do that, but if the OP or someone else does, I?ll help. -CHB If there isn?t a pathtools library on PyPI yet, this would certainly be valuable for newer developers. My view on Path is to either have everything on it or nothing on it (without removing what?s already there, of course), and since everything is so popular we should at least put everything in the one place. Top-posted from my Windows phone *From: *Mike Miller *Sent: *Monday, March 19, 2018 10:51 *To: *python-ideas at python.org *Subject: *Re: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 On 2018-03-18 10:55, Paul Moore wrote: >> Should Path() have methods to access all file operations? > > No, (Counterexample, having a Path operation to set Windows ACLs for a path). Agreed, not a big fan of everything filesystem-related in pathlib, simply because it doesn't read well. Having them scattered isn't a great experience either. Perhaps it would be better to have a filesystem package instead, maybe named "fs" that included all this stuff in one easy to use location. File stuff from os, path stuff from os.path, pathlib, utils like stat, and shutil etc? _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From jjmaldonis at gmail.com Fri Mar 23 16:47:34 2018 From: jjmaldonis at gmail.com (Jason Maldonis) Date: Fri, 23 Mar 2018 15:47:34 -0500 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> <4075V41VMzzFqVS@mail.python.org> Message-ID: > > I?ve found it odd that there doesn?t even seem to be acknowledgment among > longtime python users that the current hodgepodge is pretty dysfunctional > for new users. > I find this odd too. There have been a few comments along the lines of this being a problem for newbies or for *some* people (not sure who *some* is referring to), but I think that having these functions in different modules is weird for everyone. It just so happens that if you use file-operations a lot that you've gotten used to the system and know where to look, but it's still weird to have to use 2-3 different modules. Additionally, if you know where to look then it isn't really a hassle to import another module, whereas it is a hassle if you are less familiar with the functionality (regardless of whether you are a newbie or not). > Heck, I?ve been using Python for almost 20 years and I still have go look > up whether it?s ?delete? or ?remove? or ?unlink?. And there is so much in > os that ipython or IDE completion doesn?t help much. > Me too, although I've only been using python for ~ 10 years. > Of course, this will require a substantial amount of work to work out the > details in a PEP. I don?t have the time to do that, but if the OP or > someone else does, I?ll help. > I'd be happy to help as well, and if I can find the time I'll write the PEP (although I'd rather just contribute to it). Figuring out what to include and how to change the methods' APIs seems like one of the easier PEPs to write in terms of required knowledge (I don't know much about python's C code), so hopefully I would have a chance at doing an adequate job. If you think not, please tell me and I won't bother :) - Jason -------------- next part -------------- An HTML attachment was scrubbed... URL: From python-ideas at mgmiller.net Fri Mar 23 18:22:47 2018 From: python-ideas at mgmiller.net (Mike Miller) Date: Fri, 23 Mar 2018 15:22:47 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> <4075V41VMzzFqVS@mail.python.org> Message-ID: <0a0d60d2-64f9-815c-0724-e33b81b3920a@mgmiller.net> On 2018-03-23 13:47, Jason Maldonis wrote: > I?ve found it odd that there doesn?t even seem to be acknowledgment among > longtime python users that the current hodgepodge is pretty dysfunctional > for new users. I'll acknowledge it too, just that after X years it becomes second nature as you know. Tinkered around at lunchtime with a module to bring them under one roof. Named it "f2" because "fs" was taken on PyPI and thought the name should be very short since it could potentially be used in every script, like os and sys are. https://github.com/mixmastamyk/f2 Just poking around these questions came up about what to include: - Include file descriptors apis? - chroot? - l- functions to not follow links? - Unix dev files? obscure - pathconf? - supports_* api? - Only scandirs or walk too? - Should unpack os.path functions into the root? Handy but many of them. - path and Path in same module could confuse. - Unpack file-related shutil functions? Or leave as submodule? Perhaps this is enough to generate some comments. -Mike From wes.turner at gmail.com Fri Mar 23 18:38:17 2018 From: wes.turner at gmail.com (Wes Turner) Date: Fri, 23 Mar 2018 18:38:17 -0400 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: <0a0d60d2-64f9-815c-0724-e33b81b3920a@mgmiller.net> References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> <4075V41VMzzFqVS@mail.python.org> <0a0d60d2-64f9-815c-0724-e33b81b3920a@mgmiller.net> Message-ID: Here's a comparison table of os, os.path, shutil, pathlib, and path.py. The full version is at https://github.com/westurner/pyfilemods (README.rst) and at https://westurner.github.io/pyfilemods. I ran a few set intersections and went ahead and wrote a report to compare function/method signatures and sources. (cc'ed ironically, for the last time, I promise, from the other fork of this thread: https://mail.python.org/pipermail/python-ideas/2018-March/049375.html ) ... trio wraps pathlib methods with async; which also might as well be done in pathlib? - Does it make sense to copy the docstrings at import time every time? attr table ========== ================== == ======= ====== ======= ======= attr os os.path shutil pathlib path.py ================== == ======= ====== ======= ======= `__div__`_ X `__rdiv__`_ X `absolute`_ X `abspath`_ X X `access`_ X X `altsep`_ X X `anchor`_ X `as_posix`_ X `as_uri`_ X `atime`_ X `basename`_ X X `bytes`_ X `capitalize`_ X `casefold`_ X `cd`_ X `center`_ X `chdir`_ X X `chmod`_ X X X `chown`_ X X X `chroot`_ X X `chunks`_ X `commonpath`_ X `commonprefix`_ X `copy`_ X X `copy2`_ X X `copyfile`_ X X `copymode`_ X X `copystat`_ X X `copytree`_ X X `count`_ X `ctime`_ X `curdir`_ X X `cwd`_ X `defpath`_ X X `devnull`_ X X `dirname`_ X X `dirs`_ X `drive`_ X X `encode`_ X `endswith`_ X `exists`_ X X X `expand`_ X `expandtabs`_ X `expanduser`_ X X X `expandvars`_ X X `ext`_ X `extsep`_ X X `files`_ X `find`_ X `fnmatch`_ X X `format`_ X `format_map`_ X `get_owner`_ X `getatime`_ X X `getctime`_ X X `getcwd`_ X X `getmtime`_ X X `getsize`_ X X `glob`_ X X `group`_ X `home`_ X `in_place`_ X `index`_ X `is_absolute`_ X `is_block_device`_ X `is_char_device`_ X `is_dir`_ X `is_fifo`_ X `is_file`_ X `is_reserved`_ X `is_socket`_ X `is_symlink`_ X `isabs`_ X X `isalnum`_ X `isalpha`_ X `isdecimal`_ X `isdigit`_ X `isdir`_ X X `isfile`_ X X `isidentifier`_ X `islink`_ X X `islower`_ X `ismount`_ X X `isnumeric`_ X `isprintable`_ X `isspace`_ X `istitle`_ X `isupper`_ X `iterdir`_ X `join`_ X X `joinpath`_ X X `lchmod`_ X `lexists`_ X `lines`_ X `link`_ X X `listdir`_ X X `ljust`_ X `lower`_ X `lstat`_ X X X `lstrip`_ X `makedirs`_ X X `makedirs_p`_ X `maketrans`_ X `match`_ X `merge_tree`_ X `mkdir`_ X X X `mkdir_p`_ X `module`_ X `move`_ X X `mtime`_ X `name`_ X X X `namebase`_ X `normcase`_ X X `normpath`_ X X `open`_ X X X `os`_ X X `owner`_ X X `pardir`_ X X `parent`_ X X `parents`_ X `partition`_ X `parts`_ X `pathconf`_ X X `pathsep`_ X X `read_bytes`_ X `read_hash`_ X `read_hexhash`_ X `read_md5`_ X `read_text`_ X `readlink`_ X X `readlinkabs`_ X `realpath`_ X X `relative_to`_ X `relpath`_ X X `relpathto`_ X `remove`_ X X `remove_p`_ X `removedirs`_ X X `removedirs_p`_ X `rename`_ X X X `renames`_ X X `replace`_ X X X `resolve`_ X `rfind`_ X `rglob`_ X `rindex`_ X `rjust`_ X `rmdir`_ X X X `rmdir_p`_ X `rmtree`_ X X `rmtree_p`_ X `root`_ X `rpartition`_ X `rsplit`_ X `rstrip`_ X `samefile`_ X X X `sameopenfile`_ X `samestat`_ X `sep`_ X X `size`_ X `special`_ X `split`_ X X `splitall`_ X `splitdrive`_ X X `splitext`_ X X `splitlines`_ X `splitpath`_ X `splitunc`_ X `startswith`_ X `stat`_ X X X X X `statvfs`_ X X `stem`_ X X `strip`_ X `stripext`_ X `suffix`_ X `suffixes`_ X `swapcase`_ X `symlink`_ X X `symlink_to`_ X `text`_ X `title`_ X `touch`_ X X `translate`_ X `uncshare`_ X `unlink`_ X X X `unlink_p`_ X `upper`_ X `using_module`_ X `utime`_ X X `walk`_ X X `walkdirs`_ X `walkfiles`_ X `with_name`_ X `with_suffix`_ X X `write_bytes`_ X X `write_lines`_ X `write_text`_ X X `zfill`_ X ================== == ======= ====== ======= ======= On Fri, Mar 23, 2018 at 6:22 PM, Mike Miller wrote: > > On 2018-03-23 13:47, Jason Maldonis wrote: > >> I?ve found it odd that there doesn?t even seem to be acknowledgment >> among >> longtime python users that the current hodgepodge is pretty >> dysfunctional >> for new users. >> > > I'll acknowledge it too, just that after X years it becomes second nature > as you know. > > Tinkered around at lunchtime with a module to bring them under one roof. > Named it "f2" because "fs" was taken on PyPI and thought the name should be > very short since it could potentially be used in every script, like os and > sys are. > > https://github.com/mixmastamyk/f2 > > Just poking around these questions came up about what to include: > > - Include file descriptors apis? > - chroot? > - l- functions to not follow links? > - Unix dev files? obscure > - pathconf? > - supports_* api? > - Only scandirs or walk too? > > - Should unpack os.path functions into the root? Handy but many of > them. > - path and Path in same module could confuse. > > - Unpack file-related shutil functions? Or leave as submodule? > > > Perhaps this is enough to generate some comments. > > -Mike > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Fri Mar 23 18:57:22 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 23 Mar 2018 22:57:22 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: <0a0d60d2-64f9-815c-0724-e33b81b3920a@mgmiller.net> References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> <4075V41VMzzFqVS@mail.python.org> <0a0d60d2-64f9-815c-0724-e33b81b3920a@mgmiller.net> Message-ID: On 23 March 2018 at 22:22, Mike Miller wrote: > > On 2018-03-23 13:47, Jason Maldonis wrote: >> >> I?ve found it odd that there doesn?t even seem to be acknowledgment >> among >> longtime python users that the current hodgepodge is pretty >> dysfunctional >> for new users. > > > I'll acknowledge it too, just that after X years it becomes second nature as > you know. It's certainly easy to get so used to the current state of affairs that you forget that it's not "obvious" to a newcomer. I've been teaching a colleague Python and it's certainly something he would have problems with. > Tinkered around at lunchtime with a module to bring them under one roof. > Named it "f2" because "fs" was taken on PyPI and thought the name should be > very short since it could potentially be used in every script, like os and > sys are. > > https://github.com/mixmastamyk/f2 > > Just poking around these questions came up about what to include: > > - Include file descriptors apis? > - chroot? > - l- functions to not follow links? > - Unix dev files? obscure > - pathconf? > - supports_* api? > - Only scandirs or walk too? > > - Should unpack os.path functions into the root? Handy but many of > them. > - path and Path in same module could confuse. > > - Unpack file-related shutil functions? Or leave as submodule? > > > Perhaps this is enough to generate some comments. Yeah, this is the real issue. It's not that the general principle "have a single place where all filesystem stuff is found" is a particularly bad one, it's more that it's genuinely hard to know what the right answer is here. I'll add some other questions: - Should the module only contain cross-platform functions, or are platform-specific ones OK? - How about operations that appear on multiple levels (Steve Dower mentioned copy "that works like drag and drop", there's also about delete that handles recycle bins vs lower-level delete)? and the big one: - How will we handle backward compatibility? This is why a generic proposal like the one that started this thread is pretty useless in practice - it prompts debate, but doesn't really answer any of the important questions. Discussing functions on a case by case basis *will*, but it'll take a long time, and may miss the "big picture". A 3rd party module, like the f2 module you have created, is another approach - try it out and refine it over time, then when the various questions have been resolved, propose it for merging into the stdlib. There may be other ways to make progress, too. So I'm also OK with the general idea, I just don't see the value in simply rehashing all the old questions - we need something actionable if this is to go anywhere (and the current proposal simply isn't). Paul From chris.barker at noaa.gov Fri Mar 23 21:11:21 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Mar 2018 18:11:21 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> <4075V41VMzzFqVS@mail.python.org> Message-ID: On Fri, Mar 23, 2018 at 1:47 PM, Jason Maldonis > > I can find the time I'll write the PEP (although I'd rather just > contribute to it). Figuring out what to include and how to change the > methods' APIs seems like one of the easier PEPs to write in terms of > required knowledge (I don't know much about python's C code), so hopefully > I would have a chance at doing an adequate job. If you think not, please > tell me and I won't bother :) > Most of PEP writing (at least for a PEP like this) is capturing the debate and conclusions and capturing and moderating the discussion to some extent -- technical knowledge is the least of it. I expect you'll get plenty of help on the technical issues if there are any (I'm say if, 'cause at this point it doesn't look like anyone is proposing any *new* functionality, it's really just moving and renaming things. I suggest by starting with the a PEP outpine from a previous PEP, getting it into a a gitHub repo (it can start in your own account somewhere, then go through this thread to start sorting out the issues. I say gitHub repo 'cause then it's easy to publish and collect comments and improvements in a central place - CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Fri Mar 23 21:46:35 2018 From: chris.barker at noaa.gov (Chris Barker) Date: Fri, 23 Mar 2018 18:46:35 -0700 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> <4075V41VMzzFqVS@mail.python.org> <0a0d60d2-64f9-815c-0724-e33b81b3920a@mgmiller.net> Message-ID: On Fri, Mar 23, 2018 at 3:38 PM, Wes Turner wrote: > Here's a comparison table of os, os.path, shutil, pathlib, and path.py. > Darn, that's a big list -- exactly what we want to avoid :-( Though there are bunch of string methods in there that we can dump right off the bat. > ... trio wraps pathlib methods with async; which also might as well be done in pathlib? hmm --that's s touch one -- a full set of async file operations would be great. But where to put them???? and most of what we are talking about doesn't need asnyc, does it? deleting/renaming a single, file, etc. So my thought is that an asnyc lib should mirror / overwrite this file-operations-api. Which makes it a problem for the async lib(s) later on down the road. BTW -- this is a nice argument for making this API in the first place -- then we have a single API to make an async version of! - Does it make sense to copy the docstrings at import time every time? copy from where? is this for the async version? or are you thinking that the new methods will simply be wrappers for the olds ones -- in which case, I think no -- this should be a new API with its own docs, whether or not a particular call is any different that the "old" one. The big challenge is to identity what is "basic" or "normal" ratehr than advanced. here's a quick take on trimming down that whole table -- jsut my quick HO: attr table ========== ================== == ======= ====== ======= ======= attr os os.path shutil pathlib path.py ================== == ======= ====== ======= ======= `absolute`_ X `abspath`_ X X `access`_ X X `atime`_ X `basename`_ X X `cd`_ X `commonpath`_ X `commonprefix`_ X `copy`_ X X `copytree`_ X X `curdir`_ X X `dirname`_ X X `drive`_ X X `exists`_ X X X `expanduser`_ X X X `expandvars`_ X X `ext`_ X `fnmatch`_ X X `get_owner`_ X `getatime`_ X X `getctime`_ X X `getcwd`_ X X `getmtime`_ X X `getsize`_ X X `glob`_ X X `home`_ X `is_absolute`_ X `is_dir`_ X `is_symlink`_ X `isabs`_ X X `isdir`_ X X `isfile`_ X X `islink`_ X X `iterdir`_ X `lchmod`_ X `link`_ X X `listdir`_ X X `lstat`_ X X X `mkdir`_ X X X `makedirs`_ X X `match`_ X `merge_tree`_ X `move`_ X X `mtime`_ X `normcase`_ X X `normpath`_ X X `open`_ X X X `owner`_ X X `parent`_ X X `parents`_ X `relative_to`_ X `relpath`_ X X `relpathto`_ X `remove`_ X X `removedirs`_ X X `rename`_ X X X `replace`_ X X X `resolve`_ X `rglob`_ X `root`_ X `samefile`_ X X X `size`_ X `suffix`_ X `suffixes`_ X `symlink`_ X X `symlink_to`_ X `touch`_ X X `utime`_ X X `walk`_ X X `with_name`_ X `with_suffix`_ X X `write_bytes`_ X X `write_text`_ X X These are about permissions -- challenging how to do that in a X-platform way... And it would be good to group this sort of thing together. `chmod`_ X X X `chown`_ X X X ================== == ======= ====== ======= ======= Now that I've done that, I think it would be better to be systematic: 1) keep every method that pathlib.Path has 2) dump every method this is only a string method (and maybe everything that path has that nothign else has -- most of those are string methods 3) match the ones that are the same thing, but different names 4) match the ones that are almost the same thing:remove and removedir and removedirs - then we can hash out whether to join with flags, or keep separate 5) group by "class of functionality: - file nameing, etc - seraching (glob) - permission handling - etc. That'll get it structured to capture the discussion. -CHB > Tinkered around at lunchtime with a module to bring them under one roof. Named it "f2" because "fs" was taken on PyPI and thought the name should be very short since it could potentially be used in every script, like os and sys are. > > https://github.com/mixmastamyk/f2 > This is a great idea to have a prototype of sorts, but I do'nt think we should take the approach of: make it, put it on pypi, and see if it gains traction -- this isn't adding anythign new, very few folks are going to want to crate a dependency to get a more discoverable api, and no one going to point newbies at it. Just poking around these questions came up about what to include: > > - Include file descriptors apis? > no -- unless in an "advanced" api somewhere - chroot? > no > - l- functions to not follow links? > maybe adanced, or optional flags > - Unix dev files? obscure > don't even know what this is, so no :-) - pathconf? > again, what is this? probabaly no. > - supports_* api? > no. > - Only scandirs or walk too? > walk is really useful and not trivial to write, so yes. > - Should unpack os.path functions into the root? Handy but many of > them. > no way. > - Unpack file-related shutil functions? Or leave as submodule? > make them methods on Path. -CHB -- Christopher Barker, Ph.D. Oceanographer Emergency Response Division NOAA/NOS/OR&R (206) 526-6959 voice 7600 Sand Point Way NE (206) 526-6329 fax Seattle, WA 98115 (206) 526-6317 main reception Chris.Barker at noaa.gov -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Fri Mar 23 22:38:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 13:38:02 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87370qn43i.fsf@grothesque.org> Message-ID: On Sat, Mar 24, 2018 at 3:58 AM, Guido van Rossum wrote: > I also think it's fair to at least reconsider adding inline assignment, with > the "traditional" semantics (possibly with mandatory parentheses). This > would be easier to learn and understand for people who are familiar with it > from other languages (C++, Java, JavaScript). > > On Fri, Mar 23, 2018 at 9:34 AM, Christoph Groth > wrote: >> But wouldn't it be a good alternative to PEP 572 to *add* an assignment >> operator that is an expression to Python, and is distinct from "=", for >> example ":="? Thank you; both of these have now been incorporated into the document. ChrisA From ethan at stoneleaf.us Fri Mar 23 23:09:02 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Fri, 23 Mar 2018 20:09:02 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87370qn43i.fsf@grothesque.org> Message-ID: <5AB5C14E.4030603@stoneleaf.us> On 03/23/2018 07:38 PM, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 3:58 AM, Guido van Rossum wrote: >> On Fri, Mar 23, 2018 at 9:34 AM, Christoph Groth wrote: >>> But wouldn't it be a good alternative to PEP 572 to *add* an assignment >>> operator that is an expression to Python, and is distinct from "=", for >>> example ":="? >> >> I also think it's fair to at least reconsider adding inline assignment, with >> the "traditional" semantics (possibly with mandatory parentheses). This >> would be easier to learn and understand for people who are familiar with it >> from other languages (C++, Java, JavaScript). > > Thank you; both of these have now been incorporated into the document. I'm certainly hoping PEP 572 is rejected so we can have a follow-up PEP that only deals with the assignment-as-expression portion. No offense intended, Chris! :) In fact, maybe you could write that one too, and then have an accepted PEP to your name? ;) -- ~Ethan~ From rosuav at gmail.com Fri Mar 23 23:15:19 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 14:15:19 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5AB5C14E.4030603@stoneleaf.us> References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> Message-ID: On Sat, Mar 24, 2018 at 2:09 PM, Ethan Furman wrote: > On 03/23/2018 07:38 PM, Chris Angelico wrote: >> >> On Sat, Mar 24, 2018 at 3:58 AM, Guido van Rossum wrote: >>> >>> On Fri, Mar 23, 2018 at 9:34 AM, Christoph Groth wrote: > > >>>> But wouldn't it be a good alternative to PEP 572 to *add* an assignment >>>> operator that is an expression to Python, and is distinct from "=", for >>>> example ":="? > >>> >>> >>> I also think it's fair to at least reconsider adding inline assignment, >>> with >>> the "traditional" semantics (possibly with mandatory parentheses). This >>> would be easier to learn and understand for people who are familiar with >>> it >>> from other languages (C++, Java, JavaScript). >> >> >> Thank you; both of these have now been incorporated into the document. > > > I'm certainly hoping PEP 572 is rejected so we can have a follow-up PEP that > only deals with the assignment-as-expression portion. > > No offense intended, Chris! :) In fact, maybe you could write that one > too, and then have an accepted PEP to your name? ;) > Okay, maybe I *do* need to split them. There are other uses for statement-local names, so they're both potentially viable. Would that be the best way forward? If I do, sooner would be better than later - I'd want to grab PEP 573 while it's still available. ChrisA From ma3yuki.8mamo10 at gmail.com Sat Mar 24 00:16:38 2018 From: ma3yuki.8mamo10 at gmail.com (Masayuki YAMAMOTO) Date: Sat, 24 Mar 2018 13:16:38 +0900 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: FWIW, I thought another way which provides cache object library, it seems to just work in some cases. But it doesn't create statement local scope and might be difficult to read because looks ordinary expression doing magic. Chris, would you append the library to alternative proposal section? class Cache: """ Return a object which stores a value called with a keyword and immediately return it, then get the value using the attribute. """ def __call__(self, **kwargs): (k, v), = kwargs.items() # require one keyword setattr(self, k, v) return v c = Cache() print(c(spam="ham")) # => ham print(c.spam) # => ham L = [c(y=f(x)) + g(c.y) for x in range(5)] num = int(c.s) if c(s=eggs()).isdigit() else 0 if c(match=search(string)) is not None: print(c.match.groups()) while c(command=input()) != "quit": print("command is", c.command) -- Masayuki 2018-03-02 20:43 GMT+09:00 Chris Angelico : > After dozens of posts and a wide variety of useful opinions and > concerns being raised, here is the newest version of PEP 572 for your > debating pleasure. > > Formatted version: > > https://www.python.org/dev/peps/pep-0572/ > > There are now several more examples, greater clarity in edge cases, > and improved wording of the actual proposal and specifications. Also, > the reference implementation has been significantly enhanced, for > those who wish to try this themselves. > > ChrisA > > PEP: 572 > Title: Syntax for Statement-Local Name Bindings > Author: Chris Angelico > Status: Draft > Type: Standards Track > Content-Type: text/x-rst > Created: 28-Feb-2018 > Python-Version: 3.8 > Post-History: 28-Feb-2018, 02-Mar-2018 > > > Abstract > ======== > > Programming is all about reusing code rather than duplicating it. When > an expression needs to be used twice in quick succession but never again, > it is convenient to assign it to a temporary name with small scope. > By permitting name bindings to exist within a single statement only, we > make this both convenient and safe against name collisions. > > > Rationale > ========= > > When a subexpression is used multiple times in a list comprehension, there > are currently several ways to spell this, none of which is universally > accepted as ideal. A statement-local name allows any subexpression to be > temporarily captured and then used multiple times. > > Additionally, this syntax can in places be used to remove the need to > write an > infinite loop with a ``break`` in it. Capturing part of a ``while`` loop's > condition can improve the clarity of the loop header while still making the > actual value available within the loop body. > > > Syntax and semantics > ==================== > > In any context where arbitrary Python expressions can be used, a named > expression can appear. This must be parenthesized for clarity, and is of > the form ``(expr as NAME)`` where ``expr`` is any valid Python expression, > and ``NAME`` is a simple name. > > The value of such a named expression is the same as the incorporated > expression, with the additional side-effect that NAME is bound to that > value in all retrievals for the remainder of the current statement. > > Just as function-local names shadow global names for the scope of the > function, statement-local names shadow other names for that statement. > They can also shadow each other, though actually doing this should be > strongly discouraged in style guides. > > Assignment to statement-local names is ONLY through this syntax. Regular > assignment to the same name will remove the statement-local name and > affect the name in the surrounding scope (function, class, or module). > > Statement-local names never appear in locals() or globals(), and cannot be > closed over by nested functions. > > > Execution order and its consequences > ------------------------------------ > > Since the statement-local name binding lasts from its point of execution > to the end of the current statement, this can potentially cause confusion > when the actual order of execution does not match the programmer's > expectations. Some examples:: > > # A simple statement ends at the newline or semicolon. > a = (1 as y) > print(y) # NameError > > # The assignment ignores the SLNB - this adds one to 'a' > a = (a + 1 as a) > > # Compound statements usually enclose everything... > if (re.match(...) as m): > print(m.groups(0)) > print(m) # NameError > > # ... except when function bodies are involved... > if (input("> ") as cmd): > def run_cmd(): > print("Running command", cmd) # NameError > > # ... but function *headers* are executed immediately > if (input("> ") as cmd): > def run_cmd(cmd=cmd): # Capture the value in the default arg > print("Running command", cmd) # Works > > Some of these examples should be considered *bad code* and rejected by code > review and/or linters; they are not, however, illegal. > > > Example usage > ============= > > These list comprehensions are all approximately equivalent:: > > # Calling the function twice > stuff = [[f(x), x/f(x)] for x in range(5)] > > # External helper function > def pair(x, value): return [value, x/value] > stuff = [pair(x, f(x)) for x in range(5)] > > # Inline helper function > stuff = [(lambda y: [y,x/y])(f(x)) for x in range(5)] > > # Extra 'for' loop - see also Serhiy's optimization > stuff = [[y, x/y] for x in range(5) for y in [f(x)]] > > # Iterating over a genexp > stuff = [[y, x/y] for x, y in ((x, f(x)) for x in range(5))] > > # Expanding the comprehension into a loop > stuff = [] > for x in range(5): > y = f(x) > stuff.append([y, x/y]) > > # Wrapping the loop in a generator function > def g(): > for x in range(5): > y = f(x) > yield [y, x/y] > stuff = list(g) > > # Using a statement-local name > stuff = [[(f(x) as y), x/y] for x in range(5)] > > If calling ``f(x)`` is expensive or has side effects, the clean operation > of > the list comprehension gets muddled. Using a short-duration name binding > retains the simplicity; while the extra ``for`` loop does achieve this, it > does so at the cost of dividing the expression visually, putting the named > part at the end of the comprehension instead of the beginning. > > Statement-local name bindings can be used in any context, but should be > avoided where regular assignment can be used, just as ``lambda`` should be > avoided when ``def`` is an option. As the name's scope extends to the full > current statement, even a block statement, this can be used to good effect > in the header of an ``if`` or ``while`` statement:: > > # Current Python, not caring about function return value > while input("> ") != "quit": > print("You entered a command.") > > # Current Python, capturing return value - four-line loop header > while True: > command = input("> "); > if command == "quit": > break > print("You entered:", command) > > # Proposed alternative to the above > while (input("> ") as command) != "quit": > print("You entered:", command) > > # See, for instance, Lib/pydoc.py > if (re.search(pat, text) as match): > print("Found:", match.group(0)) > > while (sock.read() as data): > print("Received data:", data) > > Particularly with the ``while`` loop, this can remove the need to have an > infinite loop, an assignment, and a condition. It also creates a smooth > parallel between a loop which simply uses a function call as its condition, > and one which uses that as its condition but also uses the actual value. > > > Performance costs > ================= > > The cost of SLNBs must be kept to a minimum, particularly when they are not > used; the normal case MUST NOT be measurably penalized. SLNBs are expected > to be uncommon, and using many of them in a single function should > definitely > be discouraged. Thus the current implementation uses a linked list of SLNB > cells, with the absence of such a list being the normal case. This list is > used for code compilation only; once a function's bytecode has been baked > in, > execution of that bytecode has no performance cost compared to regular > assignment. > > Other Python implementations may choose to do things differently, but a > zero > run-time cost is strongly recommended, as is a minimal compile-time cost in > the case where no SLNBs are used. > > > Open questions > ============== > > 1. What happens if the name has already been used? ``(x, (1 as x), x)`` > Currently, prior usage functions as if the named expression did not > exist (following the usual lookup rules); the new name binding will > shadow the other name from the point where it is evaluated until the > end of the statement. Is this acceptable? Should it raise a syntax > error or warning? > > 2. Syntactic confusion in ``except`` statements. While technically > unambiguous, it is potentially confusing to humans. In Python 3.7, > parenthesizing ``except (Exception as e):`` is illegal, and there is no > reason to capture the exception type (as opposed to the exception > instance, as is done by the regular syntax). Should this be made > outright illegal, to prevent confusion? Can it be left to linters? > It may also (and independently) be of value to use a subscope for the > normal except clause binding, such that ``except Exception as e:`` will > no longer unbind a previous use of the name ``e``. > > 3. Similar confusion in ``with`` statements, with the difference that there > is good reason to capture the result of an expression, and it is also > very common for ``__enter__`` methods to return ``self``. In many > cases, > ``with expr as name:`` will do the same thing as ``with (expr as > name):``, > adding to the confusion. > > 4. Should closures be able to refer to statement-local names? Either way, > there will be edge cases that make no sense. Assigning to a name will > "push through" the SLNB and bind to the regular name; this means that a > statement ``x = x`` will promote the SLNB to full name, and thus has an > impact. Closing over statement-local names, however, introduces scope > and lifetime confusions, as it then becomes possible to have two > functions > in almost the same context, closing over the same name, referring to two > different cells. > > > Alternative proposals > ===================== > > Proposals of this nature have come up frequently on python-ideas. Below are > a number of alternative syntaxes, some of them specific to comprehensions, > which have been rejected in favour of the one given above. > > 1. ``where``, ``let``, ``given``:: > > stuff = [(y, x/y) where y = f(x) for x in range(5)] > stuff = [(y, x/y) let y = f(x) for x in range(5)] > stuff = [(y, x/y) given y = f(x) for x in range(5)] > > This brings the subexpression to a location in between the 'for' loop > and > the expression. It introduces an additional language keyword, which > creates > conflicts. Of the three, ``where`` reads the most cleanly, but also has > the > greatest potential for conflict (eg SQLAlchemy and numpy have ``where`` > methods, as does ``tkinter.dnd.Icon`` in the standard library). > > 2. ``with``:: > > stuff = [(y, x/y) with y = f(x) for x in range(5)] > > As above, but reusing the `with` keyword. Doesn't read too badly, and > needs > no additional language keyword. Is restricted to comprehensions, though, > and cannot as easily be transformed into "longhand" for-loop syntax. Has > the C problem that an equals sign in an expression can now create a name > binding, rather than performing a comparison. > > 3. ``with... as``:: > > stuff = [(y, x/y) with f(x) as y for x in range(5)] > > As per option 2, but using ``as`` in place of the equals sign. Aligns > syntactically with other uses of ``as`` for name binding, but a simple > transformation to for-loop longhand would create drastically different > semantics; the meaning of ``with`` inside a comprehension would be > completely different from the meaning as a stand-alone statement. > > 4. ``EXPR as NAME`` without parentheses:: > > stuff = [[f(x) as y, x/y] for x in range(5)] > > Omitting the parentheses from this PEP's proposed syntax introduces many > syntactic ambiguities. > > 5. Adorning statement-local names with a leading dot:: > > stuff = [[(f(x) as .y), x/.y] for x in range(5)] > > This has the advantage that leaked usage can be readily detected, > removing > some forms of syntactic ambiguity. However, this would be the only > place > in Python where a variable's scope is encoded into its name, making > refactoring harder. This syntax is quite viable, and could be promoted > to > become the current recommendation if its advantages are found to > outweigh > its cost. > > 6. Allowing ``(EXPR as NAME)`` to assign to any form of name. > > This is exactly the same as the promoted proposal, save that the name is > bound in the same scope that it would otherwise have. Any expression can > assign to any name, just as it would if the ``=`` operator had been > used. > > > Discrepancies in the current implementation > =========================================== > > 1. SLNBs are implemented using a special (and mostly-invisible) name > mangling. They may sometimes appear in globals() and/or locals() with > their simple or mangled names (but buggily and unreliably). They should > be suppressed as though they were guinea pigs. > > > References > ========== > > .. [1] Proof of concept / reference implementation > (https://github.com/Rosuav/cpython/tree/statement-local-variables) > > > Copyright > ========= > > This document has been placed in the public domain. > > > .. > Local Variables: > mode: indented-text > indent-tabs-mode: nil > sentence-end-double-space: t > fill-column: 70 > coding: utf-8 > End: > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 24 00:41:02 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sat, 24 Mar 2018 15:41:02 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> Message-ID: <20180324044102.GV16661@ando.pearwood.info> To keep this a manageable length, I've trimmed vigourously. Apologies in advance if I've been too enthusiastic with the trimming :-) On Sat, Mar 24, 2018 at 05:09:54AM +1100, Chris Angelico wrote: > No, I haven't yet. Sounds like a new section is needed. Thing is, > there's a HUGE family of C-like and C-inspired languages that allow > assignment expressions, and for the rest, I don't have any personal > experience. So I need input from people: what languages do you know of > that have small-scope name bindings like this? I don't know if this counts as "like this", but Lua has a do...end block that introduces a new scope. Something like this: x = 1 do: x = 2 print(x) # prints 2 end print(x) # prints 1 I think that's a neat concept, but I'm struggling to think what I would use it for. [...] > > result = (func(x), func(x)+1, func(x)*2) > > True, but outside of comprehensions, the most obvious response is > "just add another assignment statement". You can't do that in a list > comp (or equivalently in a genexp or dict comp). Yes you can: your PEP gives equivalents that work fine for list comps, starting with factorising the duplicate code out into a helper function, to using a separate loop to get assignment: [(spam, spam+1) for x in values for spam in (func(x),)] [(spam, spam+1) for spam in (func(x) for x in values)] They are the equivalent to "just add another assignment statement" for comprehensions. I acknowledge that comprehensions are the motivating example here, but I don't think they're the only justification for the concept. Strictly speaking, there's never a time that we cannot use a new assignment statement. But sometimes it is annoying or inconvenient. Consider a contrived example: TABLE = [ alpha, beta, gamma, delta, ... func(omega) + func(omega)**2 + func(omega)**3, ] Yes, I can pull out the duplication: temp = function(omega) TABLE = [ alpha, beta, gamma, delta, ... temp + temp**2 + temp**3, ] but that puts the definition of temp quite distant from its use. So this is arguably nicer: TABLE = [ alpha, beta, gamma, delta, ... (func(omega) as temp) + temp**2 + temp**3, ] > >> Just as function-local names shadow global names for the scope of the > >> function, statement-local names shadow other names for that statement. > >> (They can technically also shadow each other, though actually doing this > >> should not be encouraged.) > > > > That seems weird. > > Which part? That they shadow, or that they can shadow each other? Shadowing themselves. I'm still not convinced these should just shadow local variables. Of course locals will shadow nonlocals, which shadow globals, which shadow builtins. I'm just not sure that we gain much (enough?) to justify adding a new scope between what we already have: proposed statement-local local nonlocal class (only during class statement) global builtins I think that needs justification by more than just "it makes the implementation easier". > Shadowing is the same as nested functions (including comprehensions, > since they're implemented with functions); and if SLNBs are *not* to > shadow each other, the only way is to straight-up disallow it. Or they can just rebind to the same (statement-)local. E.g.: while ("spam" as x): assert x == "spam" while ("eggs" as x): assert x == "eggs" break assert x == "eggs" > > Why can they not be used in closures? I expect that's going to cause a > > lot of frustration. > > Conceptually, the variable stops existing at the end of that > statement. It makes for some oddities, but fewer oddities than every > other variant that I toyed with. For example, does this create one > single temporary or many different temporaries? > > def f(): > x = "outer" > funcs = {} > for i in range(10): > if (g(i) as x) > 0: > def closure(): > return x > funcs[x] = closure I think the rule should be either: - statement-locals actually *are* locals and so behave like locals; - statement-locals introduce a new scope, but still behave like locals with respect to closures. No need to introduce two separate modes of behaviour. (Or if there is such a need, then the PEP should explain it.) > > I think there's going to be a lot of confusion about which uses of "as" > > bind to a new local and which don't. > > That's the exact point of "statement-local" though. I don't think so. As I say: > > I think this proposal is conflating two unrelated concepts: > > > > - introducing new variables in order to meet DRY requirements; > > > > - introducing a new scope. If you're going to champion *both* concepts, then you need to justify them both in the PEP, not just assume its obvious why we want both together. > > "SLNB"? Undefined acronym. What is it? I presume it has something to do > > with the single-statement variable. > > Statement-Local Name Binding, from the title of the PEP. (But people > probably don't read titles.) Indeed. In case it isn't obvious, you should define the acronym the first time you use it in the PEP. > > This idea of local + single-statement names in the same expression > > strikes me as similar. Having that same sort of thing happening within a > > single statement gives me a headache: > > > > spam = (spam, ((spam + spam as spam) + spam as spam), spam) > > > > Explain that, if you will. > > Sure. First, eliminate all the name bindings: [...] The point is not that it cannot be explained, but that it requires careful thought to understand. An advantage of using just regular locals is that we don't have to think about the consequences of introducing two new scopes. Its all happening to the same "a" variable. > > Ah, how surprising -- given the tone of this PEP, I honestly thought > > that it only applied to a single statement, not compound statements. > > > > You should mention this much earlier. > > Hmm. It's right up in the Rationale section, but without an example. > Maybe an example would make it clearer? Yes :-) > >> * An SLNB cannot be the target of any form of assignment, including augmented. > >> Attempting to do so will remove the SLNB and assign to the fully-scoped name. > > > > What's the justification for this limitation? > > Not having that limitation creates worse problems, like that having > "(1 as a)" somewhere can suddenly make an assignment fail. This is > particularly notable with loop headers rather than simple statements. How and why would it fail? > >> # See, for instance, Lib/pydoc.py > >> if (re.search(pat, text) as match): > >> print("Found:", match.group(0)) > > > > I do not believe that is actually code found in Lib/pydoc.py, since that > > will be a syntax error. What are you trying to say here? > > Lib/pydoc.py has a more complicated version of the exact same > functionality. This would be a simplification of a common idiom that > can be found in the stdlib and elsewhere. Then the PEP should show a "Before" and "After". > >> Performance costs > >> ================= > >> > >> The cost of SLNBs must be kept to a minimum, particularly when they are not > >> used; the normal case MUST NOT be measurably penalized. > > > > What is the "normal case"? > > The case where you're not using any SLNBs. The PEP should make this more clear: "Any implementation must not include any significant performance cost to code that does not use statement-locals." > > It takes time, even if only a nanosecond, to bind a value to a > > name, as opposed to *not* binding it to a name. > > > > x = (spam as eggs) > > > > has to be more expensive than > > > > x = spam > > > > because the first performs two name bindings rather than one. So "MUST > > NOT" already implies this proposal *must* be rejected. Perhaps you mean > > that there SHOULD NOT be a SIGNIFICANT performance penalty. > > The mere fact that this feature exists in the language MUST NOT > measurably impact Python run-time performance. MUST NOT implies that if there is *any* measurable penalty, even a nano-second, the feature must be rejected. I think that's excessive. Surely a nanosecond cost for the normal case is a reasonable tradeoff if it buys us better expressiveness? Beware of talking in absolutes unless you really mean them. Besides, as soon as you talk performance, the question has to be, which implementation? Of course we don't want to necessarily impose unreasonable performance and maintence costs on any implementation. But surely performance cost is a quality of implementation issue. It ought to be a matter of trade-offs: is the benefit sufficient to make up for the cost? -- Steve From ncoghlan at gmail.com Sat Mar 24 04:09:56 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Mar 2018 18:09:56 +1000 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: <5AB41364.8010500@UGent.be> References: <5AB41364.8010500@UGent.be> Message-ID: On 23 March 2018 at 06:34, Jeroen Demeyer wrote: > Dear python-ideas, > > I would like to draft a PEP to change the implementation of functions and > methods in CPython. The idea for this PEP come from a discussion on > https://bugs.python.org/issue30071 > > The core idea of the PEP is to reorganize classes for functions and > methods with the goal of removing the difference between functions/methods > implemented in C and functions/methods implemented in Python. > Reading the discussion on the issue, this feels more to me like PEP 357, which added the __index__ protocol such that arbitrary objects could be used for slicing. In this case, rather than the problem being operations that require a "native Python integer", the issue is with operations which currently require a native Python function object, even though most of the operations that folks implement on them can be successfully ducktyped. As Antoine notes, unifying user-defined functions and builtin functions would be fraught with backwards compatibility problems, so you probably don't want to go down that path when your goal is "Let third parties define types that are recognised by the core interpreter as being user-defined functions". If it was just about introspection, then we could define a new protocol method or attribute, update the inspect module to respect it, and call it done. However, it seems to me that the most plausible path towards granting Cython access to the optimised fast paths for native functions would be setting Py_TPFLAGS_BASETYPE on types.FunctionType, and introducing a new Py_TPFLAGS_FUNCTION_SUBCLASS flag. We can't introduce a new protocol into those paths without slowing them down, but we can replace the current PyFunction_Check() with the typical combination of a PyFunction_Check() that accepts subclasses (by looking for the new subclass flag if the exact check fails) and PyFunction_CheckExact() (which would continue to disallow native function subclasses). The PyFunction_Check() calls in the code base would then need to be reviewed to determine which of them should be replaced with PyFunction_CheckExact(). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 24 04:35:32 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Mar 2018 18:35:32 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> Message-ID: On 24 March 2018 at 13:15, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 2:09 PM, Ethan Furman wrote: > > I'm certainly hoping PEP 572 is rejected so we can have a follow-up PEP > that > > only deals with the assignment-as-expression portion. > > > > No offense intended, Chris! :) In fact, maybe you could write that one > > too, and then have an accepted PEP to your name? ;) > > > > Okay, maybe I *do* need to split them. There are other uses for > statement-local names, so they're both potentially viable. Would that > be the best way forward? > > If I do, sooner would be better than later - I'd want to grab PEP 573 > while it's still available. > Inline assignment (whether using the "expr as name" syntax or a new dedicated "x := y" one) certainly comes up often enough that it would be worth writing up independently of any of the statement local scoping proposals. One benefit of " x := y" is that it would avoid ambiguity in the with statement and exception handling cases: with cm := cm_expr as enter_result: ... try: ... except exc_filter := exceptions_to_catch as caught_exc: ... In comprehensions and generator expressions, we'd need to explain why inline assignments in the outermost iterator expression leak but those in filter expressions, inner iterator expressions, and result expressions don't. It would also result in two different ways to handle traditional assignments: x = expr x := expr Perhaps ":=" could be explicitly restricted to only single names on the LHS, without any of the clever unpacking features of full assignment statements? Unlike full assignment statements, assignment expressions also wouldn't have anywhere to put a type annotation. That way, there'd technically be overlap between the two in the simple case of assigning to a single name, but in other circumstances, it would be obvious which one you needed to use. Cheers, Nick. P.S. Pascal was one of the first languages I used to write a non-trivial application (a game of Battleships), so I'm predisposed towards liking ":=" as an assignment operator :) -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 24 04:53:53 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Mar 2018 18:53:53 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> Message-ID: On 24 March 2018 at 04:09, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 2:00 AM, Steven D'Aprano > wrote: > > I see you haven't mentioned anything about Nick Coglan's (long ago) > > concept of a "where" block. If memory serves, it would be something > > like: > > > > value = x**2 + 2*x where: > > x = some expression > > > > These are not necessarily competing, but they are relevant. > > Definitely relevant, thanks. This is exactly what I'm looking for - > related proposals that got lost in the lengthy threads on the subject. > I'll mention it as another proposal, but if anyone has an actual post > for me to reference, that would be appreciated (just to make sure I'm > correctly representing it). > That one's a PEP reference: https://www.python.org/dev/peps/pep-3150/ If PEP 572 were to happen, then I'd see some variant of PEP 3150 as a potential future follow-on (allowing the statement local namespace for a simple statement to be populated in a trailing suite, without needing to make the case for statement locals in the first place). If inline local variable assignment were to happen instead, then PEP 3150 would continue to face the double hurdle of pitching both the semantic benefits of statement locals, while also pitching a syntax for defining them. FWIW, I like this version of the statement local proposal, and think it would avoid a lot of the quirks that otherwise arise when allowing expression level assignments. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 24 05:12:49 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sat, 24 Mar 2018 19:12:49 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180324044102.GV16661@ando.pearwood.info> References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> Message-ID: On 24 March 2018 at 14:41, Steven D'Aprano wrote: > On Sat, Mar 24, 2018 at 05:09:54AM +1100, Chris Angelico wrote: > > >> Just as function-local names shadow global names for the scope of the > > >> function, statement-local names shadow other names for that statement. > > >> (They can technically also shadow each other, though actually doing > this > > >> should not be encouraged.) > > > > > > That seems weird. > > > > Which part? That they shadow, or that they can shadow each other? > > Shadowing themselves. > > I'm still not convinced these should just shadow local variables. Of > course locals will shadow nonlocals, which shadow globals, which shadow > builtins. I'm just not sure that we gain much (enough?) to justify > adding a new scope between what we already have: > > proposed statement-local > local > nonlocal > class (only during class statement) > global > builtins > > I think that needs justification by more than just "it makes the > implementation easier". > Introducing the new scoping behaviour doesn't make the implementation easier, it makes it harder. However, there are specific aspects of how that proposed new scope works (like not being visible from nested scopes) that make the implementation easier, since they eliminate a whole swathe of otherwise complicated semantic questions :) At a user experience level, the aim of the scoping limitation is essentially to help improve "code snippet portability". Consider the following piece of code: squares = [x**2 for x in iterable] In Python 2.x, you not only have to check whether or not you're already using "squares" for something, you also need to check whether or not you're using "x", since the iteration variable leaks. In Python 3.x, you only need to check for "squares" usage, since the comprehension has its own inner scope, and any "x" binding you may have defined will be shadowed instead of being overwritten. For PEP 572, the most directly comparable example is code like this: # Any previous binding of "m" is lost completely on the next line m = re.match(...) if m: print(m.groups(0)) In order to re-use that snippet, you need to double-check the surrounding code and make sure that you're not overwriting an "m" variable already used somewhere else in the current scope. With PEP 572, you don't even need to look, since visibility of the "m" in the following snippet is automatically limited to the statement itself: if (re.match(...) as m): print(m.groups(0)) # Any previous binding of "m" is visible again here, and hence a common source of bugs is avoided :) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sat Mar 24 05:18:57 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 20:18:57 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <87tvt5lu3p.fsf@grothesque.org> References: <87tvt5lu3p.fsf@grothesque.org> Message-ID: On Sat, Mar 24, 2018 at 8:07 PM, Christoph Groth wrote: > Chris Angelico wrote: > >> Thank you; both of these have now been incorporated into the document. > > Thanks! Just a small comment. You wrote in the PEP draft: > >> # Name bindings inside list comprehensions usually won't leak >> ... >> # But occasionally they will! > > I don't understand what you mean here. If the (y as x) syntax is to > have, as you say, "the exact same semantics as regular assignment", then > assignments inside list comprehensions would always "leak". But this is > a good thing, because this is consistent with how Python behaves. Except that a list comprehension is implemented using an inner function. Very approximately: x = [n * m for n in range(4) for m in range(5)] def (iter): ret = [] for n in iter: for m in range(5): ret.append(n * m) return ret x = (iter(range(4)) So the first (outermost) iterable is actually evaluated in the caller's scope, but everything else is inside a subscope. Thus an assignment inside that first iterable WILL leak into the surrounding scope; but anywhere else, it won't. ChrisA From rosuav at gmail.com Sat Mar 24 05:25:04 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 20:25:04 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: Message-ID: On Sat, Mar 24, 2018 at 3:16 PM, Masayuki YAMAMOTO wrote: > FWIW, I thought another way which provides cache object library, it seems to > just work in some cases. But it doesn't create statement local scope and > might be difficult to read because looks ordinary expression doing magic. > Chris, would you append the library to alternative proposal section? > Done. In the interests of brevity, I used a version that just uses a dict, which is a bit less pretty but doesn't require a lengthy class statement. ChrisA From christoph at grothesque.org Sat Mar 24 05:07:38 2018 From: christoph at grothesque.org (Christoph Groth) Date: Sat, 24 Mar 2018 10:07:38 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: Message-ID: <87tvt5lu3p.fsf@grothesque.org> Chris Angelico wrote: > Thank you; both of these have now been incorporated into the document. Thanks! Just a small comment. You wrote in the PEP draft: > # Name bindings inside list comprehensions usually won't leak > ... > # But occasionally they will! I don't understand what you mean here. If the (y as x) syntax is to have, as you say, "the exact same semantics as regular assignment", then assignments inside list comprehensions would always "leak". But this is a good thing, because this is consistent with how Python behaves. Christoph From rosuav at gmail.com Sat Mar 24 05:49:08 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 20:49:08 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180324044102.GV16661@ando.pearwood.info> References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> Message-ID: On Sat, Mar 24, 2018 at 3:41 PM, Steven D'Aprano wrote: > To keep this a manageable length, I've trimmed vigourously. Apologies in > advance if I've been too enthusiastic with the trimming :-) > > On Sat, Mar 24, 2018 at 05:09:54AM +1100, Chris Angelico wrote: > >> No, I haven't yet. Sounds like a new section is needed. Thing is, >> there's a HUGE family of C-like and C-inspired languages that allow >> assignment expressions, and for the rest, I don't have any personal >> experience. So I need input from people: what languages do you know of >> that have small-scope name bindings like this? > > I don't know if this counts as "like this", but Lua has a do...end block > that introduces a new scope. Something like this: > > x = 1 > do: > x = 2 > print(x) # prints 2 > end > print(x) # prints 1 > > I think that's a neat concept, but I'm struggling to think what I would > use it for. Okay. I'll leave off for now, but if the split of PEPs happens, I'll need to revisit that. >> > result = (func(x), func(x)+1, func(x)*2) >> >> True, but outside of comprehensions, the most obvious response is >> "just add another assignment statement". You can't do that in a list >> comp (or equivalently in a genexp or dict comp). > > Yes you can: your PEP gives equivalents that work fine for list comps, > starting with factorising the duplicate code out into a helper function, > to using a separate loop to get assignment: > > [(spam, spam+1) for x in values for spam in (func(x),)] > > [(spam, spam+1) for spam in (func(x) for x in values)] > > They are the equivalent to "just add another assignment statement" for > comprehensions. They might be mechanically equivalent. They are not syntactically equivalent. This PEP is not about "hey let's do something in Python that's utterly impossible to do". It's "here's a much tidier way to spell something that currently has to be ugly". > Strictly speaking, there's never a time that we cannot use a new > assignment statement. But sometimes it is annoying or inconvenient. > Consider a contrived example: > > TABLE = [ > alpha, > beta, > gamma, > delta, > ... > func(omega) + func(omega)**2 + func(omega)**3, > ] > > > Yes, I can pull out the duplication: > > temp = function(omega) > TABLE = [ > alpha, > beta, > gamma, > delta, > ... > temp + temp**2 + temp**3, > ] > > but that puts the definition of temp quite distant from its use. So this > is arguably nicer: > > TABLE = [ > alpha, > beta, > gamma, > delta, > ... > (func(omega) as temp) + temp**2 + temp**3, > ] Right. Definitely advantageous (and another reason not to go with the comprehension-specific options). >> >> Just as function-local names shadow global names for the scope of the >> >> function, statement-local names shadow other names for that statement. >> >> (They can technically also shadow each other, though actually doing this >> >> should not be encouraged.) >> > >> > That seems weird. >> >> Which part? That they shadow, or that they can shadow each other? > > Shadowing themselves. > > I'm still not convinced these should just shadow local variables. Of > course locals will shadow nonlocals, which shadow globals, which shadow > builtins. I'm just not sure that we gain much (enough?) to justify > adding a new scope between what we already have: > > proposed statement-local > local > nonlocal > class (only during class statement) > global > builtins > > I think that needs justification by more than just "it makes the > implementation easier". Nick has answered this part better than I can, so I'll just say "yep, read his post". :) >> Shadowing is the same as nested functions (including comprehensions, >> since they're implemented with functions); and if SLNBs are *not* to >> shadow each other, the only way is to straight-up disallow it. > > Or they can just rebind to the same (statement-)local. E.g.: > > while ("spam" as x): > assert x == "spam" > while ("eggs" as x): > assert x == "eggs" > break > assert x == "eggs" That means that sometimes, ``while ("eggs" as x):`` creates a new variable, and sometimes it doesn't. Why should that be? If you change the way that "spam" is assigned to x, the semantics of the inner 'while' block shouldn't change. It creates a subscope, it uses that subscope, the subscope expires. Curtain comes down. By your proposal, you have to check whether 'x' is shadowing some other variable, and if so, what type. By mine, it doesn't matter; regardless of whether 'x' existed or not, regardless of whether there's any other x in any other scope, that loop behaves the same way. Function-local names give the same confidence. It doesn't matter what names you use inside a function (modulo 'global' or 'nonlocal' declarations) - they quietly shadow anything from the outside. You need only care about module names duplicating local names if you actually need to use both *in the same context*. Same with built-ins; it's fine to say "id = 42" inside a function as long as you aren't going to also use the built-in id() function in that exact function. Code migration is easy. >> > Why can they not be used in closures? I expect that's going to cause a >> > lot of frustration. >> >> Conceptually, the variable stops existing at the end of that >> statement. It makes for some oddities, but fewer oddities than every >> other variant that I toyed with. For example, does this create one >> single temporary or many different temporaries? >> >> def f(): >> x = "outer" >> funcs = {} >> for i in range(10): >> if (g(i) as x) > 0: >> def closure(): >> return x >> funcs[x] = closure > > I think the rule should be either: > > - statement-locals actually *are* locals and so behave like locals; > > - statement-locals introduce a new scope, but still behave like > locals with respect to closures. > > No need to introduce two separate modes of behaviour. (Or if there is > such a need, then the PEP should explain it.) That would basically mean locking in some form of semantics. For your first example, you're locking in the rule that "(g(i) as x)" is exactly the same as "x = g(i)", and you HAVE to then allow that this will potentially assign to global or nonlocal names as well (subject to the usual rules). In other words, you have assignment-as-expression without any form of subscoping. This is a plausible stance and may soon be becoming a separate PEP. But for your second, you're locking in the same oddities that a 'with' block has: that a variable is being "created" and "destroyed", yet it sticks around for the rest of the function, just in case. It's a source of some confusion to people that the name used in a 'with' statement is actually still valid afterwards. Or does it only stick around if there is a function to close over it? Honestly, I really want to toss this one into the "well don't do that" basket, and let the semantics be dictated by simplicity and cleanliness even if it means that a closure doesn't see that variable. >> > "SLNB"? Undefined acronym. What is it? I presume it has something to do >> > with the single-statement variable. >> >> Statement-Local Name Binding, from the title of the PEP. (But people >> probably don't read titles.) > > Indeed. In case it isn't obvious, you should define the acronym the > first time you use it in the PEP. Once again, I assumed too much of people. Expected them to actually read the stuff they're discussing. And once again, the universe reminds me that people aren't like that. Ah well. Will fix that next round of edits. >> >> * An SLNB cannot be the target of any form of assignment, including augmented. >> >> Attempting to do so will remove the SLNB and assign to the fully-scoped name. >> > >> > What's the justification for this limitation? >> >> Not having that limitation creates worse problems, like that having >> "(1 as a)" somewhere can suddenly make an assignment fail. This is >> particularly notable with loop headers rather than simple statements. > > How and why would it fail? a = (1 as a) With current semantics, this is equivalent to "a = 1". If assignment went into the SLNB, it would be equivalent to "pass". Which do you expect it to do? > MUST NOT implies that if there is *any* measurable penalty, even a > nano-second, the feature must be rejected. I think that's excessive. > Surely a nanosecond cost for the normal case is a reasonable tradeoff > if it buys us better expressiveness? Steve, you know how to time a piece of code. You debate these kinds of points on python-list frequently. Are you seriously trying to tell me that you could measure a single nanosecond in regular compiling and running of Python code? With my current implementation, there is an extremely small cost during compilation (a couple of checks of a pointer in a structure, and if it's never changed from its initial NULL, nothing else happens), and zero cost at run time. I believe that this counts as "no measurable penalty". > Beware of talking in absolutes unless you really mean them. > > Besides, as soon as you talk performance, the question has to be, which > implementation? > > Of course we don't want to necessarily impose unreasonable performance > and maintence costs on any implementation. But surely performance > cost is a quality of implementation issue. It ought to be a matter of > trade-offs: is the benefit sufficient to make up for the cost? I don't see where this comes in. Let's say that Jython can't implement this feature without a 10% slowdown in run-time performance even if these subscopes aren't used. What are you saying the PEP should say? That it's okay for this feature to hurt performance by 10%? Then it should be rightly rejected. Or that Jython is allowed to ignore this feature? Or what? ChrisA From p.f.moore at gmail.com Sat Mar 24 07:49:33 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 24 Mar 2018 11:49:33 +0000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87tvt5lu3p.fsf@grothesque.org> Message-ID: On 24 March 2018 at 09:18, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 8:07 PM, Christoph Groth > wrote: >> Chris Angelico wrote: >> >>> Thank you; both of these have now been incorporated into the document. >> >> Thanks! Just a small comment. You wrote in the PEP draft: >> >>> # Name bindings inside list comprehensions usually won't leak >>> ... >>> # But occasionally they will! >> >> I don't understand what you mean here. If the (y as x) syntax is to >> have, as you say, "the exact same semantics as regular assignment", then >> assignments inside list comprehensions would always "leak". But this is >> a good thing, because this is consistent with how Python behaves. > > Except that a list comprehension is implemented using an inner > function. Very approximately: > > x = [n * m for n in range(4) for m in range(5)] > > def (iter): > ret = [] > for n in iter: > for m in range(5): > ret.append(n * m) > return ret > x = (iter(range(4)) > > So the first (outermost) iterable is actually evaluated in the > caller's scope, but everything else is inside a subscope. Thus an > assignment inside that first iterable WILL leak into the surrounding > scope; but anywhere else, it won't. Wow, that's subtle (in a bad way!). I'd much rather that assignments don't leak at all - that seems to me to be the only correct design, although I understand that implementation practicalities mean it's hard to do. There's a lot of context snipped here. Is this about the variant that just does assignment without the new scope? If it is, then is there a similar issue with the actual proposal, or is that immune to this problem (I suspect that it's not immune, although the details may differ). Paul From rosuav at gmail.com Sat Mar 24 07:55:07 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 22:55:07 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87tvt5lu3p.fsf@grothesque.org> Message-ID: On Sat, Mar 24, 2018 at 10:49 PM, Paul Moore wrote: > On 24 March 2018 at 09:18, Chris Angelico wrote: >> Except that a list comprehension is implemented using an inner >> function. Very approximately: >> >> x = [n * m for n in range(4) for m in range(5)] >> >> def (iter): >> ret = [] >> for n in iter: >> for m in range(5): >> ret.append(n * m) >> return ret >> x = (iter(range(4)) >> >> So the first (outermost) iterable is actually evaluated in the >> caller's scope, but everything else is inside a subscope. Thus an >> assignment inside that first iterable WILL leak into the surrounding >> scope; but anywhere else, it won't. > > Wow, that's subtle (in a bad way!). I'd much rather that assignments > don't leak at all - that seems to me to be the only correct design, > although I understand that implementation practicalities mean it's > hard to do. > > There's a lot of context snipped here. Is this about the variant that > just does assignment without the new scope? If it is, then is there a > similar issue with the actual proposal, or is that immune to this > problem (I suspect that it's not immune, although the details may > differ). The code equivalence I gave above applies to the existing Python semantics. I'm not sure why the outermost iterable is evaluated in its enclosing context, but there must have been a good reason. ChrisA From p.f.moore at gmail.com Sat Mar 24 08:03:54 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 24 Mar 2018 12:03:54 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> Message-ID: On 24 March 2018 at 09:49, Chris Angelico wrote: >> Of course we don't want to necessarily impose unreasonable performance >> and maintence costs on any implementation. But surely performance >> cost is a quality of implementation issue. It ought to be a matter of >> trade-offs: is the benefit sufficient to make up for the cost? > > I don't see where this comes in. Let's say that Jython can't implement > this feature without a 10% slowdown in run-time performance even if > these subscopes aren't used. What are you saying the PEP should say? > That it's okay for this feature to hurt performance by 10%? Then it > should be rightly rejected. Or that Jython is allowed to ignore this > feature? Or what? I think the PEP should confirm that there's not expected to be a showstopper performance cost in implementing this feature in other Python implementations. That doesn't have to be a big deal - reaching out to the Jython, PyPy, Cython etc implementors and asking them for a quick sanity check that this doesn't impose unmanageable overheads should be sufficient. No need to make this too dogmatic. Paul From p.f.moore at gmail.com Sat Mar 24 08:06:30 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 24 Mar 2018 12:06:30 +0000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87tvt5lu3p.fsf@grothesque.org> Message-ID: On 24 March 2018 at 11:55, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 10:49 PM, Paul Moore wrote: >> On 24 March 2018 at 09:18, Chris Angelico wrote: >>> Except that a list comprehension is implemented using an inner >>> function. Very approximately: >>> >>> x = [n * m for n in range(4) for m in range(5)] >>> >>> def (iter): >>> ret = [] >>> for n in iter: >>> for m in range(5): >>> ret.append(n * m) >>> return ret >>> x = (iter(range(4)) >>> >>> So the first (outermost) iterable is actually evaluated in the >>> caller's scope, but everything else is inside a subscope. Thus an >>> assignment inside that first iterable WILL leak into the surrounding >>> scope; but anywhere else, it won't. >> >> Wow, that's subtle (in a bad way!). I'd much rather that assignments >> don't leak at all - that seems to me to be the only correct design, >> although I understand that implementation practicalities mean it's >> hard to do. >> >> There's a lot of context snipped here. Is this about the variant that >> just does assignment without the new scope? If it is, then is there a >> similar issue with the actual proposal, or is that immune to this >> problem (I suspect that it's not immune, although the details may >> differ). > > The code equivalence I gave above applies to the existing Python > semantics. I'm not sure why the outermost iterable is evaluated in its > enclosing context, but there must have been a good reason. Understood. But the current Python semantics doesn't have any way of binding a name as part of that outermost iterable. It's the interaction of the new feature (name binding in expressions) with the existing implementation (the first iterable is outside the constructed scope for the comprehension) that needs to be clarified and if necessary modified to avoid nasty consequences. Paul From rosuav at gmail.com Sat Mar 24 08:18:48 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sat, 24 Mar 2018 23:18:48 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87tvt5lu3p.fsf@grothesque.org> Message-ID: On Sat, Mar 24, 2018 at 11:06 PM, Paul Moore wrote: > On 24 March 2018 at 11:55, Chris Angelico wrote: >> On Sat, Mar 24, 2018 at 10:49 PM, Paul Moore wrote: >>> On 24 March 2018 at 09:18, Chris Angelico wrote: >>>> Except that a list comprehension is implemented using an inner >>>> function. Very approximately: >>>> >>>> x = [n * m for n in range(4) for m in range(5)] >>>> >>>> def (iter): >>>> ret = [] >>>> for n in iter: >>>> for m in range(5): >>>> ret.append(n * m) >>>> return ret >>>> x = (iter(range(4)) >>>> >>>> So the first (outermost) iterable is actually evaluated in the >>>> caller's scope, but everything else is inside a subscope. Thus an >>>> assignment inside that first iterable WILL leak into the surrounding >>>> scope; but anywhere else, it won't. >>> >>> Wow, that's subtle (in a bad way!). I'd much rather that assignments >>> don't leak at all - that seems to me to be the only correct design, >>> although I understand that implementation practicalities mean it's >>> hard to do. >>> >>> There's a lot of context snipped here. Is this about the variant that >>> just does assignment without the new scope? If it is, then is there a >>> similar issue with the actual proposal, or is that immune to this >>> problem (I suspect that it's not immune, although the details may >>> differ). >> >> The code equivalence I gave above applies to the existing Python >> semantics. I'm not sure why the outermost iterable is evaluated in its >> enclosing context, but there must have been a good reason. > > Understood. But the current Python semantics doesn't have any way of > binding a name as part of that outermost iterable. It's the > interaction of the new feature (name binding in expressions) with the > existing implementation (the first iterable is outside the constructed > scope for the comprehension) that needs to be clarified and if > necessary modified to avoid nasty consequences. And that's one reason that these assignment expressions make more sense as statement-locals. It'd effectively still be local to the list comp, even though it's in the outer scope. ChrisA From eric at trueblade.com Sat Mar 24 08:01:35 2018 From: eric at trueblade.com (Eric V. Smith) Date: Sat, 24 Mar 2018 08:01:35 -0400 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> Message-ID: On 3/24/2018 5:49 AM, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 3:41 PM, Steven D'Aprano wrote: >> To keep this a manageable length, I've trimmed vigourously. Apologies in >> advance if I've been too enthusiastic with the trimming :-) >> On Sat, Mar 24, 2018 at 05:09:54AM +1100, Chris Angelico wrote: ... >>>> "SLNB"? Undefined acronym. What is it? I presume it has something to do >>>> with the single-statement variable. >>> >>> Statement-Local Name Binding, from the title of the PEP. (But people >>> probably don't read titles.) >> >> Indeed. In case it isn't obvious, you should define the acronym the >> first time you use it in the PEP. > > Once again, I assumed too much of people. Expected them to actually > read the stuff they're discussing. And once again, the universe > reminds me that people aren't like that. Ah well. Will fix that next > round of edits. To be fair to the readers, you don't indicate you're going to use part of the title as an acronym later. I certainly didn't get it, either, and I read the title and PEP. So I'm in the group of people you assumed too much of. The traditional way to specify this would be to change part of the title or first usage to: "Statement-Local Name Binding (SLNB)". Which is a signal to the reader that you're going to use this later and they should remember it. I don't know if it's frowned upon, but I wouldn't put this in the title. Instead, I'd put it in the body of the PEP on first usage. And I'd also make that usage in the first paragraph, instead of many paragraphs in. Eric From p.f.moore at gmail.com Sat Mar 24 08:36:19 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Sat, 24 Mar 2018 12:36:19 +0000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87tvt5lu3p.fsf@grothesque.org> Message-ID: On 24 March 2018 at 12:18, Chris Angelico wrote: > On Sat, Mar 24, 2018 at 11:06 PM, Paul Moore wrote: >> On 24 March 2018 at 11:55, Chris Angelico wrote: >>> On Sat, Mar 24, 2018 at 10:49 PM, Paul Moore wrote: >>>> On 24 March 2018 at 09:18, Chris Angelico wrote: >>>>> Except that a list comprehension is implemented using an inner >>>>> function. Very approximately: >>>>> >>>>> x = [n * m for n in range(4) for m in range(5)] >>>>> >>>>> def (iter): >>>>> ret = [] >>>>> for n in iter: >>>>> for m in range(5): >>>>> ret.append(n * m) >>>>> return ret >>>>> x = (iter(range(4)) >>>>> >>>>> So the first (outermost) iterable is actually evaluated in the >>>>> caller's scope, but everything else is inside a subscope. Thus an >>>>> assignment inside that first iterable WILL leak into the surrounding >>>>> scope; but anywhere else, it won't. >>>> >>>> Wow, that's subtle (in a bad way!). I'd much rather that assignments >>>> don't leak at all - that seems to me to be the only correct design, >>>> although I understand that implementation practicalities mean it's >>>> hard to do. >>>> >>>> There's a lot of context snipped here. Is this about the variant that >>>> just does assignment without the new scope? If it is, then is there a >>>> similar issue with the actual proposal, or is that immune to this >>>> problem (I suspect that it's not immune, although the details may >>>> differ). >>> >>> The code equivalence I gave above applies to the existing Python >>> semantics. I'm not sure why the outermost iterable is evaluated in its >>> enclosing context, but there must have been a good reason. >> >> Understood. But the current Python semantics doesn't have any way of >> binding a name as part of that outermost iterable. It's the >> interaction of the new feature (name binding in expressions) with the >> existing implementation (the first iterable is outside the constructed >> scope for the comprehension) that needs to be clarified and if >> necessary modified to avoid nasty consequences. > > And that's one reason that these assignment expressions make more > sense as statement-locals. It'd effectively still be local to the list > comp, even though it's in the outer scope. OK, although I still think the interaction between the two scopes is sufficiently weird that it's worth spelling out... Paul From christoph at grothesque.org Sat Mar 24 09:29:57 2018 From: christoph at grothesque.org (Christoph Groth) Date: Sat, 24 Mar 2018 14:29:57 +0100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: Message-ID: <87po3tlhyi.fsf@grothesque.org> Chris Angelico wrote: > On Sat, Mar 24, 2018 at 8:07 PM, Christoph Groth > wrote: > > Chris Angelico wrote: > > > >> Thank you; both of these have now been incorporated into the document. > > > > Thanks! Just a small comment. You wrote in the PEP draft: > > > >> # Name bindings inside list comprehensions usually won't leak > >> ... > >> # But occasionally they will! > > > > I don't understand what you mean here. If the (y as x) syntax is to > > have, as you say, "the exact same semantics as regular assignment", then > > assignments inside list comprehensions would always "leak". But this is > > a good thing, because this is consistent with how Python behaves. > > > Except that a list comprehension is implemented using an inner > function. Very approximately: > > x = [n * m for n in range(4) for m in range(5)] > > def (iter): > ret = [] > for n in iter: > for m in range(5): > ret.append(n * m) > return ret > x = (iter(range(4)) Indeed, the Python language reference section 6.2.4 says: "Note that the comprehension is executed in a separate scope, so names assigned to in the target list don?t ?leak? into the enclosing scope." As far as I can tell this has only consequences for the loop index variables of the comprehension, because these are the only names that are assigned values inside the comprehension. After all, assignment is currently a statement in Python, so it's not possible inside a list comprehension to assign to other names. (Or am I overlooking something here?) > So the first (outermost) iterable is actually evaluated in the > caller's scope, but everything else is inside a subscope. Thus an > assignment inside that first iterable WILL leak into the surrounding > scope; but anywhere else, it won't. It seems to me that this is a CPython implementation detail. Couldn't x = [n * m for n in range(4) for m in range(5)] be equally well equivalent to def (): ret = [] for n in range(4): for m in range(5): ret.append(n * m) return ret x = () As far as I can tell, with today's Python both implementations (yours and mine) are indistinguishable by the user. Since Python 3 decided to put comprehensions in a scope of their own, that means that if an assignment operator ":=" is added to Python the list comprehension [(t := i**2, t**2) for i in (r := range(10))] should neither leak r, nor t nor i. Seems fine to me! Christoph From ncoghlan at gmail.com Sat Mar 24 10:01:16 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 00:01:16 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87tvt5lu3p.fsf@grothesque.org> Message-ID: On 24 March 2018 at 21:49, Paul Moore wrote: > On 24 March 2018 at 09:18, Chris Angelico wrote: > > So the first (outermost) iterable is actually evaluated in the > > caller's scope, but everything else is inside a subscope. Thus an > > assignment inside that first iterable WILL leak into the surrounding > > scope; but anywhere else, it won't. > > Wow, that's subtle (in a bad way!). I'd much rather that assignments > don't leak at all - that seems to me to be the only correct design, > although I understand that implementation practicalities mean it's > hard to do. > We can't do that because it would make comprehensions nigh-unusable at class scope (or, equivalently, when using exec with a separate locals namespace): class C: _sequence = "a b c d".split() _another_sequence = [f(item) for item in _sequence] "_sequence" needs to be resolved in the class scope and passed in as an argument in order for that to work, as the nested scope can't see it directly (since the implicit nested scope works like any other method definition for name resolution purposes). > There's a lot of context snipped here. Is this about the variant that > just does assignment without the new scope? If it is, then is there a > similar issue with the actual proposal, or is that immune to this > problem (I suspect that it's not immune, although the details may > differ). > PEP 572 is *mostly* immune, in that it's only the rest of the same statement that can see the name binding. The variant that *doesn't* introduce statement local scoping just leaks outright into the surrounding scope. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 24 10:14:04 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 00:14:04 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <87po3tlhyi.fsf@grothesque.org> References: <87po3tlhyi.fsf@grothesque.org> Message-ID: On 24 March 2018 at 23:29, Christoph Groth wrote: > > x = [n * m for n in range(4) for m in range(5)] > > be equally well equivalent to > > def (): > ret = [] > for n in range(4): > for m in range(5): > ret.append(n * m) > return ret > x = () > > As far as I can tell, with today's Python both implementations (yours > and mine) are indistinguishable by the user. > They can be distinguished, just not at module or function scope. To give a concrete example: ========== >>> class C: ... sequence = range(10) ... listcomp = [x for x in sequence] ... def works(data): ... return list(data) ... from_works = works(sequence) ... def fails(): ... return list(sequence) ... >>> C.listcomp [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> C.from_works [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >>> C.fails() Traceback (most recent call last): File "", line 1, in File "", line 8, in fails NameError: name 'sequence' is not defined ========== I think I did have an implementation that didn't do the "outermost iterator is evaluated in the outer scope" dance early on (back when I was still working on the initial version of the iteration variable hiding, before it was merged to the Py3k branch), but quickly changed it because relying solely on closures broke too much code (in addition to the class namespace case, you can create a situation with similar constraints by passing a separate locals namespace to exec). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 24 10:44:34 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 01:44:34 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> Message-ID: <20180324144432.GW16661@ando.pearwood.info> On Sat, Mar 24, 2018 at 07:12:49PM +1000, Nick Coghlan wrote: > > I think that needs justification by more than just "it makes the > > implementation easier". > > Introducing the new scoping behaviour doesn't make the implementation > easier, it makes it harder. [...] Perhaps I had misunderstood something Chris had said. > At a user experience level, the aim of the scoping limitation is > essentially to help improve "code snippet portability". > > Consider the following piece of code: > > squares = [x**2 for x in iterable] > > In Python 2.x, you not only have to check whether or not you're already > using "squares" for something, you also need to check whether or not you're > using "x", since the iteration variable leaks. I hear you, and I understand that some people had problems with leakage, but in my own experience, this was not a problem I ever had. On the contrary, it was occasionally useful (what was the last value x took before the comprehension finished?). The change to Python 3 non-leaking behaviour has solved no problem for me but taken away something which was nearly always harmless and very occasionally useful. So I don't find this to be an especially compelling argument. But at least comprehensions are intended to be almost entirely self-contained, so it's not actively harmful. But I can't say the same for additional sub-function scopes. > For PEP 572, the most directly comparable example is code like this: > > # Any previous binding of "m" is lost completely on the next line > m = re.match(...) > if m: > print(m.groups(0)) > > In order to re-use that snippet, you need to double-check the surrounding > code and make sure that you're not overwriting an "m" variable already used > somewhere else in the current scope. Yes. So what? I'm going to be doing that regardless of whether the interpreter places this use of m in its own scope or not. The scope as seen by the interpreter is not important. If all we cared about was avoiding name collisions, we could solve that by using 128-bit secret keys as variables: var_81c199e61e9f90fd023508aee3265ad9 We don't need multiple scopes to avoid name collisions, we just need to make sure they're all unique :-) But of course readability counts, and we write code to be read by people, not for the convenience of the interpreter. For that reason, whenever I paste a code snippet, I'm going to check the name and make a conscious decision whether to keep it or change it, and doing that means I have to check whether "m" is already in use regardless of whether or not the interpreter will keep the two (or more!) "m" variables. So this supposed benefit is really no benefit at all. I still am going to check "m" to see if it clashes. To the extent that this proposal to add sub-function scoping encourages people to do copy-paste coding without even renaming variables to something appropriate for the function they're pasted into, I think this will strongly hurts readability in the long run. > With PEP 572, you don't even need to look, since visibility of the "m" in > the following snippet is automatically limited to the statement itself: > > if (re.match(...) as m): > print(m.groups(0)) > # Any previous binding of "m" is visible again here, and hence a common > source of bugs is avoided :) Is this really a "common source of bugs"? Do you really mean to suggest that we should be able to copy and paste a code snippet into the middle of a function without checking how it integrates with the surrounding code? Because that's what it seems that you are saying. And not only that we should be able to do so, but that it is important enough that we should add a feature to encourage it? If people make a habit of pasting snippets of code into their functions without giving any thought to how it fits in with the rest of the function, then any resulting bugs are caused by carelessness and slap-dash technique, not the scoping rules of the language. The last thing I want to read is a function where the same name is used for two or three or a dozen different things, because the author happened to copy code snippets from elsewhere and didn't bother renaming things to be more appropriate. Nevermind whether the interpreter can keep track of which is which, I'm worried about *my* ability to keep track of which is which. I might be cynical about the professionalism and skills of the average programmer, but even I find it hard to believe that most people would actually do that. But since we're (surely?) going to be taking time to integrate the snippet with the rest of the function, the benefit of not having to check for duplicate variable names evaporates. We (hopefully!) will be checking for duplicates regardless of whether they are scoped to a single statement or not, because we don't want to read and maintain a function with the same name "x" representing a dozen different things at different times. I'm not opposed to re-using variable names for two different purposes within a single function. But when I do it, I do it because I made a conscious decision that: (1) the name is appropriate for both purposes; and (2) re-using the name does not lead to confusion or make the function hard to read. I don't re-use names because I've copied some snippet and can't be bothered to change the names. And I don't think we should be adding a feature to enable and justify that sort of poor practice. Comprehensions have their own scope, and that's at least harmless, if not beneficial, because they are self-contained single expressions. But this would add separate scopes to blocks: def function(): x = 1 if (spam as x): ... while (ham as x): ... # much later, deep in the function # possibly after some or all of those blocks have ended ... process(x) # which x is this? This would be three different variables all with the same name "x". To track the current value of x I have to track each of the x variables and which is currently in scope. I don't think we need sub-function scoping. I think it adds more complexity that outweighs whatever benefit it gives. -- Steve From kirillbalunov at gmail.com Sat Mar 24 11:03:53 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Sat, 24 Mar 2018 18:03:53 +0300 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87po3tlhyi.fsf@grothesque.org> Message-ID: 2018-03-24 17:14 GMT+03:00 Nick Coghlan : > > > They can be distinguished, just not at module or function scope. To give a > concrete example: > > ========== > >>> class C: > ... sequence = range(10) > ... listcomp = [x for x in sequence] > ... def works(data): > ... return list(data) > ... from_works = works(sequence) > ... def fails(): > ... return list(sequence) > ... > >>> C.listcomp > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > >>> C.from_works > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > >>> C.fails() > Traceback (most recent call last): > File "", line 1, in > File "", line 8, in fails > NameError: name 'sequence' is not defined > ========== > > I think I did have an implementation that didn't do the "outermost > iterator is evaluated in the outer scope" dance early on (back when I was > still working on the initial version of the iteration variable hiding, > before it was merged to the Py3k branch), but quickly changed it because > relying solely on closures broke too much code (in addition to the class > namespace case, you can create a situation with similar constraints by > passing a separate locals namespace to exec). > > Cheers, > Nick. > But is the example with the class appropriate in this context, it seems that it is well understood why this example will fail, and `C.fails()` should at least be a `@classmethod`:) From the docs: > Class definition blocks and arguments to exec() > and eval() > are special in > the context of name resolution. A class definition is an executable > statement that may use and define names. These references follow the normal > rules for name resolution with an exception that unbound local variables > are looked up in the global namespace. The namespace of the class > definition becomes the attribute dictionary of the class. The scope of > names defined in a class block is limited to the class block; it does not > extend to the code blocks of methods ? this includes comprehensions and > generator expressions since they are implemented using a function scope. And if we cross out class, exec and eval cases - they don't work now, they will not be affected in future. Are not these examples equivalent? With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 24 12:02:47 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 03:02:47 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> Message-ID: <20180324160247.GX16661@ando.pearwood.info> On Sat, Mar 24, 2018 at 08:49:08PM +1100, Chris Angelico wrote: > > [(spam, spam+1) for x in values for spam in (func(x),)] > > > > [(spam, spam+1) for spam in (func(x) for x in values)] > > > > They are the equivalent to "just add another assignment statement" for > > comprehensions. > > They might be mechanically equivalent. They are not syntactically > equivalent. This PEP is not about "hey let's do something in Python > that's utterly impossible to do". It's "here's a much tidier way to > spell something that currently has to be ugly". For the record, I don't think either of those are ugly. The first is a neat trick, but the second in particular is a natural, elegant and beautiful way of doing it in a functional style. And the beauty of it is, if it ever becomes too big and unwieldy for a single expression, it is easy to *literally* "just add another assignment statement": eggs = (long_and_complex_expression_with(x) for x in values) [(spam, spam+1) for spam in eggs)] So I stand by my claim that even for comprehensions, "just add another assignment statement" is always an alternative. > > while ("spam" as x): > > assert x == "spam" > > while ("eggs" as x): > > assert x == "eggs" > > break > > assert x == "eggs" > > That means that sometimes, ``while ("eggs" as x):`` creates a new > variable, and sometimes it doesn't. Why should that be? I'm not following you. If we talk implementation for a moment, my proposal is that x is just a regular local variable. So the CPython compiler sees (... as x) in the code and makes a slot for it in the function. (Other implementations may do differently.) Whether or not that local slot gets filled with a value depends on whether or not the specific (... as x) actually gets executed or not. That's no different from any other binding operation. If x is defined as global, then (... as x) will bind to the global, not the local, but otherwise will behave the same. [...] > Function-local names give the same confidence. It doesn't matter what > names you use inside a function (modulo 'global' or 'nonlocal' > declarations) - they quietly shadow anything from the outside. Yes, I get functions, and I think function-scope is a sweet spot between too few scopes and too many. Remember the bad old days of BASIC when all variables were application-global? Even if you used GOSUB as a second-rate kind of function, all the variables were still global. On the other hand, introducing sub-function scopes is, I strongly believe, too many. [...] > > I think the rule should be either: > > > > - statement-locals actually *are* locals and so behave like locals; > > > > - statement-locals introduce a new scope, but still behave like > > locals with respect to closures. > > > > No need to introduce two separate modes of behaviour. (Or if there is > > such a need, then the PEP should explain it.) > > That would basically mean locking in some form of semantics. > For your first example, you're locking in the rule that "(g(i) as x)" > is exactly the same as "x = g(i)", and you HAVE to then allow that > this will potentially assign to global or nonlocal names as well > (subject to the usual rules). In other words, you have > assignment-as-expression without any form of subscoping. This is a > plausible stance and may soon be becoming a separate PEP. Well, if we really wanted to, we could ban (expression as name) where name was declared global, but why bother? > But for your second, you're locking in the same oddities that a 'with' > block has: that a variable is being "created" and "destroyed", yet it > sticks around for the rest of the function, just in case. Where is it documented that with blocks destroy variables? They don't. `with expression as name` is a name-binding operation no different from `name = expression` and the others. With the sole special case of except blocks auto-magically deleting the exception name, the only way to unbind a name is to call `del`. What you're describing is not an oddity, but the standard way variables work in Python, and damn useful too. I have code that requires that the `with` variable is not unbound at the end of the block. > It's a source of some confusion to people that the name used in a > 'with' statement is actually still valid afterwards. The difference between "import foo" and "from foo import bar" is source of some confusion to some people. I should know, because I went through that period myself. Just because "some people" make unjustified assumptions about the semantics of a language feature doesn't necessarily mean the language feature is wrong or harmful. > Or does it only stick > around if there is a function to close over it? No, there's no need for a closure: py> with open("/tmp/foo", "w") as f: ... pass ... py> f.closed True py> f.name '/tmp/foo' > Honestly, I really want to toss this one into the "well don't do that" > basket, and let the semantics be dictated by simplicity and > cleanliness even if it means that a closure doesn't see that variable. If `(expression as name)` just bounds to a local, this is a non-problem. [...] > > Indeed. In case it isn't obvious, you should define the acronym the > > first time you use it in the PEP. > > Once again, I assumed too much of people. Expected them to actually > read the stuff they're discussing. And once again, the universe > reminds me that people aren't like that. Ah well. Will fix that next > round of edits. Sorry I don't have time to read that paragraph, so I'll just assume you are thanking me for pointing out your terrible error and offering profuse apologies. *wink* >> >> * An SLNB cannot be the target of any form of assignment, including augmented. > >> >> Attempting to do so will remove the SLNB and assign to the fully-scoped name. > >> > > >> > What's the justification for this limitation? > >> > >> Not having that limitation creates worse problems, like that having > >> "(1 as a)" somewhere can suddenly make an assignment fail. This is > >> particularly notable with loop headers rather than simple statements. > > > > How and why would it fail? > > a = (1 as a) > > With current semantics, this is equivalent to "a = 1". If assignment > went into the SLNB, it would be equivalent to "pass". Which do you > expect it to do? Sorry, I don't follow this. If assignment goes into the statement-local, then it would be equivalent to: statement-local a = 1 not pass. Anyway, this confusion disappears if a is just a local. Then it is just: local a = 1 # the right hand side (1 as a) local a = 1 # the left hand side a = ... which presumably some interpreters could optimize down to a single assignment. If they can be bothered. > > MUST NOT implies that if there is *any* measurable penalty, even a > > nano-second, the feature must be rejected. I think that's excessive. > > Surely a nanosecond cost for the normal case is a reasonable tradeoff > > if it buys us better expressiveness? > > Steve, you know how to time a piece of code. You debate these kinds of > points on python-list frequently. Are you seriously trying to tell me > that you could measure a single nanosecond in regular compiling and > running of Python code? On my computer? Not a hope. But some current generation computers have sub-nanosecond CPU clock rates, and 3.7 is due to have new timers with nanosecond resolution: https://www.python.org/dev/peps/pep-0564/ I have no difficulty in believing that soon, if not right now, people will have sufficiently fast computers that yes, a nanosecond difference could be reliably measured with sufficient care. [...] > > Of course we don't want to necessarily impose unreasonable performance > > and maintence costs on any implementation. But surely performance > > cost is a quality of implementation issue. It ought to be a matter of > > trade-offs: is the benefit sufficient to make up for the cost? > > I don't see where this comes in. Let's say that Jython can't implement > this feature without a 10% slowdown in run-time performance even if > these subscopes aren't used. Unlikely, but for the sake of the argument, okay. > What are you saying the PEP should say? > That it's okay for this feature to hurt performance by 10%? Then it > should be rightly rejected. Or that Jython is allowed to ignore this > feature? Or what? That's really for Guido to decide whether the benefit is worth the (hypothetical) cost. But why single out this feature from every other syntactic feature added to Python over its history? We have never before, as far as I can tell, demanded that a feature prove that every Python implmentation be able to support the feature with ZERO performance cost before accepting the PEP. Normally, we introduce a new feature, and expect that like any new code, the first version may not be the most efficient, but subsequent versions will be faster. The first few versions of Python 3 were significant slower than Python 2. Normally we make performance a trade-off: it's okay to make certain things a bit slower if there are sufficient other benefits. I still don't understand why you think that sort of tradeoff doesn't apply here. Effectively you seem to be saying that the value of this proposed feature is so infinitesimally small that we shouldn't accept *any* runtime cost, no matter how small, to gain this feature. -- Steve From rob.cliffe at btinternet.com Sat Mar 24 12:27:11 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 24 Mar 2018 16:27:11 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180324144432.GW16661@ando.pearwood.info> References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> Message-ID: <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> On 24/03/2018 14:44, Steven D'Aprano wrote: > On Sat, Mar 24, 2018 at 07:12:49PM +1000, Nick Coghlan wrote: > [...] > >> At a user experience level, the aim of the scoping limitation is >> essentially to help improve "code snippet portability". >> >> Consider the following piece of code: >> >> squares = [x**2 for x in iterable] >> >> In Python 2.x, you not only have to check whether or not you're already >> using "squares" for something, you also need to check whether or not you're >> using "x", since the iteration variable leaks. > [...] > >> For PEP 572, the most directly comparable example is code like this: >> >> # Any previous binding of "m" is lost completely on the next line >> m = re.match(...) >> if m: >> print(m.groups(0)) >> >> In order to re-use that snippet, you need to double-check the surrounding >> code and make sure that you're not overwriting an "m" variable already used >> somewhere else in the current scope. > Yes. So what? I'm going to be doing that regardless of whether the > interpreter places this use of m in its own scope or not. The scope as > seen by the interpreter is not important. Good for you.? But the proposed scoping rules are an extra safeguard for programmers who are less conscientious than you, or for anyone (including you) who is short of time, or misses something.? An extra level of protection against introducing a bug is IMO a Good Thing. > If all we cared about was > avoiding name collisions, we could solve that by using 128-bit secret > keys as variables: > > var_81c199e61e9f90fd023508aee3265ad9 Good luck with that. :-) > > We don't need multiple scopes to avoid name collisions, we just need to > make sure they're all unique :-) You could use the same argument to justify "We don't need separate local and global scopes".? But we have them, and it makes it easier and safer to cut-and-paste functions.? I assume you don't consider that a Bad Thing. > > But of course readability counts, and we write code to be read by > people, not for the convenience of the interpreter. > > For that reason, whenever I paste a code snippet, I'm going to check the > name and make a conscious decision whether to keep it or change it, and > doing that means I have to check whether "m" is already in use > regardless of whether or not the interpreter will keep the two (or > more!) "m" variables. So this supposed benefit is really no benefit at > all. I still am going to check "m" to see if it clashes. Same argument, same reply.? Good for you - but there's nothing wrong with an extra safety net. And you make essentially the same point a few more times, I won't repeat myself further. > > To the extent that this proposal to add sub-function scoping encourages > people to do copy-paste coding without even renaming variables to > something appropriate for the function they're pasted into, I think this > will strongly hurts readability in the long run. I think it will aid readability, precisely for the reason Nick gives: you need to make fewer checks whether variables are or are not used elsewhere. > > >> With PEP 572, you don't even need to look, since visibility of the "m" in >> the following snippet is automatically limited to the statement itself: >> >> if (re.match(...) as m): >> print(m.groups(0)) >> # Any previous binding of "m" is visible again here, and hence a common >> source of bugs is avoided :) > Is this really a "common source of bugs"? > > Do you really mean to suggest that we should be able to copy and paste a > code snippet into the middle of a function without checking how it > integrates with the surrounding code? Because that's what it seems that > you are saying. And not only that we should be able to do so, but that > it is important enough that we should add a feature to encourage it? > > If people make a habit of pasting snippets of code into their functions > without giving any thought to how it fits in with the rest of the > function, then any resulting bugs are caused by carelessness and > slap-dash technique, not the scoping rules of the language. > > The last thing I want to read is a function where the same name is used > for two or three or a dozen different things, because the author > happened to copy code snippets from elsewhere and didn't bother renaming > things to be more appropriate. Nevermind whether the interpreter can > keep track of which is which, I'm worried about *my* ability to keep > track of which is which. > > I might be cynical about the professionalism and skills of the average > programmer, but even I find it hard to believe that most people would > actually do that. But since we're (surely?) going to be taking time to > integrate the snippet with the rest of the function, the benefit of not > having to check for duplicate variable names evaporates. > > We (hopefully!) will be checking for duplicates regardless of whether > they are scoped to a single statement or not, because we don't want to > read and maintain a function with the same name "x" representing a dozen > different things at different times. > > I'm not opposed to re-using variable names for two different purposes > within a single function. But when I do it, I do it because I made a > conscious decision that: > > (1) the name is appropriate for both purposes; and > > (2) re-using the name does not lead to confusion or make the function > hard to read. > > I don't re-use names because I've copied some snippet and can't be > bothered to change the names. And I don't think we should be adding a > feature to enable and justify that sort of poor practice. > > Comprehensions have their own scope, and that's at least harmless, if > not beneficial, because they are self-contained single expressions. But > this would add separate scopes to blocks: > > def function(): > x = 1 > if (spam as x): > ... > while (ham as x): > ... > # much later, deep in the function > # possibly after some or all of those blocks have ended > ... > process(x) # which x is this? > > This would be three different variables all with the same name "x". To > track the current value of x I have to track each of the x variables > and which is currently in scope. > > I don't think we need sub-function scoping. I think it adds more > complexity that outweighs whatever benefit it gives. > > > From ericfahlgren at gmail.com Sat Mar 24 12:34:57 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Sat, 24 Mar 2018 09:34:57 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87po3tlhyi.fsf@grothesque.org> Message-ID: On Sat, Mar 24, 2018 at 7:14 AM, Nick Coghlan wrote: > ?? > >>> class C: > ... sequence = range(10) > ... listcomp = [x for x in sequence] > >>> class C: ... y = 1 ... sequence = range(10) ... listcomp = [x+y for x in sequence] ? ... ?Traceback (most recent call last): File "", line 1, in File "", line 4, in C File "", line 4, in NameError: name 'y' is not defined ? ?Ok, so how does 'y' fit into these scoping rules?? -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Sat Mar 24 12:36:55 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sat, 24 Mar 2018 16:36:55 +0000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180324160247.GX16661@ando.pearwood.info> References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324160247.GX16661@ando.pearwood.info> Message-ID: <5d967eae-b44b-ca63-523b-fd4057643acc@btinternet.com> On 24/03/2018 16:02, Steven D'Aprano wrote: > > Yes, I get functions, and I think function-scope is a sweet spot between > too few scopes and too many. Remember the bad old days of BASIC when all > variables were application-global? Even if you used GOSUB as a > second-rate kind of function, all the variables were still global. > > On the other hand, introducing sub-function scopes is, I strongly > believe, too many. > > We are all entitled to our beliefs. But the decision was made to stop a for-variable from leaking from a list comprehension - you may not agree with that decision, but it was presumably a reasonable one. Using SLNBs that don't leak into the surrounding local scope is ISTM a similar decision, and one that, if made, would be made for similar reasons. Rob Cliffe From ethan at stoneleaf.us Sat Mar 24 14:49:26 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 24 Mar 2018 11:49:26 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180324144432.GW16661@ando.pearwood.info> References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> Message-ID: <5AB69DB6.2080409@stoneleaf.us> On 03/24/2018 07:44 AM, Steven D'Aprano wrote: > I don't think we need sub-function scoping. I think it adds more > complexity that outweighs whatever benefit it gives. +1 -- ~Ethan~ From ethan at stoneleaf.us Sat Mar 24 15:01:21 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 24 Mar 2018 12:01:21 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> Message-ID: <5AB6A081.5010503@stoneleaf.us> On 03/24/2018 09:27 AM, Rob Cliffe via Python-ideas wrote: > On 24/03/2018 14:44, Steven D'Aprano wrote: >> On Sat, Mar 24, 2018 at 07:12:49PM +1000, Nick Coghlan wrote: >>> For PEP 572, the most directly comparable example is code like this: >>> >>> # Any previous binding of "m" is lost completely on the next line >>> m = re.match(...) >>> if m: >>> print(m.groups(0)) >>> >>> In order to re-use that snippet, you need to double-check the surrounding >>> code and make sure that you're not overwriting an "m" variable already used >>> somewhere else in the current scope. >> >> Yes. So what? I'm going to be doing that regardless of whether the >> interpreter places this use of m in its own scope or not. The scope as >> seen by the interpreter is not important. > > Good for you. But the proposed scoping rules are an extra safeguard for programmers who are less conscientious than > you, or for anyone (including you) who is short of time, or misses something. An extra level of protection against > introducing a bug is IMO a Good Thing. But it's not a free thing. Our cars have seat belts, not six-point restraints, and either way the best practice is to be aware of one's surroundings, not rely on the safeguards to protect us against carelessness. >> To the extent that this proposal to add sub-function scoping encourages >> people to do copy-paste coding without even renaming variables to >> something appropriate for the function they're pasted into, I think this >> will strongly hurts readability in the long run. > > I think it will aid readability, precisely for the reason Nick gives: you need to make fewer checks whether variables > are or are not used elsewhere. Extra levels of intermingled scope are extra complication (for humans, too!); extra complication does not (usually) help readability -- I agree with D'Aprano that this is not a helpful complication. -- ~Ethan~ From J.Demeyer at UGent.be Sat Mar 24 17:38:11 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 24 Mar 2018 22:38:11 +0100 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: <8fad0e0f0a7d46d890345ea99b03c97b@xmail102.UGent.be> References: <5AB41364.8010500@UGent.be> <8fad0e0f0a7d46d890345ea99b03c97b@xmail102.UGent.be> Message-ID: <5AB6C543.303@UGent.be> Dear Nick Coghlan, First of all, thanks for your insightful comments! On 2018-03-24 09:09, Nick Coghlan wrote: > As Antoine notes, unifying user-defined functions and builtin functions > would be fraught with backwards compatibility problems, so you probably > don't want to go down that path when your goal is "Let third parties > define types that are recognised by the core interpreter as being > user-defined functions". First of all, my post was mainly meant to get an answer to "is this idea PEP-able?". I have never written a PEP and PEP 1 recommends this step. So it's not clear whether you want to say "that potential PEP is a bad idea" or just "go ahead with your PEP but be aware of backwards compatibility issues". It would be good to know what your backwards compatibility worries are. If I have a clearer idea of what could break, it would be easier to see if it's possible to not break that. Anyway, many (not all though!) backwards compatibility issues could be fixed by keeping the current built-in functions as a distinct type (say, old_style_builtin although I wouldn't use that name). My vague plan in the PEP is to introduce a base class "basefunction" and have old_style_builtin and Python functions be subclasses of that. Cython functions would then be another subclass. That way, we could implement "basefunction" properly and then implement old_style_builtin to support things like __get__ not binding as a method. > If it was just about introspection, then we could define a new protocol > method or attribute, update the inspect module to respect it, and call > it done. True. In fact, I got a long way by just defining inspect.isfunction as def isfunction(obj): return hasattr(type(obj), "__code__") > However, it seems to me that the most plausible path towards granting > Cython access to the optimised fast paths for native functions would be > setting Py_TPFLAGS_BASETYPE on types.FunctionType Why FunctionType and not BuiltinFunctionType? That's really the problem here: I want a function which is as fast as a built-in function but which behaves from a user's point of view as a Python function. > We can't introduce a new protocol > into those paths without slowing them down I don't really follow you here. Are you saying that it's impossible to allow custom function classes that are as fast as the current builtin functions? Instead of adding a flag for a subclass, we could add a flag for "can be called as if it is a built-in function". Jeroen. From ncoghlan at gmail.com Sat Mar 24 22:08:08 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 12:08:08 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87po3tlhyi.fsf@grothesque.org> Message-ID: On 25 March 2018 at 02:34, Eric Fahlgren wrote: > On Sat, Mar 24, 2018 at 7:14 AM, Nick Coghlan wrote: > >> ?? >> >>> class C: >> ... sequence = range(10) >> ... listcomp = [x for x in sequence] >> > > >>> class C: > ... y = 1 > ... sequence = range(10) > ... listcomp = [x+y for x in sequence] ? > ... > ?Traceback (most recent call last): > File "", line 1, in > File "", line 4, in C > File "", line 4, in > NameError: name 'y' is not defined? > > ?Ok, so how does 'y' fit into these scoping rules?? > Everything except the outermost iterator is evaluated in the implicitly nested scope, so comprehensions at class scope have the restriction that only the outermost iterator can access names defined in the class scope. It turned out that was enough to preserve compatibility with most of the comprehensions that folks actually use at class scope. For those rare cares where it isn't, the typical resolution is to either define a helper function, or else switch to a regular for loop. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sat Mar 24 23:13:59 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 13:13:59 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87po3tlhyi.fsf@grothesque.org> Message-ID: On 25 March 2018 at 01:03, Kirill Balunov wrote: > > > 2018-03-24 17:14 GMT+03:00 Nick Coghlan : >> >> >> They can be distinguished, just not at module or function scope. To give >> a concrete example: >> >> ========== >> >>> class C: >> ... sequence = range(10) >> ... listcomp = [x for x in sequence] >> ... def works(data): >> ... return list(data) >> ... from_works = works(sequence) >> ... def fails(): >> ... return list(sequence) >> ... >> >>> C.listcomp >> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >> >>> C.from_works >> [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] >> >>> C.fails() >> Traceback (most recent call last): >> File "", line 1, in >> File "", line 8, in fails >> NameError: name 'sequence' is not defined >> ========== >> >> I think I did have an implementation that didn't do the "outermost >> iterator is evaluated in the outer scope" dance early on (back when I was >> still working on the initial version of the iteration variable hiding, >> before it was merged to the Py3k branch), but quickly changed it because >> relying solely on closures broke too much code (in addition to the class >> namespace case, you can create a situation with similar constraints by >> passing a separate locals namespace to exec). >> >> Cheers, >> Nick. >> > > But is the example with the class appropriate in this context, it seems > that it is well understood why this example will fail, and `C.fails()` > should at least be a `@classmethod`:) From the docs: > >> Class definition blocks and arguments to exec() >> and eval() >> are special in >> the context of name resolution. A class definition is an executable >> statement that may use and define names. These references follow the normal >> rules for name resolution with an exception that unbound local variables >> are looked up in the global namespace. The namespace of the class >> definition becomes the attribute dictionary of the class. The scope of >> names defined in a class block is limited to the class block; it does not >> extend to the code blocks of methods ? this includes comprehensions and >> generator expressions since they are implemented using a function scope. > > And if we cross out class, exec and eval cases - they don't work now, they > will not be affected in future. Are not these examples equivalent? > I don't follow what you're trying to say here. Comprehensions *do* work with class bodies, two-namespace exec, and two-namespace eval, and one of the consequences of the *way* they work that proposals for expression level name binding will all have to deal with is that the outermost iterable expression is evaluated in the containing scope rather than the implicitly nested one. Statement local name bindings would restrict any resulting name leakage from the outermost iterable to the affected statement, while regular name bindings would affect the entirety of the containing scope. The latter behavour is of the larger downsides of having expression level name bindings work the same way assignment statements do, since it recreates a variant of the original iterator variable name leakage that the introduction of the implicitly nested scope eliminated. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Mar 24 23:48:18 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 24 Mar 2018 20:48:18 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> Message-ID: <5AB71C02.7000702@stoneleaf.us> On 03/24/2018 01:35 AM, Nick Coghlan wrote: > In comprehensions and generator expressions, we'd need to explain why inline assignments in the outermost iterator > expression leak but those in filter expressions, inner iterator expressions, and result expressions don't. I don't understand -- could you give an example? -- ~Ethan~ From rosuav at gmail.com Sat Mar 24 23:51:46 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 14:51:46 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5AB71C02.7000702@stoneleaf.us> References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> Message-ID: On Sun, Mar 25, 2018 at 2:48 PM, Ethan Furman wrote: > On 03/24/2018 01:35 AM, Nick Coghlan wrote: > >> In comprehensions and generator expressions, we'd need to explain why >> inline assignments in the outermost iterator >> expression leak but those in filter expressions, inner iterator >> expressions, and result expressions don't. > > > I don't understand -- could you give an example? > Let's suppose we have assignment expressions. I'm going to use "(expr as name)" syntax for this example. a = [(1 as b) for c in (d as e) if (2 as f)] Which of these six names is local to the comprehension and which can leak? Due to the requirements of class scope, 'd' must be looked up in the outer scope. That means that its corresponding 'as e' must also land in the outer scope. 'a', of course, is in the outer scope. The other four are local to the inner function that implements the comprehension. ChrisA From ncoghlan at gmail.com Sat Mar 24 23:52:17 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 13:52:17 +1000 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: <5AB6C543.303@UGent.be> References: <5AB41364.8010500@UGent.be> <8fad0e0f0a7d46d890345ea99b03c97b@xmail102.UGent.be> <5AB6C543.303@UGent.be> Message-ID: On 25 March 2018 at 07:38, Jeroen Demeyer wrote: > Dear Nick Coghlan, > > First of all, thanks for your insightful comments! > > On 2018-03-24 09:09, Nick Coghlan wrote: > >> As Antoine notes, unifying user-defined functions and builtin functions >> would be fraught with backwards compatibility problems, so you probably >> don't want to go down that path when your goal is "Let third parties >> define types that are recognised by the core interpreter as being >> user-defined functions". >> > > First of all, my post was mainly meant to get an answer to "is this idea > PEP-able?". I have never written a PEP and PEP 1 recommends this step. So > it's not clear whether you want to say "that potential PEP is a bad idea" > or just "go ahead with your PEP but be aware of backwards compatibility > issues". > > It would be good to know what your backwards compatibility worries are. If > I have a clearer idea of what could break, it would be easier to see if > it's possible to not break that. > The biggest problem would have been the potential change to the descriptor behaviour - making CPython's C functions behave like instance methods instead of static methods would create a significant compatibility risk for no clear benefit. However, from the rest of your email, it sounds like you're not actually proposing to change that and I'd just misunderstood what you were suggesting, so I now think it's a promising idea that should help not only Cython, but also other binding generation projects like cffi, SWIG, Boost, etc :) Cheers, Nick. P.S. You're also right that I'd missed the fact that the fast paths that you want to follow at runtime are the PyCFunction ones, so you really do need to be able to define a hybrid type that executes like a builtin C function, but supports introspection like a native Python function. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sat Mar 24 23:59:47 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 24 Mar 2018 20:59:47 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> Message-ID: <5AB71EB3.5030308@stoneleaf.us> On 03/24/2018 08:51 PM, Chris Angelico wrote: > On Sun, Mar 25, 2018 at 2:48 PM, Ethan Furman wrote: >> On 03/24/2018 01:35 AM, Nick Coghlan wrote: >> >>> In comprehensions and generator expressions, we'd need to explain why >>> inline assignments in the outermost iterator >>> expression leak but those in filter expressions, inner iterator >>> expressions, and result expressions don't. >> >> I don't understand -- could you give an example? > > Let's suppose we have assignment expressions. I'm going to use "(expr > as name)" syntax for this example. > > a = [(1 as b) for c in (d as e) if (2 as f)] > > Which of these six names is local to the comprehension and which can > leak? Due to the requirements of class scope, 'd' must be looked up in > the outer scope. That means that its corresponding 'as e' must also > land in the outer scope. [...] Why? d = 9 def func(): e = d d is looked up outside of func, but e is still local to func. -- ~Ethan~ From rosuav at gmail.com Sun Mar 25 00:00:14 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 15:00:14 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5AB71EB3.5030308@stoneleaf.us> References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> <5AB71EB3.5030308@stoneleaf.us> Message-ID: On Sun, Mar 25, 2018 at 2:59 PM, Ethan Furman wrote: > On 03/24/2018 08:51 PM, Chris Angelico wrote: >> >> On Sun, Mar 25, 2018 at 2:48 PM, Ethan Furman wrote: >>> >>> On 03/24/2018 01:35 AM, Nick Coghlan wrote: >>> >>>> In comprehensions and generator expressions, we'd need to explain why >>>> inline assignments in the outermost iterator >>>> expression leak but those in filter expressions, inner iterator >>>> expressions, and result expressions don't. >>> >>> >>> I don't understand -- could you give an example? >> >> >> Let's suppose we have assignment expressions. I'm going to use "(expr >> as name)" syntax for this example. >> >> a = [(1 as b) for c in (d as e) if (2 as f)] >> >> Which of these six names is local to the comprehension and which can >> leak? Due to the requirements of class scope, 'd' must be looked up in >> the outer scope. That means that its corresponding 'as e' must also >> land in the outer scope. [...] > > > Why? > > d = 9 > def func(): > e = d > > d is looked up outside of func, but e is still local to func. No, it's looked up inside func, and the value is found outside of func. Consider: d = 9 def func(): e = d d = 1 What's happening with a comprehension is more like: d = 9 def func(_iter): ... body of comprehension func((d as e)) which is looking it up _outside_ the function, then passing it as a parameter. ChrisA From ethan at stoneleaf.us Sun Mar 25 00:08:09 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sat, 24 Mar 2018 21:08:09 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> <5AB71EB3.5030308@stoneleaf.us> Message-ID: <5AB720A9.5050205@stoneleaf.us> On 03/24/2018 09:00 PM, Chris Angelico wrote: > On Sun, Mar 25, 2018 at 2:59 PM, Ethan Furman wrote: >> On 03/24/2018 08:51 PM, Chris Angelico wrote: >>> Let's suppose we have assignment expressions. I'm going to use "(expr >>> as name)" syntax for this example. >>> >>> a = [(1 as b) for c in (d as e) if (2 as f)] >>> >>> Which of these six names is local to the comprehension and which can >>> leak? Due to the requirements of class scope, 'd' must be looked up in >>> the outer scope. That means that its corresponding 'as e' must also >>> land in the outer scope. [...] >> >> >> Why? >> >> d = 9 >> def func(): >> e = d >> >> d is looked up outside of func, but e is still local to func. > > No, it's looked up inside func, and the value is found outside of > func. Consider: > > d = 9 > def func(): > e = d > d = 1 That is invalid Python. > What's happening with a comprehension is more like: > > d = 9 > def func(_iter): > ... body of comprehension > func((d as e)) > > which is looking it up _outside_ the function, then passing it as a parameter. Looks like a buggy implementation detail. Any assignments that happen inside a listcomp should be effective only inside the listcomp. -- ~Ethan~ From ncoghlan at gmail.com Sun Mar 25 00:13:13 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 14:13:13 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> Message-ID: On 25 March 2018 at 13:51, Chris Angelico wrote: > On Sun, Mar 25, 2018 at 2:48 PM, Ethan Furman wrote: > > On 03/24/2018 01:35 AM, Nick Coghlan wrote: > > > >> In comprehensions and generator expressions, we'd need to explain why > >> inline assignments in the outermost iterator > >> expression leak but those in filter expressions, inner iterator > >> expressions, and result expressions don't. > > > > > > I don't understand -- could you give an example? > > > > Let's suppose we have assignment expressions. I'm going to use "(expr > as name)" syntax for this example. > > a = [(1 as b) for c in (d as e) if (2 as f)] > > Which of these six names is local to the comprehension and which can > leak? Due to the requirements of class scope, 'd' must be looked up in > the outer scope. That means that its corresponding 'as e' must also > land in the outer scope. 'a', of course, is in the outer scope. The > other four are local to the inner function that implements the > comprehension. > Adding qualifiers to the names to show which names would be resolved where: outer_a = [(1 as inner_b) for inner_c in (outer_d as outer_e) if (2 as inner_f)] print(outer_a, outer_d, outer_e) # Works print (inner_b, inner_c, inner_f) # Name error In current Python,with the outer scope's name bindings being read-only inside a comprehension, this difference in scope resolution really only matters at class scope, and when using the two-namespace form of exec and eval (since method semantics result in that impacting which names the comprehension can see). At function scope, referenced names from the outer scope get resolved as closure references, while at module scope they get resolved as global references, so you have to look at the generated bytecode to see the differences. That all changes once we start allowing name binding (in any form) at the level of expressions rather than statements: regardless of the nature of the outer scope, the exact structure of the implicit comprehension scope becomes relevant whenever you bind names inside the comprehension. Now, the notable *distinction* here relative to the old iteration variable leakage is that doing a name binding in the outermost iterable expression can always be refactored to use an ordinary assignment statement instead: outer_e = outer_d outer_a = [(1 as inner_b) for inner_c in outer_d if (2 as inner_f)] print(outer_a, outer_d, outer_e) # Works print (inner_b, inner_c, inner_f) # Name error This formulation also makes it clear that "outer_e" will only resolve inside the comprehension if "outer_d" already resolves. This means that if we did go down the expression-level-assignments-are-just-regular-name-binding-operations path, then using the embedded assignment form in such cases could be discouraged as misleadingly ambiguous by style guides and linters, without being outright prohibited by the compiler itself. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Mar 25 00:14:00 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 15:14:00 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5AB720A9.5050205@stoneleaf.us> References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> <5AB71EB3.5030308@stoneleaf.us> <5AB720A9.5050205@stoneleaf.us> Message-ID: On Sun, Mar 25, 2018 at 3:08 PM, Ethan Furman wrote: > On 03/24/2018 09:00 PM, Chris Angelico wrote: >> >> On Sun, Mar 25, 2018 at 2:59 PM, Ethan Furman wrote: >>> >>> On 03/24/2018 08:51 PM, Chris Angelico wrote: > > >>>> Let's suppose we have assignment expressions. I'm going to use "(expr >>>> as name)" syntax for this example. >>>> >>>> a = [(1 as b) for c in (d as e) if (2 as f)] >>>> >>>> Which of these six names is local to the comprehension and which can >>>> leak? Due to the requirements of class scope, 'd' must be looked up in >>>> the outer scope. That means that its corresponding 'as e' must also >>>> land in the outer scope. [...] >>> >>> >>> >>> Why? >>> >>> d = 9 >>> def func(): >>> e = d >>> >>> d is looked up outside of func, but e is still local to func. >> >> >> No, it's looked up inside func, and the value is found outside of >> func. Consider: >> >> d = 9 >> def func(): >> e = d >> d = 1 > > > That is invalid Python. No, it's perfectly valid Python... it just happens that it'll raise UnboundLocalError when it tries to look up 'd' :) >> What's happening with a comprehension is more like: >> >> d = 9 >> def func(_iter): >> ... body of comprehension >> func((d as e)) >> >> which is looking it up _outside_ the function, then passing it as a >> parameter. > > > Looks like a buggy implementation detail. Any assignments that happen > inside a listcomp should be effective only inside the listcomp. Leaky abstraction, perhaps. I don't think it's buggy as such. ChrisA From ncoghlan at gmail.com Sun Mar 25 00:24:11 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 14:24:11 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5AB720A9.5050205@stoneleaf.us> References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> <5AB71EB3.5030308@stoneleaf.us> <5AB720A9.5050205@stoneleaf.us> Message-ID: On 25 March 2018 at 14:08, Ethan Furman wrote: > Looks like a buggy implementation detail. Any assignments that happen > inside a listcomp should be effective only inside the listcomp. > No, the fact that the expression defining the outermost iterable gets evaluated in the outer scope is behaviour that's explicitly tested for in the regression test suite. The language reference spells out that this is intentional for generator expressions, where it has the added benefit of reporting errors in the outermost iterable expression at the point where the genexp is defined, rather than at the point where it gets iterated over: https://docs.python.org/3/reference/expressions.html#generator-expressions Independently of the pragmatic "getting them to work sensibly at class scope" motivation, comprehensions inherit those semantics by way of the intended semantic equivalence between "[x for x in sequence]" and "list(x for x in sequence)". Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Mar 25 00:27:10 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Mar 2018 21:27:10 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87po3tlhyi.fsf@grothesque.org> Message-ID: On Sat, Mar 24, 2018 at 7:08 PM, Nick Coghlan wrote: > On 25 March 2018 at 02:34, Eric Fahlgren wrote: > >> On Sat, Mar 24, 2018 at 7:14 AM, Nick Coghlan wrote: >> >>> ?? >>> >>> class C: >>> ... sequence = range(10) >>> ... listcomp = [x for x in sequence] >>> >> >> >>> class C: >> ... y = 1 >> ... sequence = range(10) >> ... listcomp = [x+y for x in sequence] ? >> ... >> ?Traceback (most recent call last): >> File "", line 1, in >> File "", line 4, in C >> File "", line 4, in >> NameError: name 'y' is not defined? >> >> ?Ok, so how does 'y' fit into these scoping rules?? >> > > Everything except the outermost iterator is evaluated in the implicitly > nested scope, so comprehensions at class scope have the restriction that > only the outermost iterator can access names defined in the class scope. It > turned out that was enough to preserve compatibility with most of the > comprehensions that folks actually use at class scope. > > For those rare cares where it isn't, the typical resolution is to either > define a helper function, or else switch to a regular for loop. > Just in case anyone wonders, I don't think the special rules around class scope count are a wonderful feature. 28 years ago it was the best we could do, and by the time we realized its problems -- all rare edge cases for sure, but unintuitive and hard to debug when they strike -- we were tied by backward compatibility. And we still are. (At the time there were no nested scopes, there was just local, global, builtins; class scope was treated as a function scope and that was that.) (I have a lot more to say about PEP 572 but I'll do it in the "take three" thread.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Mar 25 01:34:59 2018 From: guido at python.org (Guido van Rossum) Date: Sat, 24 Mar 2018 22:34:59 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <5AB6A081.5010503@stoneleaf.us> References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: This is a super complex topic. There are at least three separate levels of critique possible, and all are important. First there is the clarity of the PEP. Steven D'Aprano has given you great detailed feedback here and you should take it to heart (even if you disagree with his opinion about the specifics). I'd also recommend treating some of the "rejected alternatives" more like "open issues" (which are to be resolved during the review and feedback cycle). And you probably need some new terminology -- the abbreviation SNLB is awkward (I keep having to look it up), and I think we need a short, crisp name for the new variable type. Then there is the issue of syntax. While `(f() as x)` is a cool idea (and we should try to recover who deserves credit for first proposing it), it's easy to overlook in the middle of an exception. It's arguably more confusing because the scoping rules you propose are so different from the existing three other uses of `as NAME` -- and it causes an ugly wart in the PEP because two of those other uses are syntactically so close that you propose to ban SNLBs there. When it comes to alternatives, I think we've brainwashed ourselves into believing that inline assignments using `=` are evil that it's hard to objectively explain why it's bad -- we're just repeating the mantra here. I wish we could do more quantitative research into how bad this actually is in languages that do have it. We should also keep an open mind about alternative solutions present in other languages. Here it would be nice if we had some qualitative research into what other languages actually do (both about syntax and about semantics, for sure). The third issue is that of semantics. I actually see two issues here. One is whether we need a new scope (and whether it should be as weird as proposed). Steven seems to think we don't. I'm not sure that the counter-argument that we're already down that path with comprehension scopes is strong enough. The other issue is that, if we decide we *do* need (or want) statement-local scopes, the PEP must specify the exact scope of a name bound at any point in a statement. E.g. is `d[x] = (f() as x)` valid? And what should we do if a name may or may not be bound, as in `if (f(1) as x) or (f(2) as y): g(y)` -- should that be a compile-time error (since we can easily tell that y isn't always defined when `g(y)` is called) or a runtime error (as we do for unbound "classic" locals)? And there are further details, e.g. are these really not allowed to be closures? And are they single-assignment? (Or can you do e.g. `(f(1) as x) + (f(2) as x)`?) I'm not sure if there are still places in Python where evaluation order is unspecified, but I suspect there are (at the very least the reference manual is incomplete in specifying the exact rules, e.g. I can't find words specifying the evaluation order in a slice). We'll need to fix all of those, otherwise the use of local name bindings in such cases would have unspecified semantics (or the evaluation order could suddenly shift when a local name binding was added). So, there are lots of interesting questions! I do think there are somewhat compelling use cases; more than comprehensions (which I think are already over-used) I find myself frequently wishing for a better way to write m = pat.match(text) if m: g = m.group(0) if check(g): # Some check that's not easily expressed as a regex print(g) It would be nice if I could write that as if (m = pat.match(text)) and check((g = m.group(0))): print(g) or if (pat.match(text) as m) and check((m.group(0) as g)): print(g) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Sun Mar 25 03:00:37 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 17:00:37 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On 25 March 2018 at 15:34, Guido van Rossum wrote: > > This is a super complex topic. There are at least three separate levels of critique possible, and all are important. > > First there is the clarity of the PEP. Steven D'Aprano has given you great detailed feedback here and you should take it to heart (even if you disagree with his opinion about the specifics). I'd also recommend treating some of the "rejected alternatives" more like "open issues" (which are to be resolved during the review and feedback cycle). And you probably need some new terminology -- the abbreviation SNLB is awkward (I keep having to look it up), and I think we need a short, crisp name for the new variable type. I've used "ephemeral name binding" before, but that's even longer than saying ess-ell-enn-bee (for Statement Local Name Binding), and also doesn't feel right for a proposal that allows the binding to persist for the entire suite in compound statements. Given the existing namespace stack of builtin<-global<-nonlocal<-local, one potential short name would be "sublocal", to indicate that these references are even more local than locals (they're *so* local, they don't even appear in locals()!). > Then there is the issue of syntax. While `(f() as x)` is a cool idea (and we should try to recover who deserves credit for first proposing it), I know I first suggested it years ago, but I don't recall if anyone else proposed it before me. > it's easy to overlook in the middle of an exception. That I agree with - the more examples I've seen using it, the less I've liked how visually similar "(a as b)" is to "(a and b)". > It's arguably more confusing because the scoping rules you propose are so different from the existing three other uses of `as NAME` -- and it causes an ugly wart in the PEP because two of those other uses are syntactically so close that you propose to ban SNLBs there. When it comes to alternatives, I think we've brainwashed ourselves into believing that inline assignments using `=` are evil that it's hard to objectively explain why it's bad -- we're just repeating the mantra here. I wish we could do more quantitative research into how bad this actually is in languages that do have it. We should also keep an open mind about alternative solutions present in other languages. Here it would be nice if we had some qualitative research into what other languages actually do (both about syntax and about semantics, for sure). Writing "name = expr" when you meant "name == expr" remains a common enough source of bugs in languages that allow it that I still wouldn't want to bring that particular opportunity for semantically significant typos over to Python. Using "name := expr" doesn't have that problem though (since accidentally adding ":" is a much harder typo to make than leaving out "="), and has the added bonus that we could readily restrict the LHS to single names. I also quite like the way it reads in conditional expressions: value = f() if (f := lookup_function(args)) is not None else default And if we do end up going with the approach of defining a separate sublocal namespace, the fact that "n := ..." binds a sublocal, while "n = ..." and "... as n" both bind regular locals would be clearer than having the target scope of "as" be context dependent. > The third issue is that of semantics. I actually see two issues here. One is whether we need a new scope (and whether it should be as weird as proposed). Steven seems to think we don't. I'm not sure that the counter-argument that we're already down that path with comprehension scopes is strong enough. > The other issue is that, if we decide we *do* need (or want) statement-local scopes, the PEP must specify the exact scope of a name bound at any point in a statement. E.g. is `d[x] = (f() as x)` valid? And what should we do if a name may or may not be bound, as in `if (f(1) as x) or (f(2) as y): g(y)` -- should that be a compile-time error (since we can easily tell that y isn't always defined when `g(y)` is called) or a runtime error (as we do for unbound "classic" locals)? And there are further details, e.g. are these really not allowed to be closures? And are they single-assignment? (Or can you do e.g. `(f(1) as x) + (f(2) as x)`?) I think this need to more explicitly specify evaluation order applies regardless of whether we define a sublocal scope or not: expression level name binding in any form makes evaluation order (and evaluation scope!) matter in ways that we can currently gloss over, since you need to be relying on functions with side effects in order to even observe the differences. If the expression level bindings are just ordinary locals, it does open up some potentially interesting order of evaluation testing techniques, though: expected_order = list(range(3)) actual_order = iter(expected_order) defaultdict(int)[(first := next(actual_order)):(second := next(actual_order)):(third := next(actual_order))] self.assertEqual([first, second, third], expected_order) With sublocals, you'd need to explicitly promote them to regular locals to get the same effect: expected_order = list(range(3)) actual_order = iter(expected_order) __, first, second, third = defaultdict(int)[(first := next(actual_order)):(second := next(actual_order)):(third := next(actual_order))], first, second, third self.assertEqual([first, second, third], expected_order) That said, it's debatable whether *either* of those is any clearer for that task than the status quo of just using list append operations: expected_order = list(range(3)) actual_order = [] defaultdict(int)[actual_order.append(0):actual_order.append(1):actual_order.append(2)] self.assertEqual(actual_order, expected_order) > I'm not sure if there are still places in Python where evaluation order is unspecified, but I suspect there are (at the very least the reference manual is incomplete in specifying the exact rules, e.g. I can't find words specifying the evaluation order in a slice). We'll need to fix all of those, otherwise the use of local name bindings in such cases would have unspecified semantics (or the evaluation order could suddenly shift when a local name binding was added). One that surprised me earlier today is that it looks like we never transferred the generator expression wording about the scope of evaluation for the outermost iterable over to the sections describing comprehension evaluation - we only point out that the result subexpression evaluation and the iteration variable binding happen in a nested scope. (Although now I'm wondering if there might already be a docs tracker issue for that, and I just forgot about it) Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From rosuav at gmail.com Sun Mar 25 03:18:49 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 18:18:49 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Sun, Mar 25, 2018 at 4:34 PM, Guido van Rossum wrote: > This is a super complex topic. There are at least three separate levels of > critique possible, and all are important. Thank you for your detailed post. I'll respond to some of it here, and some more generally below. > First there is the clarity of the PEP. Steven D'Aprano has given you great > detailed feedback here and you should take it to heart (even if you disagree > with his opinion about the specifics). I'd also recommend treating some of > the "rejected alternatives" more like "open issues" (which are to be > resolved during the review and feedback cycle). And you probably need some > new terminology -- the abbreviation SNLB is awkward (I keep having to look > it up), and I think we need a short, crisp name for the new variable type. Agreed that it needs a new name. I've been trying to avoid looking for something that's short-yet-inaccurate, and sticking to the accurate-but-unwieldy; perhaps Nick's "sublocal" will serve the purpose? > Then there is the issue of syntax. While `(f() as x)` is a cool idea (and we > should try to recover who deserves credit for first proposing it), it's easy > to overlook in the middle of an exception. It's arguably more confusing > because the scoping rules you propose are so different from the existing > three other uses of `as NAME` -- and it causes an ugly wart in the PEP > because two of those other uses are syntactically so close that you propose > to ban SNLBs there. When it comes to alternatives, I think we've brainwashed > ourselves into believing that inline assignments using `=` are evil that > it's hard to objectively explain why it's bad -- we're just repeating the > mantra here. I wish we could do more quantitative research into how bad this > actually is in languages that do have it. We should also keep an open mind > about alternative solutions present in other languages. Here it would be > nice if we had some qualitative research into what other languages actually > do (both about syntax and about semantics, for sure). Not qualitative, but anecdotal: I do sometimes have to remind my JavaScript students to check whether they've typed enough equals signs. And that's in a language where the normal comparison operator is ===. It's *still* not uncommon to see a comparison spelled =. > The third issue is that of semantics. I actually see two issues here. One is > whether we need a new scope (and whether it should be as weird as proposed). > Steven seems to think we don't. I'm not sure that the counter-argument that > we're already down that path with comprehension scopes is strong enough. The > other issue is that, if we decide we *do* need (or want) statement-local > scopes, the PEP must specify the exact scope of a name bound at any point in > a statement. E.g. is `d[x] = (f() as x)` valid? Yes, it is. The sublocal name (I'm going to give this term a try and see how it works; if not, we can revert to "bullymong", err I mean "SLNB") remains valid for all retrievals until the end of the statement, which includes the assignment. > And what should we do if a > name may or may not be bound, as in `if (f(1) as x) or (f(2) as y): g(y)` -- > should that be a compile-time error (since we can easily tell that y isn't > always defined when `g(y)` is called) or a runtime error (as we do for > unbound "classic" locals)? The way I've been thinking about it (and this is reflected in the reference implementation) is that 'y' becomes, in effect, a new variable that doesn't collide with any other 'y' in the same function or module or anything. For the duration of this statement, 'x' and 'y' are those special variables. So it's similar to writing this: def func(): x = f(1) if x: g(y) else: y = f(2) g(y) which will raise UnboundLocalError when x is true. The same behaviour happens here. > And there are further details, e.g. are these > really not allowed to be closures? And are they single-assignment? (Or can > you do e.g. `(f(1) as x) + (f(2) as x)`?) Technically, what happens is that the second one creates _another_ sublocal name, whose scope begins from the point of assignment and goes to the end of the statement. Since this expression must all be within one statement, both sublocals will expire simultaneously, so it's effectively the same as reassigning to the same name, except that the old object won't be dereferenced until the whole statement ends. (And since Python-the-language doesn't guarantee anything about dereferenced object destruction timings, this will just be a point of curiosity.) > So, there are lots of interesting questions! I do think there are somewhat > compelling use cases; more than comprehensions (which I think are already > over-used) I find myself frequently wishing for a better way to write > > m = pat.match(text) > if m: > g = m.group(0) > if check(g): # Some check that's not easily expressed as a regex > print(g) > > It would be nice if I could write that as > > if (m = pat.match(text)) and check((g = m.group(0))): > print(g) > > or > > if (pat.match(text) as m) and check((m.group(0) as g)): > print(g) > Agreed. I'm currently thinking that I need to do what several people have suggested and break this into two completely separate PEPs: 1) Sublocal namespacing 2) Assignment expressions Sublocal names can be used in a number of ways. There could be a "with sublocal EXPR as NAME:" syntax that actually disposes of the name binding at the end of the block, and "except Exception as e:" could shadow rather than unbinding. Maybe list comprehensions could change, too - instead of creating a function, they just create a sublocal scope. That may be the best way forward. I'm not sure. ChrisA From stefan_ml at behnel.de Sun Mar 25 03:22:13 2018 From: stefan_ml at behnel.de (Stefan Behnel) Date: Sun, 25 Mar 2018 09:22:13 +0200 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: <5AB6C543.303@UGent.be> References: <5AB41364.8010500@UGent.be> <8fad0e0f0a7d46d890345ea99b03c97b@xmail102.UGent.be> <5AB6C543.303@UGent.be> Message-ID: Jeroen Demeyer schrieb am 24.03.2018 um 22:38: > On 2018-03-24 09:09, Nick Coghlan wrote: >> If it was just about introspection, then we could define a new protocol >> method or attribute, update the inspect module to respect it, and call >> it done. > > True. In fact, I got a long way by just defining inspect.isfunction as > > def isfunction(obj): > ??? return hasattr(type(obj), "__code__") This relies on the fact that Cython attaches an (empty) code object to its C implemented functions, which could be considered a hack but is needed for things like source file reporting, and also line reporting (although we don't currently use that, but an even worse hack). The main problem, though, is that a lot of code out there does not use the "inspect" module but "types" to distinguish its input. That probably got a bit better over time with the ABCs, but for things like functions, generators and coroutines, I doubt that people would easily consider them. "types" just seems too obvious at first thought. Which suggests that it could help to add a visible note to the CPython docs of the "types" module that it's often better to use "inspect" or the ABCs. Seems worth a PR. Stefan From rosuav at gmail.com Sun Mar 25 03:52:09 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 18:52:09 +1100 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: References: <5AB41364.8010500@UGent.be> <8fad0e0f0a7d46d890345ea99b03c97b@xmail102.UGent.be> <5AB6C543.303@UGent.be> Message-ID: On Sun, Mar 25, 2018 at 6:22 PM, Stefan Behnel wrote: > Jeroen Demeyer schrieb am 24.03.2018 um 22:38: >> On 2018-03-24 09:09, Nick Coghlan wrote: >>> If it was just about introspection, then we could define a new protocol >>> method or attribute, update the inspect module to respect it, and call >>> it done. >> >> True. In fact, I got a long way by just defining inspect.isfunction as >> >> def isfunction(obj): >> return hasattr(type(obj), "__code__") > > This relies on the fact that Cython attaches an (empty) code object to its > C implemented functions, which could be considered a hack but is needed for > things like source file reporting, and also line reporting (although we > don't currently use that, but an even worse hack). > > The main problem, though, is that a lot of code out there does not use the > "inspect" module but "types" to distinguish its input. That probably got a > bit better over time with the ABCs, but for things like functions, > generators and coroutines, I doubt that people would easily consider them. > "types" just seems too obvious at first thought. > > Which suggests that it could help to add a visible note to the CPython docs > of the "types" module that it's often better to use "inspect" or the ABCs. > Seems worth a PR. If people are checking "if type(x) is types.FunctionType", they're doing it wrong. But if the check is "isinstance(x, types.FunctionType)", maybe the solution is to register builtin_function_or_method as a virtual subclass? ChrisA From ncoghlan at gmail.com Sun Mar 25 04:04:19 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 18:04:19 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On 25 March 2018 at 17:18, Chris Angelico wrote: > Agreed. I'm currently thinking that I need to do what several people > have suggested and break this into two completely separate PEPs: > > 1) Sublocal namespacing > 2) Assignment expressions > > Sublocal names can be used in a number of ways. There could be a "with > sublocal EXPR as NAME:" syntax that actually disposes of the name > binding at the end of the block, The scoping affects the name binding rather than the expression evaluation, so I'd expect any such variant to be: with EXPR as sublocal NAME: ... > and "except Exception as e:" could > shadow rather than unbinding. Maybe list comprehensions could change, > too - instead of creating a function, they just create a sublocal > scope. > > That may be the best way forward. I'm not sure. I think you can treat it as an open design question within the current PEP by tweaking the PEP title to be "Name binding as an expression". If we allow expression level name binding at all, it will be an either/or choice between binding to a new sublocal scope and binding regular locals, and you can handle that by describing sublocals as your current preferred option, but point out that the same *syntactic* idea could be adopted without introducing the sublocals semantics (in the latter case, the distinction created by the PEP would just be between "assignment statements" and "assignment expressions", rather than between "local assignments" and "sublocal assignments"). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ctaank at gmail.com Sun Mar 25 05:23:15 2018 From: ctaank at gmail.com (Cammil Taank) Date: Sun, 25 Mar 2018 10:23:15 +0100 Subject: [Python-ideas] Inplace assignment for "boolean or"? Message-ID: Hi, I find a common idiom in Python is: x = x or 'some other value' This is highly reminiscent of the problem inplace operators solve. Would it be a good idea to consider an inplace operator for this, perhaps: x or= 'some other value' ? Thanks, Cammil -------------- next part -------------- An HTML attachment was scrubbed... URL: From christoph at grothesque.org Sun Mar 25 05:31:09 2018 From: christoph at grothesque.org (Christoph Groth) Date: Sun, 25 Mar 2018 11:31:09 +0200 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: (Nick Coghlan's message of "Sun, 25 Mar 2018 00:14:04 +1000") References: <87po3tlhyi.fsf@grothesque.org> Message-ID: <87fu4olcwy.fsf@grothesque.org> Nick Coghlan wrote: > On 24 March 2018 at 23:29, Christoph Groth wrote: > > x = [n * m for n in range(4) for m in range(5)] > > > > be equally well equivalent to > > > > def (): > > ret = [] > > for n in range(4): > > for m in range(5): > > ret.append(n * m) > > return ret > > x = () > > > > As far as I can tell, with today's Python both implementations (yours > > and mine) are indistinguishable by the user. > > > > They can be distinguished, just not at module or function scope. To give a > concrete example: > > ========== > >>> class C: > ... sequence = range(10) > ... listcomp = [x for x in sequence] > ... def works(data): > ... return list(data) > ... from_works = works(sequence) > ... def fails(): > ... return list(sequence) > ... > >>> C.listcomp > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > >>> C.from_works > [0, 1, 2, 3, 4, 5, 6, 7, 8, 9] > >>> C.fails() > Traceback (most recent call last): > File "", line 1, in > File "", line 8, in fails > NameError: name 'sequence' is not defined > ========== Thanks a lot for this example. So you mean that while listcomp = [x for x in sequence] works above listcomp = [(y,x) for y in sequence for x in sequence] doesn't (but it would if the class was replaced by a function). That's indeed a bit strange, and I would consider it somewhat of a wart of the language. But as far as I can tell remaining compatible with the above behavior does not force us to leak assignments from the outermost scope of a comprehension. I.e. there's nothing in the language currently that forces listcomp = [x for x in (r := sequence)] to leak the name "r". Granted, it's a bit strange if in the above line the name "sequence" is evaluated in class scope but the name "r" is set in the comprehension scope, but since currently there is no way to assign values to names in comprehensions this "hybrid" behavior would be backwards-compatible, and less surprising than leaking "r". This could be fixed if backwards compatbility may be ever broken again, but until then I don't expect it to be a problem. From christoph at grothesque.org Sun Mar 25 05:55:59 2018 From: christoph at grothesque.org (Christoph Groth) Date: Sun, 25 Mar 2018 11:55:59 +0200 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: Message-ID: <87bmfclbrk.fsf@grothesque.org> Nick Coghlan wrote: > It would also result in two different ways to handle traditional > assignments: > > x = expr > x := expr > > Perhaps ":=" could be explicitly restricted to only single names on > the LHS, without any of the clever unpacking features of full > assignment statements? Unlike full assignment statements, assignment > expressions also wouldn't have anywhere to put a type annotation. Why would "=" have to be kept for anything else but backwards compatibility and saving keystrokes? The expression var: annotation := value could assign a value to a name and it would have the corresponding value. I guess that it's problematic for the language grammar to allow things like for i in (var: annotation := value): print(i) In that case such constructions could be declared illegal. > P.S. Pascal was one of the first languages I used to write a > non-trivial application (a game of Battleships), so I'm predisposed > towards liking ":=" as an assignment operator :) Having an assignment operator that is optically distinct from '=' is IMO a strong point of Pascal. For Python, an even better assignment operator would be IMO "<-", but this is not possible without breaking backwards compatibility, and also it's a bit difficult to type on an English keyboard. From rosuav at gmail.com Sun Mar 25 06:13:56 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 21:13:56 +1100 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <87fu4olcwy.fsf@grothesque.org> References: <87po3tlhyi.fsf@grothesque.org> <87fu4olcwy.fsf@grothesque.org> Message-ID: On Sun, Mar 25, 2018 at 8:31 PM, Christoph Groth wrote: > That's indeed a bit strange, and I would consider it somewhat of a wart > of the language. But as far as I can tell remaining compatible with the > above behavior does not force us to leak assignments from the outermost > scope of a comprehension. I.e. there's nothing in the language > currently that forces > > listcomp = [x for x in (r := sequence)] > > to leak the name "r". > > Granted, it's a bit strange if in the above line the name "sequence" is > evaluated in class scope but the name "r" is set in the comprehension > scope, but since currently there is no way to assign values to names in > comprehensions this "hybrid" behavior would be backwards-compatible, and > less surprising than leaking "r". It seems fine in a simple example, but remember, an assignment expression can be used ANYWHERE in an expression. Consider: listcomp = [x for x in obj[r := f()][x + r] ] Which parts happen in the inner scope and which in the outer scope? If 'r' is created in a subscope, it has to be a subscope of the outer scope; if it's not subscoped, it has to be directly in the outer scope. It can't sanely be in the inner scope. ChrisA From steve at pearwood.info Sun Mar 25 06:31:03 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 21:31:03 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: <20180325103103.GA16661@ando.pearwood.info> On Sun, Mar 25, 2018 at 05:00:37PM +1000, Nick Coghlan wrote: > Given the existing namespace stack of > builtin<-global<-nonlocal<-local, one potential short name would be > "sublocal", to indicate that these references are even more local than > locals (they're *so* local, they don't even appear in locals()!). If we go down this track, +1 on the name "sublocal". [...] > And if we do end up going with the approach of defining a separate > sublocal namespace, the fact that "n := ..." binds a sublocal, while > "n = ..." and "... as n" both bind regular locals would be clearer > than having the target scope of "as" be context dependent. The scope issue is a good argument for avoiding "as" if we have sublocal binding. One thing I like about the (expression as name) syntax is that the expression comes first. The Pascal-style := binding syntax reverses that. While we're bike-shedding, here are some alternatives to compare: target = default if (expression as name) is None else name target = default if (name := expression) is None else name target = default if (expression -> name) is None else name target = default if (name <- expression) is None else name The arrow assignment operators <- and -> are both used by R. A dedicated non-ASCII forward arrow is also used by some programmable calculators, including HP and TI. But let's not start using non-ASCII symbols yet. If we don't like a symbolic operator, we could channel BASIC from the 1970s and write something like this: target = default if (let expression = name) is None else name Pros: - requiring the keyword "let" prevents the "equals versus assignment" class of errors; - easier to search for a keyword than a symbolic operator; Cons: - more verbose; - looks like BASIC; - requires a new keyword. -- Steve From steve at pearwood.info Sun Mar 25 06:53:57 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 25 Mar 2018 21:53:57 +1100 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: References: <5AB41364.8010500@UGent.be> <8fad0e0f0a7d46d890345ea99b03c97b@xmail102.UGent.be> <5AB6C543.303@UGent.be> Message-ID: <20180325105357.GB16661@ando.pearwood.info> On Sun, Mar 25, 2018 at 06:52:09PM +1100, Chris Angelico wrote: > If people are checking "if type(x) is types.FunctionType", they're > doing it wrong. That depends on what "it" is that they're doing. If they want a callable, then they're doing it wrong. If they want a function regardless of whether it's written in Python or C or Cython, then they're doing it wrong. But if they *specifically* want a Python function, perhaps so they can pull it apart in ways you can't do to functions written in other languages, then it is fine. -- Steve From rosuav at gmail.com Sun Mar 25 07:21:26 2018 From: rosuav at gmail.com (Chris Angelico) Date: Sun, 25 Mar 2018 22:21:26 +1100 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: <20180325105357.GB16661@ando.pearwood.info> References: <5AB41364.8010500@UGent.be> <8fad0e0f0a7d46d890345ea99b03c97b@xmail102.UGent.be> <5AB6C543.303@UGent.be> <20180325105357.GB16661@ando.pearwood.info> Message-ID: On Sun, Mar 25, 2018 at 9:53 PM, Steven D'Aprano wrote: > On Sun, Mar 25, 2018 at 06:52:09PM +1100, Chris Angelico wrote: > >> If people are checking "if type(x) is types.FunctionType", they're >> doing it wrong. > > That depends on what "it" is that they're doing. > > If they want a callable, then they're doing it wrong. > > If they want a function regardless of whether it's written in Python or > C or Cython, then they're doing it wrong. > > But if they *specifically* want a Python function, perhaps so they can > pull it apart in ways you can't do to functions written in other > languages, then it is fine. > My point was that checking if the type of something *is* some type object, that's the wrong way to do things, and that a proper isinstance check has other ways to solve the underlying problem. ChrisA From christoph at grothesque.org Sun Mar 25 08:44:21 2018 From: christoph at grothesque.org (Christoph Groth) Date: Sun, 25 Mar 2018 14:44:21 +0200 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: Message-ID: <87605kl3yy.fsf@grothesque.org> Chris Angelico wrote: > On Sun, Mar 25, 2018 at 8:31 PM, Christoph Groth > wrote: > > That's indeed a bit strange, and I would consider it somewhat of a wart > > of the language. But as far as I can tell remaining compatible with the > > above behavior does not force us to leak assignments from the outermost > > scope of a comprehension. I.e. there's nothing in the language > > currently that forces > > > > listcomp = [x for x in (r := sequence)] > > > > to leak the name "r". > > It seems fine in a simple example, but remember, an assignment > expression can be used ANYWHERE in an expression. Consider: > > listcomp = [x for x in obj[r := f()][x + r] ] > > Which parts happen in the inner scope and which in the outer scope? If > 'r' is created in a subscope, it has to be a subscope of the outer > scope; if it's not subscoped, it has to be directly in the outer > scope. It can't sanely be in the inner scope. The example you give is indeed difficult to understand, but all programming languages, even Python, allow to write confusing code. Already today we can have listcomp = [x for x in obj[f()][x + r]] and, while valid, that's hardly an example worthy of emulation. Compared to this, the example that you give is indeed even more treacherous, because innocent people could assume that the 'r' in '[x + r]' is the one set just before with ':='. But is this obscure behavior really a problem? A bug can only happen if 'r' is also defined in the surrounding scope, otherwise there will be an error. And if this is indeed considered a problem, such confusing situations could be detected and flagged as an error by the Python compiler until the underlying inconsistency in the language is fixed. I think that it's a helpful guideline to imagine what the ideal behavior should be if we were not constrained by backwards compatibility, and then try to follow it. In the case at hand, we all seem to agree that the fact that the outermost iterator of a comprehension is evaluated in the surrounding scope is somewhat of a wart, although one that is rarely visible. The existence of a wart shouldn't pull us further into the wrong direction. From ncoghlan at gmail.com Sun Mar 25 09:43:06 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Sun, 25 Mar 2018 23:43:06 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <87605kl3yy.fsf@grothesque.org> References: <87605kl3yy.fsf@grothesque.org> Message-ID: On 25 March 2018 at 22:44, Christoph Groth wrote: > I think that it's a helpful guideline to imagine what the ideal behavior > should be if we were not constrained by backwards compatibility, and > then try to follow it. In the case at hand, we all seem to agree that > the fact that the outermost iterator of a comprehension is evaluated in > the surrounding scope is somewhat of a wart, although one that is rarely > visible. There's no such agreement, since generator expressions have worked that way since they were first introduced almost 15 years ago: https://www.python.org/dev/peps/pep-0289/#the-details It's much easier to see the difference with generator expressions, since the evaluation of the loop body is delayed until the generator is iterated over, while the evaluation of the outermost iterator is immediate (so that any exceptions are more easily traced to the code responsible for them, and so that they don't need to create a closure in the typical case). With comprehensions, the entire nested scope gets evaluated eagerly, so it's harder to detect that there's a difference in the evaluation scope of the outermost iterator in normal use. That difference *does* exist though, and we're not going to tie ourselves into knots to try to hide it (since the exact same discrepancy will necessarily exist for generator expressions, and semantic consistency between genexps and the corresponding comprehensions is highly desirable). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From schesis at gmail.com Sun Mar 25 11:14:20 2018 From: schesis at gmail.com (Zero Piraeus) Date: Sun, 25 Mar 2018 12:14:20 -0300 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: : On 25 March 2018 at 02:34, Guido van Rossum wrote: > [...] I think we need a short, crisp name for the new variable type. Disposables? -[]z. From guido at python.org Sun Mar 25 11:24:07 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Mar 2018 08:24:07 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Sun, Mar 25, 2018 at 12:18 AM, Chris Angelico wrote: > [...] > Agreed. I'm currently thinking that I need to do what several people > have suggested and break this into two completely separate PEPs: > > 1) Sublocal namespacing > 2) Assignment expressions > > Sublocal names can be used in a number of ways. There could be a "with > sublocal EXPR as NAME:" syntax that actually disposes of the name > binding at the end of the block, and "except Exception as e:" could > shadow rather than unbinding. Maybe list comprehensions could change, > too - instead of creating a function, they just create a sublocal > scope. > > That may be the best way forward. I'm not sure. > > ChrisA > I don't think the PEP should be split up into two PEPs. The two topics are too closely related. But you can have separate discussions about each issue in the PEP. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Sun Mar 25 11:50:03 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Mar 2018 08:50:03 -0700 Subject: [Python-ideas] Fixing class scope brainstorm Message-ID: In the PEP 572 threads there's some grumbling about class scopes. Here's a random brainstorm. How about we change the scoping rules so that only functions defined with 'def' (or inside one of those) cannot see the class scope, and comprehensions and lambdas treat the class scope as an outer scope so they can close over class variables? Not sure what should happen to nested classes, they should probably be lumped in with the 'def' scopes. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From elazarg at gmail.com Sun Mar 25 12:42:52 2018 From: elazarg at gmail.com (Elazar) Date: Sun, 25 Mar 2018 16:42:52 +0000 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On Sun, Mar 25, 2018 at 5:51 PM Guido van Rossum wrote: > In the PEP 572 threads there's some grumbling about class scopes. > > Here's a random brainstorm. How about we change the scoping rules so that > only functions defined with 'def' (or inside one of those) cannot see the > class scope, and comprehensions and lambdas treat the class scope as an > outer scope so they can close over class variables? > > Not sure what should happen to nested classes, they should probably be > lumped in with the 'def' scopes. > > I would expect classes to behave like comprehensions - i.e. code that is not delayed will see the class' scope as an enclosing scope (since it is not namespaced yet so there's no other way to refer to it) and delayed code (e.g. lambdas or defs) will not. Elazar -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Sun Mar 25 12:46:44 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Sun, 25 Mar 2018 09:46:44 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> <5AB71EB3.5030308@stoneleaf.us> <5AB720A9.5050205@stoneleaf.us> Message-ID: <5AB7D274.1020100@stoneleaf.us> On 03/24/2018 09:24 PM, Nick Coghlan wrote: > No, the fact that the expression defining the outermost iterable gets evaluated in the outer scope is behaviour that's > explicitly tested for in the regression test suite. > > The language reference spells out that this is intentional for generator expressions, where it has the added benefit of > reporting errors in the outermost iterable expression at the point where the genexp is defined, rather than at the point > where it gets iterated over: https://docs.python.org/3/reference/expressions.html#generator-expressions > > Independently of the pragmatic "getting them to work sensibly at class scope" motivation, comprehensions inherit those > semantics by way of the intended semantic equivalence between "[x for x in sequence]" and "list(x for x in sequence)". Thank you (everyone!) for your patience. Using the latest example from Angelico and myself: --> d = 9 ... def func(_iter): ... ... body of comprehension --> func((d as e)) The sticking point is the `func((d as e))`, which to my mind should happen inside the comprehension, but needs to happen outside -- and the reason it has to happen outside is so that the interpretor can verify that `d` actually exists; however, I think we can have both: --> def func(_iter): ... ... ... e = d ... ... --> d --> func() This way, the assignment does not leak, but the referenced name is still looked up and verified to exist outside the function call. I don't know how easy/difficult that would be to implement. At this point a simple confirmation that I understand and that in theory the above would solve both issues is what I'm looking for. Or, of course, the reasons why the above would not, in theory, work. -- ~Ethan~ From george at fischhof.hu Sun Mar 25 13:59:28 2018 From: george at fischhof.hu (George Fischhof) Date: Sun, 25 Mar 2018 19:59:28 +0200 Subject: [Python-ideas] New PEP proposal -- Pathlib Module ShouldContain All File Operations -- version 2 In-Reply-To: References: <83F52502-CAD7-4F7B-B7C6-C658DD55169D@barrys-emacs.org> <4075V41VMzzFqVS@mail.python.org> <0a0d60d2-64f9-815c-0724-e33b81b3920a@mgmiller.net> Message-ID: Hi folks, according to comments it seems, it would better to propose a new module. I start to work out a new proposal. I created a BitBucket repo to be able to work together with folks who want help ;-) the repo address is: https://bitbucket.org/GeorgeFischhof/pep_for_filesystem_operations.git -- o -- shutil and path object relation: the documentation laks the information. For functions of os module it is written that functions updated in 3.6 and from that point they can use pathlib objects. I will check the status. (if the work together I will submit an issue report to update the documentation accordingly) BR, George -------------- next part -------------- An HTML attachment was scrubbed... URL: From eryksun at gmail.com Sun Mar 25 19:38:16 2018 From: eryksun at gmail.com (eryk sun) Date: Sun, 25 Mar 2018 23:38:16 +0000 Subject: [Python-ideas] New PEP proposal -- Pathlib Module Should Contain All File Operations -- version 2 In-Reply-To: References: Message-ID: On Sat, Mar 17, 2018 at 10:42 AM, George Fischhof wrote: > > All functions from os module accept path-like objects, > and none of the shutil functions. shutil indirectly supports __fspath__ paths via os and os.path. One exception is shutil.disk_usage() on Windows, which only supports str strings. This is fixed in 3.7, in resolution of issue 32556. Maybe it should be backported to 3.6. I like the idea of a high-level library that provides a subset of commonly used os, io, and shutil functionality in one place. But maybe a new module isn't required. shutil could be extended since its design goal is to provide "high-level operations on files and collections of files". That said, pathlib's goal to support "concrete paths [that] provide I/O operations" does seem incomplete. It should support copy, copytree, rmtree, and move methods. Also, a `parents` option should be added to Path.rmdir to implement removedirs, which mirrors how Path.mkdir implements makedirs. > os.link => path.hardlink_to I'm surprised this doesn't already exist. From tim.peters at gmail.com Sun Mar 25 19:40:34 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 25 Mar 2018 18:40:34 -0500 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: [Chris Angelico ] > ... > Not qualitative, but anecdotal: I do sometimes have to remind my > JavaScript students to check whether they've typed enough equals > signs. And that's in a language where the normal comparison operator > is ===. It's *still* not uncommon to see a comparison spelled =. I wonder whether Guido remembers this ;-) In the very, very, VERY early days, Python didn't have "==". Plain single "=" was used for both assignment and equality testing. So way back then. using "=" for embedded assignment too was intractable on the face of it. I'm not clear on why it changed. I remember writing to Guido about how to disambiguate between the "bind" and "test for equality" intents in isolated expressions typed at the interactive prompt, and next thing I knew the language changed to use "==" for the latter. In any case, I really don't want to see plain "=" for embedded assignments now. It's been the source of some of the worst C debugging nightmares I've wasted months of my life tracking down. Here's one that baffled an office full of MIT grads for half a day before I noticed the real problem: assert(n=2); You can fill in the rest of the story yourself - but you'll miss the full extent of the agony it caused ;-) Guido's original intuition was right: regardless of programming experience, it remains sorely tempting to write "x = y" when equality testing is intended. To this day I routinely get a syntax error in Python when doing that by mistake. For which I'm eternally grateful. Any other way of spelling it would be preferable. Requiring parentheses around it isn't enough; e.g., if (x = 1) or (y = 2): would almost certainly not do what was intended either. There's also that many newcomers from C-like languages habitually put all `if` and `while` tests in parens. I'm fond enough of ":=". Icon used that for assignment (embedded or otherwise), and I don't recall any bugs due to that. It was hard to confuse for "==" (whether conceptual confusion or visual confusion). That was just prone to the _other_ problem with embedded assignments: staring and staring trying to find the code where a name was most recently bound - "oh! it was bound inside the third nested clause in the `while` test two pages back". So it would be nice to combine embedded assignment with some notion of limited scope - but I'm much more concerned that the spelling not be easily confusable with "==". But not really a fan of overly wordy spellings either. There you go! All the rest follows trivially from the Zen of Python ;-) From rosuav at gmail.com Sun Mar 25 19:54:31 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Mar 2018 10:54:31 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Mon, Mar 26, 2018 at 10:40 AM, Tim Peters wrote: > Here's one that baffled an office full of MIT grads for half a day > before I noticed the real problem: > > assert(n=2); > > You can fill in the rest of the story yourself - but you'll miss the > full extent of the agony it caused ;-) I have to confess that my eye jumped down to the code before reading all of the text above it, and as a result, I thought you were pointing out that "n=2" for assignment would conflict with named argument usage. Which it does, but that wasn't your point :) Is there any way that ":=" can legally occur in Python source (ignoring string literals) currently? A colon is always followed by a 'suite' or a 'test', neither of which can start with '=', and annotated assignment has to have something between the ':' and '='. If it's 100% unambiguous, it could be the solution to the current wonkiness with 'as' having multiple meanings; in fact, there would then be a new form of consistency: 'as' binds the special result of a statement, but ':=' binds arbitrary expressions. ChrisA From guido at python.org Sun Mar 25 21:24:30 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Mar 2018 18:24:30 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Sun, Mar 25, 2018 at 4:40 PM, Tim Peters wrote: > [Chris Angelico ] > > ... > > Not qualitative, but anecdotal: I do sometimes have to remind my > > JavaScript students to check whether they've typed enough equals > > signs. And that's in a language where the normal comparison operator > > is ===. It's *still* not uncommon to see a comparison spelled =. > > I wonder whether Guido remembers this ;-) In the very, very, VERY > early days, Python didn't have "==". Plain single "=" was used for > both assignment and equality testing. So way back then. using "=" for > embedded assignment too was intractable on the face of it. > Wow, I did not remember this. In fact I had to track down the 0.9.1 release that's somewhere on the web to see for myself. :-) Should add this to the HOPL-IV paper if I end up writing it (I'm still far from decided either way). > I'm not clear on why it changed. I remember writing to Guido about > how to disambiguate between the "bind" and "test for equality" intents > in isolated expressions typed at the interactive prompt, and next > thing I knew the language changed to use "==" for the latter. > Hm, that's probably why -- the desire for top-level expressions to allow comparison. Also probably the realization that this is one thing where (at the time) this particular difference with C/C++ was just annoying for most new users. I'm assuming that <>, the ancient alternate spelling for != (that Barry still misses), came from the same source: ABC ( https://homepages.cwi.nl/~steven/abc/qr.html#TESTS). But there was no compelling reason to remove <> (only to add !=) so it lingered until 3.0. Presumably ABC got both from Pascal ( https://www.tutorialspoint.com/pascal/pascal_relational_operators.htm). > In any case, I really don't want to see plain "=" for embedded > assignments now. > [...] > > I'm fond enough of ":=". Icon used that for assignment (embedded or > otherwise), and I don't recall any bugs due to that. It was hard to > confuse for "==" (whether conceptual confusion or visual confusion). > Most languages I learned in the '70s used it: both Algols, Pascal. (Though not Fortran.) > That was just prone to the _other_ problem with embedded assignments: > staring and staring trying to find the code where a name was most > recently bound - "oh! it was bound inside the third nested clause in > the `while` test two pages back". So it would be nice to combine > embedded assignment with some notion of limited scope - but I'm much > more concerned that the spelling not be easily confusable with "==". > But not really a fan of overly wordy spellings either. > The "two pages back" problem can happen just as easy with regular assignments or for-loop control variables. > There you go! All the rest follows trivially from the Zen of Python ;-) > I gotta say I'm warming up to := in preference over 'as', *if* we're going to do this at all (not a foregone conclusion at all). The scope question is far from easy though. I find it particularly grating that an inline assignment occurs in an 'if' statement, its scope is the entire body of the 'if'. If that body is two pages long, by the end of it the reader (or even the writer!) may well have lost track of where it was defined and may be confused by the consequence past the end of the body. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Sun Mar 25 21:29:52 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Mar 2018 12:29:52 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Mon, Mar 26, 2018 at 12:24 PM, Guido van Rossum wrote: > I gotta say I'm warming up to := in preference over 'as', *if* we're going > to do this at all (not a foregone conclusion at all). So am I, primarily due to its lack of syntactic ambiguities. > The scope question is far from easy though. I find it particularly grating > that an inline assignment occurs in an 'if' statement, its scope is the > entire body of the 'if'. If that body is two pages long, by the end of it > the reader (or even the writer!) may well have lost track of where it was > defined and may be confused by the consequence past the end of the body. I think this one can be given to style guides. The useful situations (eg regex match capturing) are sufficiently valuable that the less-useful ones can just come along for the ride, just like "x = lambda: ..." is perfectly valid even though "def" would be preferable. ChrisA From tim.peters at gmail.com Sun Mar 25 23:34:19 2018 From: tim.peters at gmail.com (Tim Peters) Date: Sun, 25 Mar 2018 22:34:19 -0500 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: [Tim] >> I wonder whether Guido remembers this ;-) In the very, very, VERY >> early days, Python didn't have "==". Plain single "=" was used for >> both assignment and equality testing. [Guido] > Wow, I did not remember this. In fact I had to track down the 0.9.1 release > that's somewhere on the web to see for myself. :-) Should add this to the > HOPL-IV paper if I end up writing it (I'm still far from decided either > way). See? I'm still good for _something_ sometimes ;-) >> I'm not clear on why it changed. I remember writing to Guido about >> how to disambiguate between the "bind" and "test for equality" intents >> in isolated expressions typed at the interactive prompt, and next >> thing I knew the language changed to use "==" for the latter. > Hm, that's probably why -- the desire for top-level expressions to allow > comparison. Also probably the realization that this is one thing where (at > the time) this particular difference with C/C++ was just annoying for most > new users. I don't have my email from those days, and have futilely tried to recall details. IIRC, it had never been discussed on the mailing list before, or in any private emails before. It just popped up one day when I was working in a Python shell, and there was _something_ subtle about it. You wrote back and expressed disappointment - that you had really wanted to keep "=" for both purposes. I started writing a reply suggesting a way out of whatever-the-heck the problem was, but before I finished the reply the next day you had already changed the implementation! Things moved quickly back then :-) Anyway, if your time machine is in good working order, I'd be pleased if you went back and restored the original vision. If, e.g., we needed to type >>> (x = y) True at the shell to get a top-level equality comparison, BFD. I can't believe it was _that_ simple, though. > I'm assuming that <>, the ancient alternate spelling for != (that Barry > still misses), came from the same source: ABC > (https://homepages.cwi.nl/~steven/abc/qr.html#TESTS). But there was no > compelling reason to remove <> (only to add !=) so it lingered until 3.0. > Presumably ABC got both from Pascal > (https://www.tutorialspoint.com/pascal/pascal_relational_operators.htm). Good inspirations! As noted next, at least Pascal used ":=" for assignment too. > ... > Most languages I learned in the '70s used it: both Algols, Pascal. (Though > not Fortran.) I mentioned Icon because I'm sure Pascal didn't have "embedded assignments" at all. Unsure about Algol, but I'd be surprised (certainly not Fortran). Icon has no "statements" at all: _everything_ in Icon is an expression, generating zero or more values. Embedded assignments are frequently used in idiomatic Icon, so I think it's especially relevant that I recall no bugs due to Icon's use of ":=" (for assignment) and "==" (for equality). Programmers simply never used one when the other was intended. In C, essentially everyone uses "=" when they intend "==" at times, and - as noted - I _still_ do that in Python regularly to this day. I'd be screwed if I got an unintended assignment instead of a SyntaxError. > ... > The "two pages back" problem can happen just as easy with regular > assignments or for-loop control variables. Yup, but eyeballs don't have to scan every square inch of the screen for those: `for` statement targets are easy to find, and assignment statement targets start flush with the first non-blank character of an assignment statement, where the eye naturally leaps to. When assignments can be embedded anywhere, you have to look everywhere to find them. But so it goes. Even if that can't be _stopped_, it's a matter of good practice to avoid making code inscrutable. > ... > I gotta say I'm warming up to := in preference over 'as', *if* we're going > to do this at all (not a foregone conclusion at all). I'm not assuming it will go in, I just want to nudge the PEP toward a proposal that doesn't suck so bad it's obviously doomed ;-) I'm uncertain whether I'd support it anyway. I do know that, e.g., if m := match(string) is not None: # do something with m violates my sense of outrage less than anything else I've seen ;-) And, ya, I'd _use_ it if it were implemented. But I can (continue to!) live without it. > The scope question is far from easy though. I find it particularly grating > that an inline assignment occurs in an 'if' statement, its scope is the > entire body of the 'if'. If that body is two pages long, by the end of it > the reader (or even the writer!) may well have lost track of where it was > defined and may be confused by the consequence past the end of the body. See my "every square inch" above ;-) At least if the scope _is_ limited to the body of the `if`, it's far more limited than in C or Icon. Of course I'm more interested in whether it can be used to write clearer code than in whether it can be abused to write muddier code. List comprehensions leapt to mind there. They're wonderfully clear in prudent doses, but for a while half my Stackoverflow answers started by chiding the questioner for an irrational fear of writing obvious loops instead ;-) From guido at python.org Mon Mar 26 00:34:13 2018 From: guido at python.org (Guido van Rossum) Date: Sun, 25 Mar 2018 21:34:13 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Sun, Mar 25, 2018 at 6:29 PM, Chris Angelico wrote: > On Mon, Mar 26, 2018 at 12:24 PM, Guido van Rossum > wrote: > > The scope question is far from easy though. I find it particularly > grating > > that an inline assignment occurs in an 'if' statement, its scope is the > > entire body of the 'if'. If that body is two pages long, by the end of it > > the reader (or even the writer!) may well have lost track of where it was > > defined and may be confused by the consequence past the end of the body. > > I think this one can be given to style guides. The useful situations > (eg regex match capturing) are sufficiently valuable that the > less-useful ones can just come along for the ride, just like "x = > lambda: ..." is perfectly valid even though "def" would be preferable. > Not so fast. There's a perfectly reasonable alternative to sublocal scopes -- just let it assign to a local variable in the containing scope. That's the same as what Python does for for-loop variables. Note that for comprehensions it still happens to do the right thing (assuming we interpret the comprehension's private local scope to be the containing scope). This alternative has significant advantages in my view -- it doesn't require a whole new runtime mechanism to implement it (in CPython you can just generate bytecode to store and load locals), and it doesn't require a new section in the documentation to explain the new type of variable scope. Also it would make Steven happy. :-) Perhaps we could even remove the requirement to parenthesize the new form of assignment, so we could say that at the statement level " = " and " := " just mean the same thing, or "=" is a shorthand for ":=", or whatever. In most cases it still makes sense to parenthesize it, since the := operator should have the same priority as the regular assignment operator, which means that "if x := f() and x != 42:" is broken and should be written as "if (x := f()) and x != 42:". But that could be a style guide issue. (Also note that I'm not proposing to add "+:=" etc., even though in C that's supported.) It would still require carefully defining execution order in all cases, but we should probably do that anyway. At some point we could introduce a "block" statement similar to Lua's do/end or C's blocks (also found in many other languages). But there's not really a lot of demand for this -- style guides justly frown upon functions so long that they would benefit much. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Mar 26 00:56:38 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Mar 2018 15:56:38 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Mon, Mar 26, 2018 at 3:34 PM, Guido van Rossum wrote: > On Sun, Mar 25, 2018 at 6:29 PM, Chris Angelico wrote: >> >> On Mon, Mar 26, 2018 at 12:24 PM, Guido van Rossum >> wrote: >> > The scope question is far from easy though. I find it particularly >> > grating >> > that an inline assignment occurs in an 'if' statement, its scope is the >> > entire body of the 'if'. If that body is two pages long, by the end of >> > it >> > the reader (or even the writer!) may well have lost track of where it >> > was >> > defined and may be confused by the consequence past the end of the body. >> >> I think this one can be given to style guides. The useful situations >> (eg regex match capturing) are sufficiently valuable that the >> less-useful ones can just come along for the ride, just like "x = >> lambda: ..." is perfectly valid even though "def" would be preferable. > > > Not so fast. There's a perfectly reasonable alternative to sublocal scopes > -- just let it assign to a local variable in the containing scope. That's > the same as what Python does for for-loop variables. Note that for > comprehensions it still happens to do the right thing (assuming we interpret > the comprehension's private local scope to be the containing scope). > > This alternative has significant advantages in my view -- it doesn't require > a whole new runtime mechanism to implement it (in CPython you can just > generate bytecode to store and load locals), and it doesn't require a new > section in the documentation to explain the new type of variable scope. Also > it would make Steven happy. :-) I'm still liking the sublocal system, but making assignment expressions capable of standing plausibly without them is a Good Thing. > Perhaps we could even remove the requirement to parenthesize the new form of > assignment, so we could say that at the statement level " = " and > " := " just mean the same thing, or "=" is a shorthand for ":=", > or whatever. In most cases it still makes sense to parenthesize it, since > the := operator should have the same priority as the regular assignment > operator, which means that "if x := f() and x != 42:" is broken and should > be written as "if (x := f()) and x != 42:". But that could be a style guide > issue. (Also note that I'm not proposing to add "+:=" etc., even though in C > that's supported.) That's about where I was thinking of putting it; "test" gets defined potentially as "NAME := test". It has to right-associate. At the moment, I'm going to be restricting it to simple names only, so you can't say "x[1] := 2". That may be changed later. ChrisA From kirillbalunov at gmail.com Mon Mar 26 04:14:43 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 26 Mar 2018 11:14:43 +0300 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: Hi Chris, would you mind to add this syntactic form `(expr -> var)` to alternative syntax section, with the same semantics as `(expr as var)`. It seems to me that I've seen this form previously in some thread (can't find where), but it does not appear in alt. syntax section. As for me this form has several benefits: 1. Currently it is a SyntaxError Really there exist some intersection with syntax to annotate function return type, but it has much smaller impact than `as` variant. 2. This form looks It looks as readable (also the expression comes first, which appeals to me) as the 'as' variant. 3. It is clearly distinguishable from the usual assignment statement (which also appeals to me) Suppose someday someone will want to have a type hint on a local variable (I think sublocal are safer on this part), then: ``` while (x: int := counter): do_some_stuff ``` vs ``` while (counter -> x: int): do_some_stuff ``` Maybe it is too subjective, but the second form looks better for me. taking in all the ?dvantages of the `as` form. Also this '->' form can be extended to some sort of tuple unpacking. I don't think that tuple unpacking is a good example, but nevertheless:) And will make further attempts to introduce ` +:=` impossible. With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertomartinezp at gmail.com Mon Mar 26 04:35:17 2018 From: robertomartinezp at gmail.com (=?UTF-8?Q?Roberto_Mart=C3=ADnez?=) Date: Mon, 26 Mar 2018 08:35:17 +0000 Subject: [Python-ideas] Generalized version of contextlib.close Message-ID: Hi, sometimes I need to use contextlib.close but over methods with a different name, for example stop(), halt(), etc. For those cases I have to write my own contextlib.close specialized version with a hard-coded method name. I think adding a "method" argument to contextlib.close can be very useful: @contextmanager def closing(thing, method="close"): try: yield thing finally: getattr(thing, method)() Or maybe something even more generic: @contextmanager def calling(fn, *args, **kwargs): try: yield finally: fn(*args, **kwargs) Best regards, Roberto -------------- next part -------------- An HTML attachment was scrubbed... URL: From rosuav at gmail.com Mon Mar 26 04:58:50 2018 From: rosuav at gmail.com (Chris Angelico) Date: Mon, 26 Mar 2018 19:58:50 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Mon, Mar 26, 2018 at 7:14 PM, Kirill Balunov wrote: > Hi Chris, would you mind to add this syntactic form `(expr -> var)` to > alternative syntax section, with the same semantics as `(expr as var)`. It > seems to me that I've seen this form previously in some thread (can't find > where), but it does not appear in alt. syntax section. Can do. I'm in the middle of some large edits, and will try to remember this when I get to that section. If you see another posting of the PEP and I haven't included it, please remind me and I'll add it. ChrisA From christoph at grothesque.org Mon Mar 26 05:06:50 2018 From: christoph at grothesque.org (Christoph Groth) Date: Mon, 26 Mar 2018 11:06:50 +0200 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: (Nick Coghlan's message of "Sun, 25 Mar 2018 23:43:06 +1000") References: <87605kl3yy.fsf@grothesque.org> Message-ID: <87a7uvmcid.fsf@grothesque.org> Nick Coghlan wrote: > On 25 March 2018 at 22:44, Christoph Groth wrote: > > I think that it's a helpful guideline to imagine what the ideal > > behavior should be if we were not constrained by backwards > > compatibility, and then try to follow it. In the case at hand, we > > all seem to agree that the fact that the outermost iterator of a > > comprehension is evaluated in the surrounding scope is somewhat of a > > wart, although one that is rarely visible. > > There's no such agreement, since generator expressions have worked > that way since they were first introduced almost 15 years ago: > https://www.python.org/dev/peps/pep-0289/#the-details I was referring to [1], but I see now that things are more subtle. Guido didn't express a dislike for the special treatment of the outermost iterator, just (if I understand correctly) for the consequences of this in class scope. [1] https://mail.python.org/pipermail/python-ideas/2018-March/049461.html > It's much easier to see the difference with generator expressions, > since the evaluation of the loop body is delayed until the generator > is iterated over, while the evaluation of the outermost iterator is > immediate (so that any exceptions are more easily traced to the code > responsible for them, and so that they don't need to create a closure > in the typical case). Thanks a lot for the explaination. I know many of the more obscure aspects of Python, but I wasn't aware of this. I'm still not sure whether I agree that it is a good idea to treat the evaluation of the outermost iterator differently from the rest. It can be especially confusing that the outermost iterator appears in the middle of the generator expression, surrounded by code that is evaluated in the inner scope. But I don't want to start a futile discussion about backwards-incompatible changes here. > With comprehensions, the entire nested scope gets evaluated eagerly, > so it's harder to detect that there's a difference in the evaluation > scope of the outermost iterator in normal use. That difference *does* > exist though, and we're not going to tie ourselves into knots to try > to hide it (since the exact same discrepancy will necessarily exist > for generator expressions, and semantic consistency between genexps > and the corresponding comprehensions is highly desirable). With tying into knots you mean my suggestion to bind both 'a' and 'b' at the internal scope in gen = (i+j for i in x[a := aa] for j in (b := bb)) even if x[a := aa] is evaluated in the outer scope? I tend to agree that this is indeed too arcane. But then how about making ":=" work always at the outer scope (and thus reserve the inner scope of generators only for the index variables), i.e. make the above equivalent to: def g(it): nonlocal b for i in it: for j in (b := bb): yield i+j gen = g(iter(x[a := aa])) That would be less confusing and indeed it could even turn out useful to be able to access names that were bound inside the generator outside of it. Note that the use of ":=" inside generator expressions would be optional, so it can be considered fair to treat such assignments differently from the implicit assignments to the loop variables. From steve at pearwood.info Mon Mar 26 07:18:23 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 22:18:23 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: <20180326111823.GF16661@ando.pearwood.info> On Mon, Mar 26, 2018 at 11:14:43AM +0300, Kirill Balunov wrote: > Hi Chris, would you mind to add this syntactic form `(expr -> var)` to > alternative syntax section, with the same semantics as `(expr as var)`. It > seems to me that I've seen this form previously in some thread (can't find > where), but it does not appear in alt. syntax section. That was probably my response to Nick: https://mail.python.org/pipermail/python-ideas/2018-March/049472.html I compared four possible choices: target = default if (expression as name) is None else name target = default if (name := expression) is None else name target = default if (expression -> name) is None else name target = default if (name <- expression) is None else name The two arrow assignment operators <- and -> are both taken from R. If we go down the sublocal scope path, which I'm not too keen on, then Nick's earlier comments convince me that we should avoid "as". In that case, my preferences are: (best) -> := <- as (worst) If we just bind to regular locals, then my preferences are: (best) as -> := <- (worst) Preferences are subject to change :-) -- Steve From kirillbalunov at gmail.com Mon Mar 26 07:42:32 2018 From: kirillbalunov at gmail.com (Kirill Balunov) Date: Mon, 26 Mar 2018 14:42:32 +0300 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180326111823.GF16661@ando.pearwood.info> References: <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> <20180326111823.GF16661@ando.pearwood.info> Message-ID: 2018-03-26 14:18 GMT+03:00 Steven D'Aprano : > That was probably my response to Nick: > > https://mail.python.org/pipermail/python-ideas/2018-March/049472.html > > I compared four possible choices: > > target = default if (expression as name) is None else name > target = default if (name := expression) is None else name > target = default if (expression -> name) is None else name > target = default if (name <- expression) is None else name > Yes, most likely :) > > The two arrow assignment operators <- and -> are both taken from R. > > I was also thinking about `<-` variant (but with a Haskell in mind), but with the current Python rules, it seems that it does not fit: >>> x = 10 >>> (x <- 5) False With kind regards, -gdg -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Mar 26 08:31:28 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Mon, 26 Mar 2018 23:31:28 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180326111823.GF16661@ando.pearwood.info> Message-ID: <20180326123128.GG16661@ando.pearwood.info> On Mon, Mar 26, 2018 at 02:42:32PM +0300, Kirill Balunov wrote: > I was also thinking about `<-` variant (but with a Haskell in mind), but > with the current Python rules, it seems that it does not fit: Ah, of course not, the dreaded unary operator strikes again! (I was just chatting with Chris about unary operators off-list earlier today.) -- Steve From ctaank at gmail.com Mon Mar 26 08:48:34 2018 From: ctaank at gmail.com (Cammil Taank) Date: Mon, 26 Mar 2018 13:48:34 +0100 Subject: [Python-ideas] In-place assignment for "boolean or"? Message-ID: Hi, I find a common idiom in Python is: x = x or 'some other value' This is highly reminiscent of the problem inplace operators solve. Would it be a good idea to consider an inplace operator for this, perhaps: x or= 'some other value' ? Thanks, Cammil -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Mon Mar 26 09:33:06 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Mon, 26 Mar 2018 23:33:06 +1000 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On 26 March 2018 at 01:50, Guido van Rossum wrote: > In the PEP 572 threads there's some grumbling about class scopes. > > Here's a random brainstorm. How about we change the scoping rules so that > only functions defined with 'def' (or inside one of those) cannot see the > class scope, and comprehensions and lambdas treat the class scope as an > outer scope so they can close over class variables? > > Not sure what should happen to nested classes, they should probably be > lumped in with the 'def' scopes. I think it's mainly in comprehensions that folks may find the current behaviour surprising, as that's the case where we define a function and immediately call it, so it mostly looks like regular inline code execution. Once explicitly delayed invocation is involved (lambdas, def statements), folks seem to be more comfortable with the idea "Oh, of course that's going to behave like a method at class scope". Generator expressions also mostly get a pass, since the *iteration* is delayed, even though the generator-iterator itself is created immediately. One possibility that has occurred to me (but I've never investigated to check the technical feasibility) is to see whether we might be able to define a notion of "implied arguments" for the implicit comprehension and generator expression function definitions. The gist of that idea would be: - when walking the symbol table for a comprehension or genexp, keep track of every variable name mentioned in that subtree which is not resolved in that subtree (which we already have the machinery to do, as it's needed for compiling functions in general) - treat all those names as *implied arguments* to the implicit function, and call it with the right references in the right positions While we'd likely still want to keep evaluation of the outermost iterator outside the nested scope for the sake of better genexp tracebacks, that would still be enough to enable cases like: class C: y = 1 values = [x+y for x in range(10)] The trick would be to make it so that that comprehension gets expanded as: def _listcomp(_outermost_iter, y): result = [] for x in _outermost_iter: result.append(x+y) return result _listcomp_result = _listcomp(range(10), y) One of the more attractive aspects of this possibility (assuming it can be made to work) is that it isn't even a special case in the runtime name lookup rules - it's just changing the definition of the implicit functions that we create for comprehensions and generator expressions to make sure that they can refer to *any* name that resolves in the namespace where they're called (even invisible-to-the-compiler names like those injected via __prepare__ methods, or assignments via locals() at class scope). Cheers, Nick. P.S. I suspect we'd also find that this served as a performance optimisation, since it wouldn't only prebind class locals, it would prebind references to builtins and module globals as well. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From steve at pearwood.info Mon Mar 26 09:41:19 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Mar 2018 00:41:19 +1100 Subject: [Python-ideas] In-place assignment for "boolean or"? In-Reply-To: References: Message-ID: <20180326134118.GI16661@ando.pearwood.info> On Mon, Mar 26, 2018 at 01:48:34PM +0100, Cammil Taank wrote: > Hi, > > I find a common idiom in Python is: > > x = x or 'some other value' I don't find it very common. I don't think I've ever used it or seen it. I've occasionally used: spam = eggs or 'something' but mostly I use: result = function(eggs or 'something') (and even that is quite unusual). But others may use it more often. > This is highly reminiscent of the problem inplace operators solve. Not really: the other augmented operators all connect to ordinary operators that can be overloaded. But `or` is special, it's a short-cutting operator that is treated specially by the interpreter. And even if x is mutable, there's no way to make this an in-place operation. After `x = x or y`, there are only two possibilities: - x is still a reference to whatever it was before the operation; - or x now refers to the same object as y. So this is very different behaviour to (say) += and the other augmented operators. > Would it be a good idea to consider an inplace operator for this, perhaps: > > x or= 'some other value' Mixing a keyword with a symbol like that looks really weird. Others may disagree, but to me, this pattern isn't common enough to justify specialist syntax. I might be a little less negative if the syntax didn't look so strange to me. But only a little. -- Steve From ctaank at gmail.com Mon Mar 26 09:56:41 2018 From: ctaank at gmail.com (Cammil Taank) Date: Mon, 26 Mar 2018 14:56:41 +0100 Subject: [Python-ideas] In-place assignment for "boolean or"? In-Reply-To: <20180326134118.GI16661@ando.pearwood.info> References: <20180326134118.GI16661@ando.pearwood.info> Message-ID: Given the arguments you've raised, I am now inclined to agree with you. However, for completeness of the discussion, I want to indicate the primary intended use case: def fn(x=None): if not x: x = some_complex_or_runtime_dependent_result() For me this appears quite frequently. Admittedly the condition should likely be `x is None`. On 26 March 2018 at 14:41, Steven D'Aprano wrote: > On Mon, Mar 26, 2018 at 01:48:34PM +0100, Cammil Taank wrote: > > Hi, > > > > I find a common idiom in Python is: > > > > x = x or 'some other value' > > I don't find it very common. I don't think I've ever used it or seen it. > I've occasionally used: > > spam = eggs or 'something' > > but mostly I use: > > result = function(eggs or 'something') > > (and even that is quite unusual). But others may use it more often. > > > > This is highly reminiscent of the problem inplace operators solve. > > Not really: the other augmented operators all connect to ordinary > operators that can be overloaded. But `or` is special, it's a > short-cutting operator that is treated specially by the interpreter. > > And even if x is mutable, there's no way to make this an in-place > operation. After `x = x or y`, there are only two possibilities: > > - x is still a reference to whatever it was before the operation; > - or x now refers to the same object as y. > > So this is very different behaviour to (say) += and the other augmented > operators. > > > > Would it be a good idea to consider an inplace operator for this, > perhaps: > > > > x or= 'some other value' > > Mixing a keyword with a symbol like that looks really weird. > > Others may disagree, but to me, this pattern isn't common enough to > justify specialist syntax. I might be a little less negative if the > syntax didn't look so strange to me. > > But only a little. > > > -- > Steve > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Mon Mar 26 10:33:52 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 26 Mar 2018 15:33:52 +0100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On 26/03/18 02:24, Guido van Rossum wrote: > I gotta say I'm warming up to := in preference over 'as',*if* we're going > to do this at all (not a foregone conclusion at all). I have the usual objection to glyphs (hard to look up or get help on), but ":=" raises two issues all of its own. * On the plus side, it looks like some kind of assignment. People reading through the code will not be overly surprised to find it results in a name binding. * On the minus side, it doesn't quite look like an assignment statement. While my crystal ball is cloudy, I can well imagine beginners becoming very confused over which symbol to use in which circumstance, and a lot of swearing when: x := f() if (y = g(x)) is not None: h(y) results in syntax errors. I'm inclined to think you want assignment expressions to look unlike assignment statements to avoid this sort of confusion. -- Rhodri James *-* Kynesim Ltd From p.f.moore at gmail.com Mon Mar 26 10:34:00 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Mon, 26 Mar 2018 15:34:00 +0100 Subject: [Python-ideas] In-place assignment for "boolean or"? In-Reply-To: References: <20180326134118.GI16661@ando.pearwood.info> Message-ID: See https://www.python.org/dev/peps/pep-0505/ On 26 March 2018 at 14:56, Cammil Taank wrote: > Given the arguments you've raised, I am now inclined to agree with you. > > However, for completeness of the discussion, I want to indicate the primary > intended use case: > > def fn(x=None): > if not x: > x = some_complex_or_runtime_dependent_result() > > For me this appears quite frequently. Admittedly the condition should likely > be `x is None`. > > > On 26 March 2018 at 14:41, Steven D'Aprano wrote: >> >> On Mon, Mar 26, 2018 at 01:48:34PM +0100, Cammil Taank wrote: >> > Hi, >> > >> > I find a common idiom in Python is: >> > >> > x = x or 'some other value' >> >> I don't find it very common. I don't think I've ever used it or seen it. >> I've occasionally used: >> >> spam = eggs or 'something' >> >> but mostly I use: >> >> result = function(eggs or 'something') >> >> (and even that is quite unusual). But others may use it more often. >> >> >> > This is highly reminiscent of the problem inplace operators solve. >> >> Not really: the other augmented operators all connect to ordinary >> operators that can be overloaded. But `or` is special, it's a >> short-cutting operator that is treated specially by the interpreter. >> >> And even if x is mutable, there's no way to make this an in-place >> operation. After `x = x or y`, there are only two possibilities: >> >> - x is still a reference to whatever it was before the operation; >> - or x now refers to the same object as y. >> >> So this is very different behaviour to (say) += and the other augmented >> operators. >> >> >> > Would it be a good idea to consider an inplace operator for this, >> > perhaps: >> > >> > x or= 'some other value' >> >> Mixing a keyword with a symbol like that looks really weird. >> >> Others may disagree, but to me, this pattern isn't common enough to >> justify specialist syntax. I might be a little less negative if the >> syntax didn't look so strange to me. >> >> But only a little. >> >> >> -- >> Steve >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ > > > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > From mertz at gnosis.cx Mon Mar 26 10:44:33 2018 From: mertz at gnosis.cx (David Mertz) Date: Mon, 26 Mar 2018 07:44:33 -0700 Subject: [Python-ideas] In-place assignment for "boolean or"? In-Reply-To: References: <20180326134118.GI16661@ando.pearwood.info> Message-ID: I actually HAVE used this idiom moderately often. But unless you use overly long names, the current spelling is no trouble at all. In fact, it's not clear why your function is spelled as it is rather than with boolean shortcutting (assuming you like the shortcutting aesthetically). On Mon, Mar 26, 2018 at 6:56 AM, Cammil Taank wrote: > > However, for completeness of the discussion, I want to indicate the > primary intended use case: > def fn(x=None): if not x: x = some_complex_or_runtime_dependent_result() def fn2(x=None): x = x or some_complex_or_runtime_dependent_result() It looks like the boolean shortcutting version makes a line one character longer; but it does save a line and an indent level. Of course, if the 'not x' suite contains multiple statements, you need the 'if' either way. If, as you write, the condition is actually 'x is not None', you want the suite for sure. That's much better than a contorted ternary IMO. There were a couple PEPs about "None-shortcutting" for this not-so-special case of comparing with None. But also IMO, they all looked incredibly ugly for little benefit (lots of non-obvious syntax characters that look like Perl or APL) -- Keeping medicines from the bloodstreams of the sick; food from the bellies of the hungry; books from the hands of the uneducated; technology from the underdeveloped; and putting advocates of freedom in prisons. Intellectual property is to the 21st century what the slave trade was to the 16th. -------------- next part -------------- An HTML attachment was scrubbed... URL: From rhodri at kynesim.co.uk Mon Mar 26 10:11:50 2018 From: rhodri at kynesim.co.uk (Rhodri James) Date: Mon, 26 Mar 2018 15:11:50 +0100 Subject: [Python-ideas] In-place assignment for "boolean or"? In-Reply-To: References: Message-ID: <2044880d-0b25-464b-e51e-72e1685397ab@kynesim.co.uk> On 26/03/18 13:48, Cammil Taank wrote: > Hi, > > I find a common idiom in Python is: > > x = x or 'some other value' I believe this is a historic idiom from back before the ternary operator was added to Python: x = x if x else 'some other value' though honestly both of those are poor ways of spelling if not x: x = 'some other value' > This is highly reminiscent of the problem inplace operators solve. It really isn't, particularly not in Python > Would it be a good idea to consider an inplace operator for this, perhaps: > > x or= 'some other value' I don't find the combination of letters and symbols appealing, sorry. -- Rhodri James *-* Kynesim Ltd From ncoghlan at gmail.com Mon Mar 26 10:57:28 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Mar 2018 00:57:28 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On 26 March 2018 at 14:34, Guido van Rossum wrote: > Not so fast. There's a perfectly reasonable alternative to sublocal scopes > -- just let it assign to a local variable in the containing scope. That's > the same as what Python does for for-loop variables. Note that for > comprehensions it still happens to do the right thing (assuming we interpret > the comprehension's private local scope to be the containing scope). I finally remembered one of the original reasons that allowing embedded assignment to target regular locals bothered me: it makes named subexpressions public members of the API if you use them at class or module scope. (I sent an off-list email to Chris about that yesterday, so the next update to the PEP is going to take it into account). Similarly, if you use a named subexpression in a generator or coroutine and it gets placed in the regular locals() namespace, then you've now made that reference live for as long as the generator or coroutine does, even if you never need it again. By contrast, the sublocals idea strives to keep the *lifecycle* impact of naming a subexpression as negligible as possible - while a named subexpression might live a little longer than it used to as an anonymous subexpression (or substantially longer in the case of compound statement headers), it still wouldn't survive past the end of the statement where it appeared. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From guido at python.org Mon Mar 26 11:57:43 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Mar 2018 08:57:43 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Mon, Mar 26, 2018 at 7:57 AM, Nick Coghlan wrote: > On 26 March 2018 at 14:34, Guido van Rossum wrote: > > Not so fast. There's a perfectly reasonable alternative to sublocal > scopes > > -- just let it assign to a local variable in the containing scope. That's > > the same as what Python does for for-loop variables. Note that for > > comprehensions it still happens to do the right thing (assuming we > interpret > > the comprehension's private local scope to be the containing scope). > > I finally remembered one of the original reasons that allowing > embedded assignment to target regular locals bothered me: it makes > named subexpressions public members of the API if you use them at > class or module scope. (I sent an off-list email to Chris about that > yesterday, so the next update to the PEP is going to take it into > account). > > Similarly, if you use a named subexpression in a generator or > coroutine and it gets placed in the regular locals() namespace, then > you've now made that reference live for as long as the generator or > coroutine does, even if you never need it again. > > By contrast, the sublocals idea strives to keep the *lifecycle* impact > of naming a subexpression as negligible as possible - while a named > subexpression might live a little longer than it used to as an > anonymous subexpression (or substantially longer in the case of > compound statement headers), it still wouldn't survive past the end of > the statement where it appeared. > But this is not new: if you use a for-loop to initialize some class-level structure you have the same problem. There is also a standard solution (just 'del' it). -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Mon Mar 26 12:51:37 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 26 Mar 2018 17:51:37 +0100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: <0332cf7e-d9d6-503d-58d1-c58e8368dbd2@btinternet.com> On 26/03/2018 16:57, Guido van Rossum wrote: > On Mon, Mar 26, 2018 at 7:57 AM, Nick Coghlan > wrote: > > On 26 March 2018 at 14:34, Guido van Rossum > wrote: > > Not so fast. There's a perfectly reasonable alternative to > sublocal scopes > > -- just let it assign to a local variable in the containing > scope. That's > > the same as what Python does for for-loop variables. Note that for > > comprehensions it still happens to do the right thing (assuming > we interpret > > the comprehension's private local scope to be the containing scope). > > I finally remembered one of the original reasons that allowing > embedded assignment to target regular locals bothered me: it makes > named subexpressions public members of the API if you use them at > class or module scope. (I sent an off-list email to Chris about that > yesterday, so the next update to the PEP is going to take it into > account). > > Similarly, if you use a named subexpression in a generator or > coroutine and it gets placed in the regular locals() namespace, then > you've now made that reference live for as long as the generator or > coroutine does, even if you never need it again. > > By contrast, the sublocals idea strives to keep the *lifecycle* impact > of naming a subexpression as negligible as possible - while a named > subexpression might live a little longer than it used to as an > anonymous subexpression (or substantially longer in the case of > compound statement headers), it still wouldn't survive past the end of > the statement where it appeared. > > > But this is not new: if you use a for-loop to initialize some > class-level structure? you have the same problem. There is also a > standard solution (just 'del' it). True.? But there is a case for also saying that the for-loop variable's scope should have been limited to the for-loop-suite. (Not that it's feasible to make that change now, of course.) If I had a time-machine, I would add an assignment character (probably looking something like <- ) to the original ASCII character set.? Then "=" means equality - job done. Actually, probably a right-assignment character ( -> ) as well. Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Mon Mar 26 13:03:44 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Mon, 26 Mar 2018 18:03:44 +0100 Subject: [Python-ideas] In-place assignment for "boolean or"? In-Reply-To: References: Message-ID: <6ee9ce4d-3331-8eea-6eab-51b5a54e67aa@btinternet.com> On 26/03/2018 13:48, Cammil Taank wrote: > Hi, > > I find a common idiom in Python is: > > x = x or 'some other value' > > This is highly reminiscent of the problem inplace operators solve. > > Would it be a good idea to consider an inplace operator for this, perhaps: > > x or= 'some other value' > > ? > > Thanks, > > Cammil I have used the idiom sometimes: ??? Dir = Dir or os.curdir ??? FileSpec = FileSpec or '*' But I think using an or= operator looks bad and obscure.? And it's one more thing to learn when learning Python. Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Mon Mar 26 11:11:17 2018 From: guido at python.org (Guido van Rossum) Date: Mon, 26 Mar 2018 15:11:17 +0000 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On Mon, Mar 26, 2018 at 6:33 AM, Nick Coghlan wrote: > On 26 March 2018 at 01:50, Guido van Rossum wrote: > > In the PEP 572 threads there's some grumbling about class scopes. > > > > Here's a random brainstorm. How about we change the scoping rules so that > > only functions defined with 'def' (or inside one of those) cannot see the > > class scope, and comprehensions and lambdas treat the class scope as an > > outer scope so they can close over class variables? > > > > Not sure what should happen to nested classes, they should probably be > > lumped in with the 'def' scopes. > > I think it's mainly in comprehensions that folks may find the current > behaviour surprising, as that's the case where we define a function > and immediately call it, so it mostly looks like regular inline code > execution. > > Once explicitly delayed invocation is involved (lambdas, def > statements), folks seem to be more comfortable with the idea "Oh, of > course that's going to behave like a method at class scope". Generator > expressions also mostly get a pass, since the *iteration* is delayed, > even though the generator-iterator itself is created immediately. > > One possibility that has occurred to me (but I've never investigated > to check the technical feasibility) is to see whether we might be able > to define a notion of "implied arguments" for the implicit > comprehension and generator expression function definitions. > > The gist of that idea would be: > > - when walking the symbol table for a comprehension or genexp, keep > track of every variable name mentioned in that subtree which is not > resolved in that subtree (which we already have the machinery to do, > as it's needed for compiling functions in general) > - treat all those names as *implied arguments* to the implicit > function, and call it with the right references in the right positions > > While we'd likely still want to keep evaluation of the outermost > iterator outside the nested scope for the sake of better genexp > tracebacks, that would still be enough to enable cases like: > > class C: > y = 1 > values = [x+y for x in range(10)] > > The trick would be to make it so that that comprehension gets expanded as: > > def _listcomp(_outermost_iter, y): > result = [] > for x in _outermost_iter: > result.append(x+y) > return result > > _listcomp_result = _listcomp(range(10), y) > > One of the more attractive aspects of this possibility (assuming it > can be made to work) is that it isn't even a special case in the > runtime name lookup rules - it's just changing the definition of the > implicit functions that we create for comprehensions and generator > expressions to make sure that they can refer to *any* name that > resolves in the namespace where they're called (even > invisible-to-the-compiler names like those injected via __prepare__ > methods, or assignments via locals() at class scope). > > Cheers, > Nick. > > P.S. I suspect we'd also find that this served as a performance > optimisation, since it wouldn't only prebind class locals, it would > prebind references to builtins and module globals as well. > Honestly that sounds way too complex. In my ideal world, only functions defined with 'def' are methods whose __get__ auto-binds the first argument (unless @classmethod or @staticmethod is present), and do not get to "see" the class scope. This is how you define methods after all. Lambdas used at class scope do not get this benefit, and they see the class scope as just another closure; ditto for comprehensions and genexprs. I don't think that the "is the call delayed" idea is the proper way to distinguish the two cases here. IMO the key concept is "is it used to define a method". I betcha if you see a lambda here it's because there's some calculation going on whose result will be stored as a class variable. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Mon Mar 26 18:49:01 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Tue, 27 Mar 2018 09:49:01 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: <20180326224901.GJ16661@ando.pearwood.info> On Mon, Mar 26, 2018 at 03:33:52PM +0100, Rhodri James wrote: > While my crystal ball is cloudy, I can well imagine beginners becoming > very confused over which symbol to use in which circumstance, and a lot > of swearing when: > > x := f() > if (y = g(x)) is not None: > h(y) > > results in syntax errors. I remember as a beginner being terribly confused when writing dicts and constantly writing {key=value}. It is part of the learning process, and while we shouldn't intentionally make things harder for beginners just for the sake of making it harder, we shouldn't necessarily give them veto over new features :-) (I must admit that even now, if I'm tired and distracted I occasionally make this same mistake.) But we also have the opportunity to make things easier for them. I presume that the syntax error could diagnose the error and tell them how to fix it: SyntaxError: cannot use := in a stand-alone statement, use = SyntaxError: cannot use = in an assignment expression, use := or similar. Problem solved. -- Steve From brenbarn at brenbarn.net Mon Mar 26 19:09:39 2018 From: brenbarn at brenbarn.net (BrenBarn) Date: Mon, 26 Mar 2018 16:09:39 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180326111823.GF16661@ando.pearwood.info> References: <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> <20180326111823.GF16661@ando.pearwood.info> Message-ID: <5aab245f-9b74-46c7-51f2-4917eed0fee8@brenbarn.net> On 03/26/2018 04:18 AM, Steven D'Aprano wrote: > On Mon, Mar 26, 2018 at 11:14:43AM +0300, Kirill Balunov wrote: >> Hi Chris, would you mind to add this syntactic form `(expr -> var)` to >> alternative syntax section, with the same semantics as `(expr as var)`. It >> seems to me that I've seen this form previously in some thread (can't find >> where), but it does not appear in alt. syntax section. > > That was probably my response to Nick: > > https://mail.python.org/pipermail/python-ideas/2018-March/049472.html > > I compared four possible choices: > > target = default if (expression as name) is None else name > target = default if (name := expression) is None else name > target = default if (expression -> name) is None else name > target = default if (name <- expression) is None else name > > > The two arrow assignment operators <- and -> are both taken from R. > > If we go down the sublocal scope path, which I'm not too keen on, then > Nick's earlier comments convince me that we should avoid "as". In that > case, my preferences are: > > (best) -> := <- as (worst) > > If we just bind to regular locals, then my preferences are: > > (best) as -> := <- (worst) > > Preferences are subject to change :-) Obviously we're bikeshedding here, but personally I detest these kinds of operators. To me - is a minus sign and < and > are less-than and greater-than. Trying to re-use these characters in ways that depend on their visual form strikes me as very ugly. <= makes sense because its meaning is a combination of the *meanings* of < and =, but <- as assignment is not a combination of the meanings of < and -. If we need a two-character operator, I prefer something like := that doesn't try to be a picture. From tim.peters at gmail.com Tue Mar 27 00:02:32 2018 From: tim.peters at gmail.com (Tim Peters) Date: Mon, 26 Mar 2018 23:02:32 -0500 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: [Guido] > ... > Not so fast. There's a perfectly reasonable alternative to sublocal scopes > -- just let it assign to a local variable in the containing scope. That's > the same as what Python does for for-loop variables. That's certainly what I would _expect_ if I never read the docs, conditioned by experience with Python's `for` and embedded assignments in at least C and Icon. But I have to confess I already gave up trying to stay up-to-date with all of Python's _current_ scope rules. It's not what I want to think about. It's easier all around to try not to reuse names in clever ways to begin with. From ncoghlan at gmail.com Tue Mar 27 09:56:27 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Tue, 27 Mar 2018 23:56:27 +1000 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On 27 March 2018 at 01:11, Guido van Rossum wrote: > Honestly that sounds way too complex. In my ideal world, only functions > defined with 'def' are methods whose __get__ auto-binds the first argument > (unless @classmethod or @staticmethod is present), and do not get to "see" > the class scope. This is how you define methods after all. Lambdas used at > class scope do not get this benefit, and they see the class scope as just > another closure; ditto for comprehensions and genexprs. > > I don't think that the "is the call delayed" idea is the proper way to > distinguish the two cases here. IMO the key concept is "is it used to define > a method". I betcha if you see a lambda here it's because there's some > calculation going on whose result will be stored as a class variable. Aye, I don't disagree with that. The implicit functions used in the comprehension & generator expression cases are just potentially simpler to handle, as we don't care about their API signatures, which means we can freely pollute their APIs with eager name bindings if we choose to do so. Lambda expressions are different, since their public API matters, so we can't mess with it freely to pass in extra name references. What we potentially *could* do though is allow implicit additions to their __closure__ attributes in a way that isn't permitted for regular function definitions, such that in the following code: class C: x = 1 f = staticmethod(lambda: print(x)) we would do the following things differently for lambdas (vs def functions): 1. We'd pass "x" down as a candidate for nonlocal name resolution while compiling the lambda 2. If "x" did get referenced from a lambda, we'd start allowing non-optimised code objects to have a list of cell vars (the way functions already do), and define a new "STORE_CLASSDEREF" opcode (as the counterpart to LOAD_CLASSDEREF) What STORE_CLASSDEREF would need to do is write updates both to the current locals namespace (so the expectations of a class body are met) *and* to the nominated cell object (so any references from lambda expressions are updated). The biggest downside I'd see to this is that for an incredibly long time, we've pushed the view that "lambda expressions are just regular functions that don't know their own name". Making it so that lambdas can close over class attributes breaks that equivalence, and if we were to use the cell based approach I suggest above, the seams would be visible in the case where a lambda expression references an attribute that gets rebound after class creation: >>> C.f() 1 >>> C.x = 2 >>> C.f() # With a cell based approach, this wouldn't change 1 All of those potential complexities are specific to lambda functions though - since the implicit functions created for lambda expressions and generator expressions are called and then immediately thrown away, they don't need a persistent cell reference to the closed over variable name. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Tue Mar 27 10:00:32 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Wed, 28 Mar 2018 00:00:32 +1000 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On 27 March 2018 at 01:57, Guido van Rossum wrote: > On Mon, Mar 26, 2018 at 7:57 AM, Nick Coghlan wrote: >> By contrast, the sublocals idea strives to keep the *lifecycle* impact >> of naming a subexpression as negligible as possible - while a named >> subexpression might live a little longer than it used to as an >> anonymous subexpression (or substantially longer in the case of >> compound statement headers), it still wouldn't survive past the end of >> the statement where it appeared. > > > But this is not new: if you use a for-loop to initialize some class-level > structure you have the same problem. There is also a standard solution > (just 'del' it). Right, but that's annoying, too, and adds "Am I polluting a namespace I care about?" to something that would ideally be a purely statement local consideration (and currently is for comprehensions and generator expressions). Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From jsbueno at python.org.br Tue Mar 27 10:25:10 2018 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 27 Mar 2018 11:25:10 -0300 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: > > Honestly that sounds way too complex. In my ideal world, only functions > defined with 'def' are methods whose __get__ auto-binds the first argument > (unless @classmethod or @staticmethod is present), and do not get to "see" > the class scope. This is how you define methods after all. Lambdas used at > class scope do not get this benefit, and they see the class scope as just > another closure; ditto for comprehensions and genexprs. > > I don't think that the "is the call delayed" idea is the proper way to > distinguish the two cases here. IMO the key concept is "is it used to define > a method". I betcha if you see a lambda here it's because there's some > calculation going on whose result will be stored as a class variable. Nevertheless, treating lambdas differently from "def functions" in this context alone can be complicated. Even if not the most intuitive thing on writing code, everywhere else lambda s can't even be distinguished from "regular functions" once created. And I have created a lot of "one line methods" as lambdas over the years. (that is a lot of code that would be broke with no reason). Even calling "type" with a couple "lambda" expressionsin the namespace parameter is something very useful, and might be crippled if lambda's can't be methods at all (although that is a different context, I don't think things should be changed at this point) But there are two different things about lambdas in class bodies: - enabling a lambda to "see" the outer class scope: that is ok, as it would just make things simpler, and won't break existing code, as this ability did not exist before. - preventing a lambda to become a method though its `__get__`: this breakes stuff. As it is not incompatible with a lambda that can "see" the class scope, I see no reason for doing that (code might not work if a lambda with cell classes pointing to the class body, but my feeling is that it would just work correctly if such cells are permitted to start with) js -><- From jsbueno at python.org.br Tue Mar 27 10:32:22 2018 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 27 Mar 2018 11:32:22 -0300 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On 27 March 2018 at 10:56, Nick Coghlan wrote: > ... Making it so that lambdas > can close over class attributes breaks that equivalence, and if we > were to use the cell based approach I suggest above, the seams would > be visible in the case where a lambda expression references an > attribute that gets rebound after class creation: > > >>> C.f() > 1 > >>> C.x = 2 > >>> C.f() # With a cell based approach, this wouldn't change > 1 > Yes - but that would be the intention of the code beign written as in your example - class C: x = 1 f = staticmethod(lambda: print(x)) While, the classic behavior can be attained by doing: class C: x = 1 f = classmethod(lambda cls: print(cls.x)) And the behavior in both cases if one of no-surprises for me. For coders who don't have the mechanism of class creation in their mind, that could come as a surprise, but it is better than being inconsistent. TL;DR: +1 for your approach - I am just saying your perceived drawback is not one IMHO. > Cheers, > Nick. > > -- > Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ From ethan at stoneleaf.us Tue Mar 27 10:52:49 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Mar 2018 07:52:49 -0700 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5AB7D274.1020100@stoneleaf.us> References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> <5AB71EB3.5030308@stoneleaf.us> <5AB720A9.5050205@stoneleaf.us> <5AB7D274.1020100@stoneleaf.us> Message-ID: <5ABA5AC1.8090301@stoneleaf.us> On 03/25/2018 09:46 AM, Ethan Furman wrote: > On 03/24/2018 09:24 PM, Nick Coghlan wrote: > >> No, the fact that the expression defining the outermost iterable gets evaluated in the outer scope is behaviour that's >> explicitly tested for in the regression test suite. >> >> The language reference spells out that this is intentional for generator expressions, where it has the added benefit of >> reporting errors in the outermost iterable expression at the point where the genexp is defined, rather than at the point >> where it gets iterated over: https://docs.python.org/3/reference/expressions.html#generator-expressions >> >> Independently of the pragmatic "getting them to work sensibly at class scope" motivation, comprehensions inherit those >> semantics by way of the intended semantic equivalence between "[x for x in sequence]" and "list(x for x in sequence)". > > Thank you (everyone!) for your patience. > > Using the latest example from Angelico and myself: > > --> d = 9 > ... def func(_iter): > ... ... body of comprehension > --> func((d as e)) > > The sticking point is the `func((d as e))`, which to my mind should happen inside the comprehension, but needs to happen > outside -- and the reason it has to happen outside is so that the interpretor can verify that `d` actually exists; > however, I think we can have both: > > --> def func(_iter): > ... ... > ... e = d > ... ... > --> d > --> func() > > This way, the assignment does not leak, but the referenced name is still looked up and verified to exist outside the > function call. > > I don't know how easy/difficult that would be to implement. At this point a simple confirmation that I understand and > that in theory the above would solve both issues is what I'm looking for. Or, of course, the reasons why the above > would not, in theory, work. Any insights here? I would rather not say the same things in another thread if we can resolve this question here and be done with it. :) -- ~Ethan~ From klahnakoski at mozilla.com Tue Mar 27 10:56:16 2018 From: klahnakoski at mozilla.com (Kyle Lahnakoski) Date: Tue, 27 Mar 2018 10:56:16 -0400 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: Message-ID: <382ba443-7026-7009-db1c-d226285c8f46@mozilla.com> On 2018-03-23 06:01, Chris Angelico wrote: > https://www.python.org/dev/peps/pep-0572/ > A suggestion: Under the rejected "Special-casing comprehensions", you show "prefix-local-name-bindings": Name bindings that appear before the loop, like: > stuff = [(y, x/y) where y = f(x) for x in range(5)] Please add mention of rejecting "postfix-local-name-bindings": Name bindings that happen after the loop.? For example: > stuff = [(y, x/y) for x in range(5) where y = f(x)] Since all the same reasoning applies to both prefix and postfix variations, maybe distinguishing between prefix and postfix can be done in the last paragraph of "Special-casing comprehensions". Thank you. -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Mar 27 11:22:15 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Mar 2018 08:22:15 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: On Tue, Mar 27, 2018 at 7:00 AM, Nick Coghlan wrote: > On 27 March 2018 at 01:57, Guido van Rossum wrote: > > On Mon, Mar 26, 2018 at 7:57 AM, Nick Coghlan > wrote: > >> By contrast, the sublocals idea strives to keep the *lifecycle* impact > >> of naming a subexpression as negligible as possible - while a named > >> subexpression might live a little longer than it used to as an > >> anonymous subexpression (or substantially longer in the case of > >> compound statement headers), it still wouldn't survive past the end of > >> the statement where it appeared. > > > > > > But this is not new: if you use a for-loop to initialize some class-level > > structure you have the same problem. There is also a standard solution > > (just 'del' it). > > Right, but that's annoying, too, and adds "Am I polluting a namespace > I care about?" to something that would ideally be a purely statement > local consideration (and currently is for comprehensions and generator > expressions). > The standard reply here is that if you can't tell at a glance whether that's the case, your code is too complex. The Zen of Python says "Namespaces are one honking great idea -- let's do more of those!" and in this case that means refactor into smaller namespaces, i.e. functions/methods. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Mar 27 11:27:32 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Mar 2018 16:27:32 +0100 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On 27 March 2018 at 15:32, Joao S. O. Bueno wrote: > Yes - but that would be the intention of the code beign written as in > your example - > > class C: > x = 1 > f = staticmethod(lambda: print(x)) > > While, the classic behavior can be attained by doing: > > class C: > x = 1 > f = classmethod(lambda cls: print(cls.x)) > > And the behavior in both cases if one of no-surprises for me. For > coders who don't have the mechanism of class creation in their > mind, that could come as a surprise, but it is better than being > inconsistent. I wouldn't describe myself as "having the mechanism of class creation in my mind", but I'm not 100% sure that's a necessary criterion here. In an ideal world, Python's semantics is supposed to be intuitive, which means that understanding subtle details shouldn't be necessary to anticipate the behaviour of certain constructs. Looking at the following definitions: class C: x = 1 y1 = x y2 = (lambda: x)() def meth1(self): return self.x meth2 = lambda self: self.x @staticmethod def sm1(): return x sm2 = staticmethod(lambda: x) @classmethod def cm1(cls): return cls.x cm2 = classmethod(lambda cls: cls.x) I would expect meth1 and meth2 to be equivalent. I'd expect cm1 and cm2 to be equivalent. I would *not* expect sm1 to work, and I'd expect sm2 to fail for the same reason - namely that sm1 has no access to the class or an instance of it, so it should not be able to reference x. And so we should get NameError. These to me don't imply any sort of "understanding of how classes are created" - they simply need an understanding of how methods (normal, static and class) get access to the class/instance. They also require that you *don't* expect a class statement to create a scope that can be closed over. I didn't really think about that before I started analysing this code, but once I did, I realised that I've never expected that. So my reaction to a bare nonlocal variable reference in a method (whether defined in a def statement or as a lambda) would be "wait, what would that mean? I guess it's the global". I wouldn't even be looking at the x defined in the class at that point. The definition of y2 follows that rule as well - even though the fact that it's not defining a method, but it's "just" a bare lambda, slightly muddies the water. The odd one out is y1. I actually can't quite explain the logic that allows y1 to refer to the value of x, even though it's the most "natural" case. As I said, I don't think of a class as defining a new scope, rather I think of it as bundling together a set of statements that will be run to populate the class (maybe that's "having the mechanism of class creation in my mind"?). So I guess y1 = x is just normal statement sequencing. So I guess I'm saying that I don't really see a problem with the current behaviour here. Classes *don't* create a scope. So you don't close over class variables. For further emphasis of this point of view: @staticmethod def sm1(): nonlocal x return x gives nonlocal x ^ SyntaxError: no binding for nonlocal 'x' found which again I can understand as "there's no outer scope containing x". Having said all this, I guess you could say that I'm too close to the current behaviour to see its deficiencies. Maybe that's true. But I don't actually *use* any of this on a regular basis. I worked out all of the above based on intuition from how I understood Python's class model, plus a few small experiments that mostly just confirmed what I expected. If adding the ability to refer to a bare x in sm1/sm2 or y2 [1] means complicating the behaviour to the point where my mental model needs to involve injecting arguments into nested scopes, I'm a strong -1.If there's a need to "fix" comprehensions, then I'd much rather that we do so in a way that doesn't change the current behaviour or execution model. Specifically, I'd actually much *rather* that these 3 cases continue giving NameError. The comprehension cases: y3 = [x+i for i in (0,1,2)] #1 y4 = [1+i for i in (x, x+1)] #2 (#1 fails with NameError, and #2 works) strike me as odd corner cases that are a bit odd, but are not worth disrupting the normal cases for. And they should be viewed as oddities in how comprehensions look names up, and not as peculiarities of class scope. Sorry - I intended that to be an "it's all pretty simple and obvious" comment, but it turned into a rather long post. But I do think the basic idea remains simple. Paul [1] The other cases don't need to refer to a bare x, they already have perfectly good ways of accessing the class variable x. From jsbueno at python.org.br Tue Mar 27 11:51:30 2018 From: jsbueno at python.org.br (Joao S. O. Bueno) Date: Tue, 27 Mar 2018 12:51:30 -0300 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: Well, there is an idiom to "keep everything as is", and work around the current limitations: class A: def b(): x = 1 d = [i + x for i in range(2)] return locals() locals().update(b()) del b Maybe if we could find a syntactic sugar for this idiom (with an abuse of the `with` keyword, for example), people could be happy, with class bodies working by default as they are now, and enabling the reuse of defined members by using the special syntax - class A: with class: x = 1 d = [i + x for in range(2)] On 27 March 2018 at 12:27, Paul Moore wrote: > On 27 March 2018 at 15:32, Joao S. O. Bueno wrote: >> Yes - but that would be the intention of the code beign written as in >> your example - >> >> class C: >> x = 1 >> f = staticmethod(lambda: print(x)) >> >> While, the classic behavior can be attained by doing: >> >> class C: >> x = 1 >> f = classmethod(lambda cls: print(cls.x)) >> >> And the behavior in both cases if one of no-surprises for me. For >> coders who don't have the mechanism of class creation in their >> mind, that could come as a surprise, but it is better than being >> inconsistent. > > I wouldn't describe myself as "having the mechanism of class creation > in my mind", but I'm not 100% sure that's a necessary criterion here. > In an ideal world, Python's semantics is supposed to be intuitive, > which means that understanding subtle details shouldn't be necessary > to anticipate the behaviour of certain constructs. > > Looking at the following definitions: > > class C: > x = 1 > > y1 = x > y2 = (lambda: x)() > > def meth1(self): > return self.x > meth2 = lambda self: self.x > > @staticmethod > def sm1(): > return x > sm2 = staticmethod(lambda: x) > > @classmethod > def cm1(cls): > return cls.x > cm2 = classmethod(lambda cls: cls.x) > > I would expect meth1 and meth2 to be equivalent. I'd expect cm1 and > cm2 to be equivalent. I would *not* expect sm1 to work, and I'd expect > sm2 to fail for the same reason - namely that sm1 has no access to the > class or an instance of it, so it should not be able to reference x. > And so we should get NameError. > > These to me don't imply any sort of "understanding of how classes are > created" - they simply need an understanding of how methods (normal, > static and class) get access to the class/instance. They also require > that you *don't* expect a class statement to create a scope that can > be closed over. I didn't really think about that before I started > analysing this code, but once I did, I realised that I've never > expected that. > > So my reaction to a bare nonlocal variable reference in a method > (whether defined in a def statement or as a lambda) would be "wait, > what would that mean? I guess it's the global". I wouldn't even be > looking at the x defined in the class at that point. > > The definition of y2 follows that rule as well - even though the fact > that it's not defining a method, but it's "just" a bare lambda, > slightly muddies the water. > > The odd one out is y1. I actually can't quite explain the logic that > allows y1 to refer to the value of x, even though it's the most > "natural" case. As I said, I don't think of a class as defining a new > scope, rather I think of it as bundling together a set of statements > that will be run to populate the class (maybe that's "having the > mechanism of class creation in my mind"?). So I guess y1 = x is just > normal statement sequencing. > > So I guess I'm saying that I don't really see a problem with the > current behaviour here. Classes *don't* create a scope. So you don't > close over class variables. > > For further emphasis of this point of view: > > @staticmethod > def sm1(): > nonlocal x > return x > > gives > > nonlocal x > ^ > SyntaxError: no binding for nonlocal 'x' found > > which again I can understand as "there's no outer scope containing x". > > Having said all this, I guess you could say that I'm too close to the > current behaviour to see its deficiencies. Maybe that's true. But I > don't actually *use* any of this on a regular basis. I worked out all > of the above based on intuition from how I understood Python's class > model, plus a few small experiments that mostly just confirmed what I > expected. > > If adding the ability to refer to a bare x in sm1/sm2 or y2 [1] means > complicating the behaviour to the point where my mental model needs to > involve injecting arguments into nested scopes, I'm a strong -1.If > there's a need to "fix" comprehensions, then I'd much rather that we > do so in a way that doesn't change the current behaviour or execution > model. Specifically, I'd actually much *rather* that these 3 cases > continue giving NameError. > > The comprehension cases: > > y3 = [x+i for i in (0,1,2)] #1 > y4 = [1+i for i in (x, x+1)] #2 > > (#1 fails with NameError, and #2 works) strike me as odd corner cases > that are a bit odd, but are not worth disrupting the normal cases for. > And they should be viewed as oddities in how comprehensions look names > up, and not as peculiarities of class scope. > > Sorry - I intended that to be an "it's all pretty simple and obvious" > comment, but it turned into a rather long post. But I do think the > basic idea remains simple. > > Paul > > [1] The other cases don't need to refer to a bare x, they already have > perfectly good ways of accessing the class variable x. From p.f.moore at gmail.com Tue Mar 27 12:52:18 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Mar 2018 17:52:18 +0100 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On 27 March 2018 at 16:51, Joao S. O. Bueno wrote: > Well, there is an idiom to "keep everything as is", and work around > the current limitations: > > class A: > def b(): > x = 1 > d = [i + x for i in range(2)] > return locals() > locals().update(b()) > del b > > Maybe if we could find a syntactic sugar for this idiom > (with an abuse of the `with` keyword, for example), people could be happy, > with class bodies working by default as they are now, and enabling the > reuse of defined members by using the special syntax - > > class A: > with class: > x = 1 > d = [i + x for in range(2)] I'd actually like to see some real world use cases to get a feel for whether this is even worth worrying about. (I'm not saying it isn't, just that it's hard to get a feel for the importance based on artificial examples). BTW, for an alternative workaround that avoids nested scopes altogether: >>> import operator >>> from functools import partial >>> class C: ... x = 1 ... d = list(map(partial(operator.add, x), range(2))) ... >>> c = C() >>> c.d [1, 2] No, I'm not suggesting that's clearer - but it does (at least in my mind) make it more obvious that the problem is with comprehensions, not with class scopes. Paul From guido at python.org Tue Mar 27 13:19:38 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Mar 2018 10:19:38 -0700 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On Tue, Mar 27, 2018 at 6:56 AM, Nick Coghlan wrote: > [...] The implicit functions used in the > comprehension & generator expression cases are just potentially > simpler to handle, as we don't care about their API signatures, which > means we can freely pollute their APIs with eager name bindings if we > choose to do so. [...] > Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope issues for comprehensions and genexprs. There may still be breakage, when the code defines a global x that is overridden by a class-level x, and a class-level comprehension references x assuming it to be the global. So we need to tread carefully even here -- but this case is weird already: x = 42 class C: x = [1, 2, 3] z = [x+y for y in x] # [43, 44, 45] -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From ericfahlgren at gmail.com Tue Mar 27 13:28:59 2018 From: ericfahlgren at gmail.com (Eric Fahlgren) Date: Tue, 27 Mar 2018 10:28:59 -0700 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On Tue, Mar 27, 2018 at 9:52 AM, Paul Moore wrote: > I'd actually like to see some real world use cases to get a feel for > whether this is even worth worrying about. (I'm not saying it isn't, > just that it's hard to get a feel for the importance based on > artificial examples). > ?The only reason I brought it up was because I ran into this about three weeks ago porting some old Python 2 to 3, where there was a class level comprehension that referenced a class variable on the lhs. I simply removed it by enumerating the cases by hand, no big deal, but it did take me a while to figure out why ?that no longer worked. My specific case looked approximately like this: class Plugin: plugin_dir = 'somepath' plugin_names = [os.join(plugin_dir, name) for name in ('list', 'of', 'names')] -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Tue Mar 27 14:12:46 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 27 Mar 2018 19:12:46 +0100 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On 27 March 2018 at 18:19, Guido van Rossum wrote: > On Tue, Mar 27, 2018 at 6:56 AM, Nick Coghlan wrote: > >> [...] The implicit functions used in the >> comprehension & generator expression cases are just potentially >> simpler to handle, as we don't care about their API signatures, which >> means we can freely pollute their APIs with eager name bindings if we >> choose to do so. [...] >> > > Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope > issues for comprehensions and genexprs. > Removing the implicit function scope in comprehensions is something I wanted for long time. It would not only "fix" the scoping, but will also fix the yield inside comprehensions. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From ethan at stoneleaf.us Tue Mar 27 14:43:54 2018 From: ethan at stoneleaf.us (Ethan Furman) Date: Tue, 27 Mar 2018 11:43:54 -0700 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: <5ABA90EA.10001@stoneleaf.us> On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote: > On 27 March 2018 at 18:19, Guido van Rossum wrote: >> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope issues for comprehensions and genexprs. > > Removing the implicit function scope in comprehensions is something I wanted for long time. > It would not only "fix" the scoping, but will also fix the yield inside comprehensions. Can we do it without leaking names? -- ~Ethan~ From guido at python.org Tue Mar 27 14:45:54 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Mar 2018 11:45:54 -0700 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: <5ABA90EA.10001@stoneleaf.us> References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On Tue, Mar 27, 2018 at 11:43 AM, Ethan Furman wrote: > On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote: > >> On 27 March 2018 at 18:19, Guido van Rossum wrote: >> > > Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope >>> issues for comprehensions and genexprs. >>> >> >> Removing the implicit function scope in comprehensions is something I >> wanted for long time. >> It would not only "fix" the scoping, but will also fix the yield inside >> comprehensions. >> > > Can we do it without leaking names? > Assuming you're concerned about leaking names out of the comprehension into the class scope, that shouldn't be a problem. (The solution actually involves leaking names *into* the comprehension scope, but I'm not sure that should be called "leaking". :-) I do notice that we probably can't easily solve this using the existing closure machinery ("cells") because if we were to have cells in the class dict, it would confuse everything else that looks in the class namespace. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From p.f.moore at gmail.com Tue Mar 27 14:47:55 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Tue, 27 Mar 2018 19:47:55 +0100 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: <5ABA90EA.10001@stoneleaf.us> References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On 27 March 2018 at 19:43, Ethan Furman wrote: > On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote: >> >> On 27 March 2018 at 18:19, Guido van Rossum wrote: > >>> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope >>> issues for comprehensions and genexprs. >> >> >> Removing the implicit function scope in comprehensions is something I >> wanted for long time. >> It would not only "fix" the scoping, but will also fix the yield inside >> comprehensions. > > Can we do it without leaking names? To me, that would be the ideal. I assume there are significant technical challenges, though, as otherwise I'd have thought that would have been the approach taken when Python 3 fixed the name leaking issue from Python 2. Paul From levkivskyi at gmail.com Tue Mar 27 14:51:16 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 27 Mar 2018 19:51:16 +0100 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: <5ABA90EA.10001@stoneleaf.us> References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On 27 March 2018 at 19:43, Ethan Furman wrote: > On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote: > >> On 27 March 2018 at 18:19, Guido van Rossum wrote: >> > > Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope >>> issues for comprehensions and genexprs. >>> >> >> Removing the implicit function scope in comprehensions is something I >> wanted for long time. >> It would not only "fix" the scoping, but will also fix the yield inside >> comprehensions. >> > > Can we do it without leaking names? > > If you mean this [i for i in range(5)] i # NameError then yes, this is possible. Serhiy outlined the implementation few moths ago. The rough idea is to use automatic re-naming. The only problem with this is that if someone will step into debugger one will see a name like .0.i instead of i. But this can be solved in the debuggers. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From levkivskyi at gmail.com Tue Mar 27 14:53:20 2018 From: levkivskyi at gmail.com (Ivan Levkivskyi) Date: Tue, 27 Mar 2018 19:53:20 +0100 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On 27 March 2018 at 19:47, Paul Moore wrote: > On 27 March 2018 at 19:43, Ethan Furman wrote: > > On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote: > >> > >> On 27 March 2018 at 18:19, Guido van Rossum wrote: > > > >>> Hm, so maybe we shouldn't touch lambda, but we can at least fix the > scope > >>> issues for comprehensions and genexprs. > >> > >> > >> Removing the implicit function scope in comprehensions is something I > >> wanted for long time. > >> It would not only "fix" the scoping, but will also fix the yield inside > >> comprehensions. > > > > Can we do it without leaking names? > > To me, that would be the ideal. I assume there are significant > technical challenges, though, as otherwise I'd have thought that would > have been the approach taken when Python 3 fixed the name leaking > issue from Python 2. > > Yes, this will be certainly a big PR, but if we agree to do this, I volunteer to make the PR. -- Ivan -------------- next part -------------- An HTML attachment was scrubbed... URL: From guido at python.org Tue Mar 27 15:10:48 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Mar 2018 12:10:48 -0700 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On Tue, Mar 27, 2018 at 11:51 AM, Ivan Levkivskyi wrote: > On 27 March 2018 at 19:43, Ethan Furman wrote: > >> On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote: >> >>> On 27 March 2018 at 18:19, Guido van Rossum wrote: >>> >> >> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope >>>> issues for comprehensions and genexprs. >>>> >>> >>> Removing the implicit function scope in comprehensions is something I >>> wanted for long time. >>> It would not only "fix" the scoping, but will also fix the yield inside >>> comprehensions. >>> >> >> Can we do it without leaking names? >> >> > If you mean this > > [i for i in range(5)] > > i # NameError > > then yes, this is possible. Serhiy outlined the implementation few moths > ago. The rough idea is to use automatic re-naming. > The only problem with this is that if someone will step into debugger one > will see a name like .0.i instead of i. > But this can be solved in the debuggers. > Oh, sorry, I misread what you were talking about. You're proposing going back to the Python 2 shared namespace. I'm not at all excited about that, and I'm not convinced that adjusting debuggers to hide the name mangling is effective -- it does nothing about other forms of introspection. Also depending on how PEP 572 falls there may be assignments in there. Plus there may be object lifetime consequences. -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From greg.ewing at canterbury.ac.nz Tue Mar 27 17:28:01 2018 From: greg.ewing at canterbury.ac.nz (Greg Ewing) Date: Wed, 28 Mar 2018 10:28:01 +1300 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: <5ABA90EA.10001@stoneleaf.us> Message-ID: <5ABAB761.1090105@canterbury.ac.nz> Guido van Rossum wrote: > I do notice that we probably can't easily solve this using the existing > closure machinery ("cells") because if we were to have cells in the > class dict, it would confuse everything else that looks in the class > namespace. Maybe it could be done with a new kind of cell object that holds a reference to the class dict and an attribute name. -- Greg From ctaank at gmail.com Tue Mar 27 18:45:53 2018 From: ctaank at gmail.com (Cammil Taank) Date: Tue, 27 Mar 2018 23:45:53 +0100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! Message-ID: > > ... From here, > the most important concern and question is: Is there any other syntax > or related proposal that ought to be mentioned here? I am not sure if this is valid, but perhaps this is an alternative syntax which might be simpler: ``name! expr`` So for example, instead of: stuff = [[(f(x) as y), x/y] for x in range(5)] stuff = [[y! f(x), x/y] for x in range(5)] As far as I can tell there would be no conflicts with the current uses of "!". One potential source of ambiguity would be in: x = y! a + b # should y be a or (a + b)? I think this is solved by requiring the target expression to be non-greedy. If you want a larger named expression, you can always use parenthesis. i.e. ``x = y! (z + z)`` I feel brevity and minimised punctuation are important for the adoption of statement-locals, and personally I feel it reads well. I also happen to prefer the name preceding the expression, though I suspect this is quite subjective. Also, apologies if I have grossly misunderstood something. Cammil -------------- next part -------------- An HTML attachment was scrubbed... URL: From rob.cliffe at btinternet.com Tue Mar 27 19:08:24 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 28 Mar 2018 00:08:24 +0100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: <7c10e67c-54ff-ac2f-d7d1-dceb4d8e6ed2@btinternet.com> On 27/03/2018 16:22, Guido van Rossum wrote: > On Tue, Mar 27, 2018 at 7:00 AM, Nick Coghlan > wrote: > > On 27 March 2018 at 01:57, Guido van Rossum > wrote: > > On Mon, Mar 26, 2018 at 7:57 AM, Nick Coghlan > > wrote: > >> By contrast, the sublocals idea strives to keep the *lifecycle* impact > >> of naming a subexpression as negligible as possible - while a named > >> subexpression might live a little longer than it used to as an > >> anonymous subexpression (or substantially longer in the case of > >> compound statement headers), it still wouldn't survive past the > end of > >> the statement where it appeared. > > > > > > But this is not new: if you use a for-loop to initialize some > class-level > > structure? you have the same problem. There is also a standard > solution > > (just 'del' it). > > Right, but that's annoying, too, and adds "Am I polluting a namespace > I care about?" to something that would ideally be a purely statement > local consideration (and currently is for comprehensions and generator > expressions). > > > The standard reply here is that if you can't tell at a glance whether > that's the case, your code is too complex. The Zen of Python says > "Namespaces are one honking great idea -- let's do more of those!" and > in this case that means refactor into smaller namespaces, i.e. > functions/methods. > This is not always satisfactory.? If your for-loop uses 20 already-defined-locals, do you want to refactor it into a function with 20 parameters? Regards Rob Cliffe -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Tue Mar 27 20:19:21 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Wed, 28 Mar 2018 11:19:21 +1100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <7c10e67c-54ff-ac2f-d7d1-dceb4d8e6ed2@btinternet.com> References: <7c10e67c-54ff-ac2f-d7d1-dceb4d8e6ed2@btinternet.com> Message-ID: <20180328001920.GK16661@ando.pearwood.info> On Wed, Mar 28, 2018 at 12:08:24AM +0100, Rob Cliffe via Python-ideas wrote: > > On 27/03/2018 16:22, Guido van Rossum wrote: > >The standard reply here is that if you can't tell at a glance whether > >that's the case, your code is too complex. The Zen of Python says > >"Namespaces are one honking great idea -- let's do more of those!" and > >in this case that means refactor into smaller namespaces, i.e. > >functions/methods. > > > This is not always satisfactory.? If your for-loop uses 20 > already-defined-locals, do you want to refactor it into a function with > 20 parameters? The standard reply here is that if your for-loop needs 20 locals, your function is horribly over-complex and you may need to rethink your design. And if you don't think "20 locals" is too many, okay, how about 50? 100? 1000? At some point we'll all agree that the function is too complex. We don't have an obligation to solve every problem of excess complexity, especially when the nominal solution involves adding complexity elsewhere. For 25 years, the solution to complex functions in Python has been to refactor or simplify them. That strategy has worked well in practice, not withstanding your hypothetical function. If you genuinely do have a function that is so highly coupled with so many locals that it is hard to refactor, then you have my sympathy but we have no obligation to add a band-aid for it to the language. Putting the loop variable in its own scope doesn't do anything about the real problem: you have a loop that needs to work with twenty other local variables. Any other modification to the loop will run into the same problem: you have to check the rest of the function to ensure you're not clobbering one of the twenty other variables. Special-casing the loop variable seems hardly justified. If there is a justification for introducing sub-local scoping, then I think it needs to be something better than pathologically over-complex functions. -- Steve From rob.cliffe at btinternet.com Tue Mar 27 20:40:30 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Wed, 28 Mar 2018 01:40:30 +0100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <20180328001920.GK16661@ando.pearwood.info> References: <7c10e67c-54ff-ac2f-d7d1-dceb4d8e6ed2@btinternet.com> <20180328001920.GK16661@ando.pearwood.info> Message-ID: On 28/03/2018 01:19, Steven D'Aprano wrote: > On Wed, Mar 28, 2018 at 12:08:24AM +0100, Rob Cliffe via Python-ideas wrote: >> On 27/03/2018 16:22, Guido van Rossum wrote: >>> The standard reply here is that if you can't tell at a glance whether >>> that's the case, your code is too complex. The Zen of Python says >>> "Namespaces are one honking great idea -- let's do more of those!" and >>> in this case that means refactor into smaller namespaces, i.e. >>> functions/methods. >>> >> This is not always satisfactory.? If your for-loop uses 20 >> already-defined-locals, do you want to refactor it into a function with >> 20 parameters? > The standard reply here is that if your for-loop needs 20 locals, your > function is horribly over-complex and you may need to rethink your > design. > > And if you don't think "20 locals" is too many, okay, how about 50? 100? > 1000? At some point we'll all agree that the function is too complex. > > We don't have an obligation to solve every problem of excess complexity, > especially when the nominal solution involves adding complexity > elsewhere. > > For 25 years, the solution to complex functions in Python has been to > refactor or simplify them. That strategy has worked well in practice, > not withstanding your hypothetical function. > > If you genuinely do have a function that is so highly coupled with so > many locals that it is hard to refactor, then you have my sympathy but > we have no obligation to add a band-aid for it to the language. It's a fact of life that some tasks *are* complicated.? I daresay most aren't, or don't need to be, but some are. > > Putting the loop variable in its own scope doesn't do anything about the > real problem: you have a loop that needs to work with twenty other local > variables. Any other modification to the loop will run into the same > problem: you have to check the rest of the function to ensure you're not > clobbering one of the twenty other variables. Special-casing the loop > variable seems hardly justified. > > If there is a justification for introducing sub-local scoping, then I > think it needs to be something better than pathologically over-complex > functions. > > But putting the loop variable in its own scope solves one problem: it ensures that the variable is confined to that loop, and you don't have to worry about whether a variable of the same name occurs elsewhere in your function.? In other words it increases local transparency (I'm not sure that's the right phrase, but I'm struggling to bring a more appropriate one to mind) and hence increases readability. (I understand your point about being able to inspect the for-loop variable after the for-loop has terminated - I've probably done it myself - but it's a matter of opinion whether that convenience outweighs the cleanliness of confining the for-variable's scope.) Regards Rob Cliffe From guido at python.org Tue Mar 27 22:15:40 2018 From: guido at python.org (Guido van Rossum) Date: Tue, 27 Mar 2018 19:15:40 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: <7c10e67c-54ff-ac2f-d7d1-dceb4d8e6ed2@btinternet.com> <20180328001920.GK16661@ando.pearwood.info> Message-ID: This thread is dead. On Tue, Mar 27, 2018 at 5:40 PM, Rob Cliffe via Python-ideas < python-ideas at python.org> wrote: > > > On 28/03/2018 01:19, Steven D'Aprano wrote: > >> On Wed, Mar 28, 2018 at 12:08:24AM +0100, Rob Cliffe via Python-ideas >> wrote: >> >>> On 27/03/2018 16:22, Guido van Rossum wrote: >>> >>>> The standard reply here is that if you can't tell at a glance whether >>>> that's the case, your code is too complex. The Zen of Python says >>>> "Namespaces are one honking great idea -- let's do more of those!" and >>>> in this case that means refactor into smaller namespaces, i.e. >>>> functions/methods. >>>> >>>> This is not always satisfactory. If your for-loop uses 20 >>> already-defined-locals, do you want to refactor it into a function with >>> 20 parameters? >>> >> The standard reply here is that if your for-loop needs 20 locals, your >> function is horribly over-complex and you may need to rethink your >> design. >> >> And if you don't think "20 locals" is too many, okay, how about 50? 100? >> 1000? At some point we'll all agree that the function is too complex. >> >> We don't have an obligation to solve every problem of excess complexity, >> especially when the nominal solution involves adding complexity >> elsewhere. >> >> For 25 years, the solution to complex functions in Python has been to >> refactor or simplify them. That strategy has worked well in practice, >> not withstanding your hypothetical function. >> >> If you genuinely do have a function that is so highly coupled with so >> many locals that it is hard to refactor, then you have my sympathy but >> we have no obligation to add a band-aid for it to the language. >> > It's a fact of life that some tasks *are* complicated. I daresay most > aren't, or don't need to be, but some are. > >> >> Putting the loop variable in its own scope doesn't do anything about the >> real problem: you have a loop that needs to work with twenty other local >> variables. Any other modification to the loop will run into the same >> problem: you have to check the rest of the function to ensure you're not >> clobbering one of the twenty other variables. Special-casing the loop >> variable seems hardly justified. >> >> If there is a justification for introducing sub-local scoping, then I >> think it needs to be something better than pathologically over-complex >> functions. >> >> >> But putting the loop variable in its own scope solves one problem: it > ensures that the variable is confined to that loop, and you don't have to > worry about whether a variable of the same name occurs elsewhere in your > function. In other words it increases local transparency (I'm not sure > that's the right phrase, but I'm struggling to bring a more appropriate one > to mind) and hence increases readability. > (I understand your point about being able to inspect the for-loop variable > after the for-loop has terminated - I've probably done it myself - but it's > a matter of opinion whether that convenience outweighs the cleanliness of > confining the for-variable's scope.) > Regards > Rob Cliffe > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From brenbarn at brenbarn.net Tue Mar 27 22:17:18 2018 From: brenbarn at brenbarn.net (BrenBarn) Date: Tue, 27 Mar 2018 19:17:18 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: References: Message-ID: <4e76176c-394a-0f8b-b3d9-3792967cd397@brenbarn.net> On 03/23/2018 03:01 AM, Chris Angelico wrote: > Apologies for letting this languish; life has an annoying habit of > getting in the way now and then. My simple response to all of this is that it's not worth it. Each new example convinces me more and more that in almost every case, sublocal assignments DECREASE readability as long as they occur inline. If the statement is very simple, the sublocal assignments make it complex. If it is complex, they do not aid in seeing parallelism between different pieces that reuse the same value, because the sublocal assignment itself creates an asymmetry. The only alternatives that I see as increasing readability are the "rejected" alternatives in which the sublocal assignment is moved "out of order" so that all references to it look the same and are separated from the (single) assignment --- i.e., the variants of the form "x = a+b with a='foo', b='bar'". (I think someone already mentioned this, but these variants, even if rejected, probably shouldn't be placed under the header of "special-casing comprehensions". Extracting the assignment to a with-clause makes sense outside of comprehensions too. It would make more sense to label them as "out of order" or "non-inline" or perhaps "cleft assignment", by analogy with cleft constructions in natural language.) From erik.m.bray at gmail.com Wed Mar 28 13:22:50 2018 From: erik.m.bray at gmail.com (Erik Bray) Date: Wed, 28 Mar 2018 19:22:50 +0200 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: <20180323112554.3957d0b1@fsol> References: <5AB41364.8010500@UGent.be> <9f7967f961e041ed9458543be77c0338@xmail102.UGent.be> <5AB49DDD.8040405@UGent.be> <20180323112554.3957d0b1@fsol> Message-ID: On Fri, Mar 23, 2018 at 11:25 AM, Antoine Pitrou wrote: > On Fri, 23 Mar 2018 07:25:33 +0100 > Jeroen Demeyer wrote: > >> On 2018-03-23 00:36, Antoine Pitrou wrote: >> > It does make sense, since the proposal sounds ambitious (and perhaps >> > impossible without breaking compatibility). >> >> Well, *some* breakage of backwards compatibility will be unavoidable. >> >> >> My plan (just a plan for now!) is to preserve backwards compatibility in >> the following ways: >> >> * Existing Python attributes of functions/methods should continue to >> exist and behave the same >> >> * The inspect module should give the same results as now (by changing >> the implementation of some of the functions in inspect to match the new >> classes) >> >> * Everything from the documented Python/C API. >> >> >> This means that I might break compatibility in the following ways: >> >> * Changing the classes of functions/methods (this is the whole point of >> this PEP). So anything involving isinstance() checks might break. >> >> * The undocumented parts of the Python/C API, in particular the C structure. > > One breaking change would be to add __get__ to C functions. This means > e.g. the following: > > class MyClass: > my_open = open > > would make my_open a MyClass method, therefore you would need to spell > it: > > class MyClass: > my_open = staticmethod(open) > > ... if you wanted MyClass().my_open('some file') to continue to work. > > Of course that might be considered a minor annoyance. I don't really see your point in this example. For one: why would anyone do this? Is this based on a real example? 2) That's how any function works. If you put some arbitrary function in a class body, and it's not able to accept an instance of that class as its first argument, then it will always be broken unless you make it a staticmethod. I don't see how there should be any difference there if the function were implemented in Python or in C. Thanks, E From solipsis at pitrou.net Wed Mar 28 13:28:03 2018 From: solipsis at pitrou.net (Antoine Pitrou) Date: Wed, 28 Mar 2018 19:28:03 +0200 Subject: [Python-ideas] PEP proposal: unifying function/method classes References: <5AB41364.8010500@UGent.be> <9f7967f961e041ed9458543be77c0338@xmail102.UGent.be> <5AB49DDD.8040405@UGent.be> <20180323112554.3957d0b1@fsol> Message-ID: <20180328192803.67293baf@fsol> On Wed, 28 Mar 2018 19:22:50 +0200 Erik Bray wrote: > > I don't really see your point in this example. For one: why would > anyone do this? Is this based on a real example? Yes, I remember doing that when writing unit tests for open() and open()-like functions. You would write a base test case: class TestBase: open_func = None def test_something(self): self.open_func(...) # etc. and then instantiate concrete subclasses: class BuiltinOpenTest(TestBase, unittest.TestCase): open_func = open class PyOpenTest(TestBase, unittest.TestCase): open_func = staticmethod(_pyio.open) > 2) That's how any > function works. If you put some arbitrary function in a class body, > and it's not able to accept an instance of that class as its first > argument, then it will always be broken unless you make it a > staticmethod. I don't see how there should be any difference there if > the function were implemented in Python or in C. I agree that there *should* be no difference. The point is, there is currently a difference and suppressing it may break compatibility. I don't think the breakage is important enough to stop us from streamlining things here. Just pointing out the risk exists. Regards Antoine. From storchaka at gmail.com Wed Mar 28 13:40:44 2018 From: storchaka at gmail.com (Serhiy Storchaka) Date: Wed, 28 Mar 2018 20:40:44 +0300 Subject: [Python-ideas] PEP proposal: unifying function/method classes In-Reply-To: References: <5AB41364.8010500@UGent.be> <9f7967f961e041ed9458543be77c0338@xmail102.UGent.be> <5AB49DDD.8040405@UGent.be> <20180323112554.3957d0b1@fsol> Message-ID: 28.03.18 20:22, Erik Bray ????: > On Fri, Mar 23, 2018 at 11:25 AM, Antoine Pitrou wrote: >> One breaking change would be to add __get__ to C functions. This means >> e.g. the following: >> >> class MyClass: >> my_open = open >> >> would make my_open a MyClass method, therefore you would need to spell >> it: >> >> class MyClass: >> my_open = staticmethod(open) >> >> ... if you wanted MyClass().my_open('some file') to continue to work. >> >> Of course that might be considered a minor annoyance. > > I don't really see your point in this example. For one: why would > anyone do this? Is this based on a real example? 2) That's how any > function works. If you put some arbitrary function in a class body, > and it's not able to accept an instance of that class as its first > argument, then it will always be broken unless you make it a > staticmethod. I don't see how there should be any difference there if > the function were implemented in Python or in C. Yes, there are real examples. Good or bad, it is well known behavior of builtin functions, and real code depends on it. For sure changing it will break a lot of user code (and also the Python stdlib and tests). Just try to implements the support of the descriptor protocol in builtin functions and you will see how much code this will break. From j.van.dorp at deonet.nl Thu Mar 29 05:43:20 2018 From: j.van.dorp at deonet.nl (Jacco van Dorp) Date: Thu, 29 Mar 2018 11:43:20 +0200 Subject: [Python-ideas] threading lock and isinstance Message-ID: Currently, you cannot use isinstance checks on threading locks, because they're created by a factory function instead of being actual classes. Now that we've got __subclasshook__ and __instancecheck__, is there still a reason other than "history" that we can't use isinstance here ? There could exist a thin wrapper class along the lines of: class Lock: def __new__(): return __allocate_lock() # Call factory function @classmethod def __subclasshook__(cls, inst): # code... As far as I can think, no code would be broken by this - even current introspection which we're trying to replace would work just fine. My C experience is rather limited, so i dont know if it's hard to write the subclass /isinstance checks. While probably not that important, would people consider this to be a good idea ? (I found this bug report: https://bugs.python.org/issue3352 which has a post of Nick Coghlan from 2008-07-14 22:25 where this is mentioned - but in respect to multiprocessing. However, that thread is rather old and dead. I could not find any other references to this.) From contact at brice.xyz Thu Mar 29 06:19:35 2018 From: contact at brice.xyz (Brice Parent) Date: Thu, 29 Mar 2018 12:19:35 +0200 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: > Hm, so maybe we shouldn't touch lambda, but we can at least fix the > scope issues for comprehensions and genexprs. > > There may still be breakage, when the code defines a global x that is > overridden by a class-level x, and a class-level comprehension > references x assuming it to be the global. So we need to tread > carefully even here -- but this case is weird already: > > x = 42 > class C: > ??? x = [1, 2, 3] > ??? z = [x+y for y in x]? # [43, 44, 45] > Wow! I had to try it myself! If I had came across something like the following in a code review : x = [1, 2] class C: ??? x = [3, 4, 5] ??? z = [x for _ in x] I would have expected C.z to equal either `[[1, 2], [1, 2]]` or `[[3, 4, 5], [3, 4, 5], [3, 4, 5]]`, but surely not `[[1, 2], [1, 2], [1, 2]]`! Is that intentional, or the result of other way-more-logical decisions? - Brice -------------- next part -------------- An HTML attachment was scrubbed... URL: From ncoghlan at gmail.com Thu Mar 29 10:55:33 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2018 00:55:33 +1000 Subject: [Python-ideas] PEP 572 version 2: Statement-Local Name Bindings In-Reply-To: <5ABA5AC1.8090301@stoneleaf.us> References: <87370qn43i.fsf@grothesque.org> <5AB5C14E.4030603@stoneleaf.us> <5AB71C02.7000702@stoneleaf.us> <5AB71EB3.5030308@stoneleaf.us> <5AB720A9.5050205@stoneleaf.us> <5AB7D274.1020100@stoneleaf.us> <5ABA5AC1.8090301@stoneleaf.us> Message-ID: On 28 March 2018 at 00:52, Ethan Furman wrote: > On 03/25/2018 09:46 AM, Ethan Furman wrote: >> >> On 03/24/2018 09:24 PM, Nick Coghlan wrote: >> >>> No, the fact that the expression defining the outermost iterable gets >>> evaluated in the outer scope is behaviour that's >>> explicitly tested for in the regression test suite. >>> >>> The language reference spells out that this is intentional for generator >>> expressions, where it has the added benefit of >>> reporting errors in the outermost iterable expression at the point where >>> the genexp is defined, rather than at the point >>> where it gets iterated over: >>> https://docs.python.org/3/reference/expressions.html#generator-expressions >>> >>> Independently of the pragmatic "getting them to work sensibly at class >>> scope" motivation, comprehensions inherit those >>> semantics by way of the intended semantic equivalence between "[x for x >>> in sequence]" and "list(x for x in sequence)". >> >> >> Thank you (everyone!) for your patience. >> >> Using the latest example from Angelico and myself: >> >> --> d = 9 >> ... def func(_iter): >> ... ... body of comprehension >> --> func((d as e)) >> >> The sticking point is the `func((d as e))`, which to my mind should happen >> inside the comprehension, but needs to happen >> outside -- and the reason it has to happen outside is so that the >> interpretor can verify that `d` actually exists; >> however, I think we can have both: >> >> --> def func(_iter): >> ... ... >> ... e = d >> ... ... >> --> d >> --> func() >> >> This way, the assignment does not leak, but the referenced name is still >> looked up and verified to exist outside the >> function call. >> >> I don't know how easy/difficult that would be to implement. At this point >> a simple confirmation that I understand and >> that in theory the above would solve both issues is what I'm looking for. >> Or, of course, the reasons why the above >> would not, in theory, work. > > > Any insights here? I would rather not say the same things in another thread > if we can resolve this question here and be done with it. :) Just what Chris said earlier: the expression defining the outermost iterable can be arbitrarily complex, and what you're suggesting would only be feasible for a top-level name binding, not for nested ones. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Mar 29 11:12:13 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2018 01:12:13 +1000 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: Message-ID: On 28 March 2018 at 03:19, Guido van Rossum wrote: > On Tue, Mar 27, 2018 at 6:56 AM, Nick Coghlan wrote: >> >> [...] The implicit functions used in the >> comprehension & generator expression cases are just potentially >> simpler to handle, as we don't care about their API signatures, which >> means we can freely pollute their APIs with eager name bindings if we >> choose to do so. [...] > > > Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope > issues for comprehensions and genexprs. Yeah, the "immediately call the function and throw it away" aspect makes them much easier to deal with, and they're also the case that currently feels weird to me (since the nested function is *supposed* to be a hidden implementation detail, but it's existence currently leaks through here). I don't have that niggle with lambdas, since they're visibly defining a new nested scope, so having them behave like method definitions doesn't feel strange. > There may still be breakage, when the code defines a global x that is > overridden by a class-level x, and a class-level comprehension references x > assuming it to be the global. So we need to tread carefully even here -- but > this case is weird already: > > x = 42 > class C: > x = [1, 2, 3] > z = [x+y for y in x] # [43, 44, 45] Hmm, potentially more concerning might be cases where methods are currently ignored, but in a future Python start shadowing builtins for class level comprehensions and genexps. We're getting to the level of "rare case" (methods shadowing builtins) intersecting with "rare case" (using a comprehension or genexp at class scope) there though, so it should be feasible to handle through a regular deprecation cycle. So I'd guess this would need a future import ("from __future__ import implicit_scopes", perhaps? I'm struggling to come up with a good name, since it's essentially "implicit comprehension and generator expression nested scopes, version two"), and an associated deprecation warning for free variable references that would start resolving differently. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Thu Mar 29 11:27:10 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2018 01:27:10 +1000 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On 28 March 2018 at 04:47, Paul Moore wrote: > On 27 March 2018 at 19:43, Ethan Furman wrote: >> On 03/27/2018 11:12 AM, Ivan Levkivskyi wrote: >>> >>> On 27 March 2018 at 18:19, Guido van Rossum wrote: >> >>>> Hm, so maybe we shouldn't touch lambda, but we can at least fix the scope >>>> issues for comprehensions and genexprs. >>> >>> >>> Removing the implicit function scope in comprehensions is something I >>> wanted for long time. >>> It would not only "fix" the scoping, but will also fix the yield inside >>> comprehensions. >> >> Can we do it without leaking names? > > To me, that would be the ideal. I assume there are significant > technical challenges, though, as otherwise I'd have thought that would > have been the approach taken when Python 3 fixed the name leaking > issue from Python 2. It isn't avoiding the names leaking that's particularly challenging (there are several viable ways to do that), it's making sure that inner scopes can still see them. For example: lazy_and_eager = [((lambda: i), (lambda i=i: i)) for i in range(3)] for lazy, eager in lazy_and_eager: print(lazy(), eager()) # prints: 2 0 2 1 2 2 While it's *technically* feasible to hide "i" from the outer scope without hiding it from inner scopes with a variable renaming based approach, you end up having to design and document a separate lexical scoping mechanism that's distinct from the one that functions already use. I really didn't want to do that, so I proposed just using the existing scoping mechanism instead, and while that has definitely had its quirks, I'm still happy enough with it a decade later to call it a successful approach. Cheers, Nick. P.S. This is also a contributing factor to one of the aspects of the sublocals proposal: disallowing closing over them means that a renaming based approach *can* be used to keep them separate from regular local variable names. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From julia.hiyeon.kim at gmail.com Thu Mar 29 11:42:01 2018 From: julia.hiyeon.kim at gmail.com (Julia Kim) Date: Thu, 29 Mar 2018 08:42:01 -0700 Subject: [Python-ideas] Sets, Dictionaries Message-ID: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> My suggestion is to change the syntax for creating an empty set and an empty dictionary as following. an_empty_set = {} an_empty_dictionary = {:} Compatibility issues could be resolved with a program which takes a Python program (codes) as a text and edits it. Sent from my iPhone From rosuav at gmail.com Thu Mar 29 11:48:32 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 30 Mar 2018 02:48:32 +1100 Subject: [Python-ideas] Sets, Dictionaries In-Reply-To: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> References: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> Message-ID: On Fri, Mar 30, 2018 at 2:42 AM, Julia Kim wrote: > My suggestion is to change the syntax for creating an empty set and an empty dictionary as following. > > an_empty_set = {} > an_empty_dictionary = {:} > > > Compatibility issues could be resolved with a program which takes a Python program (codes) as a text and edits it. > Unfortunately, that's not sufficient for backward compatibility. Imagine reading something on Stack Overflow that has this line of code in it: words = {} Does that make an empty set or an empty dict? How would anyone know? Are you going to go through every piece of code on Stack Overflow and change it? What about all the blogs out there? Printed books? Simply transforming code doesn't work. Having the same syntax perform drastically different things on different versions of the interpreter is almost certainly not going to fly. The only way that this might be usable is if you use a __future__ directive. And if it were done that way, I would expect most code out there to NOT use the directive, and therefore to keep today's semantics - which means the change effectively hasn't happened for most people. ChrisA From stephanh42 at gmail.com Thu Mar 29 12:00:41 2018 From: stephanh42 at gmail.com (Stephan Houben) Date: Thu, 29 Mar 2018 16:00:41 +0000 Subject: [Python-ideas] Sets, Dictionaries In-Reply-To: References: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> Message-ID: Perhaps one day we will be able to use ? for the empty set. That would actually match conventional notation. Note that this is not valid syntax today (not a legal Unicode identifier). Stephan Op do 29 mrt. 2018 17:49 schreef Chris Angelico : > On Fri, Mar 30, 2018 at 2:42 AM, Julia Kim > wrote: > > My suggestion is to change the syntax for creating an empty set and an > empty dictionary as following. > > > > an_empty_set = {} > > an_empty_dictionary = {:} > > > > > > Compatibility issues could be resolved with a program which takes a > Python program (codes) as a text and edits it. > > > > Unfortunately, that's not sufficient for backward compatibility. > Imagine reading something on Stack Overflow that has this line of code > in it: > > words = {} > > Does that make an empty set or an empty dict? How would anyone know? > Are you going to go through every piece of code on Stack Overflow and > change it? What about all the blogs out there? Printed books? > > Simply transforming code doesn't work. Having the same syntax perform > drastically different things on different versions of the interpreter > is almost certainly not going to fly. > > The only way that this might be usable is if you use a __future__ > directive. And if it were done that way, I would expect most code out > there to NOT use the directive, and therefore to keep today's > semantics - which means the change effectively hasn't happened for > most people. > > ChrisA > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From clint.hepner at gmail.com Thu Mar 29 12:04:15 2018 From: clint.hepner at gmail.com (Clint Hepner) Date: Thu, 29 Mar 2018 12:04:15 -0400 Subject: [Python-ideas] Sets, Dictionaries In-Reply-To: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> References: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> Message-ID: > On 2018 Mar 29 , at 11:42 a, Julia Kim wrote: > > My suggestion is to change the syntax for creating an empty set and an empty dictionary as following. > > an_empty_set = {} > an_empty_dictionary = {:} If you are willing to accept {:} as an empty dict, then surely {,} would suffice as an empty set, with no backwards compatibility issues at all. Also, this is also not a new idea (https://mail.python.org/pipermail/python-3000/2006-April/001286.html). I don't know if this was never seriously considered further, or if it was formally rejected. > > Compatibility issues could be resolved with a program which takes a Python program (codes) as a text and edits it. It's not that simple. This would require changing the vast majority of Python scripts ever written, including code which has never and will never care about `set` objects. Depending on the setting would require version-control check-ins and probably code review. Is cleaning up a minor divergence from mathematical notation really worth that kind of churn? -- Clint From rosuav at gmail.com Thu Mar 29 12:06:02 2018 From: rosuav at gmail.com (Chris Angelico) Date: Fri, 30 Mar 2018 03:06:02 +1100 Subject: [Python-ideas] Sets, Dictionaries In-Reply-To: References: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> Message-ID: On Fri, Mar 30, 2018 at 3:00 AM, Stephan Houben wrote: > Perhaps one day we will be able to use > > ? > > for the empty set. > That would actually match conventional notation. > Maybe, but that symbol generally means the unique immutable empty set in mathematics, so a closer equivalent would be frozenset(), in the same way that () gives you a singleton immutable empty tuple. But yes, I would like to some day have a literal for an empty set. Just not "{}". ChrisA From p.f.moore at gmail.com Thu Mar 29 12:53:13 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Thu, 29 Mar 2018 17:53:13 +0100 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On 29 March 2018 at 16:27, Nick Coghlan wrote: > On 28 March 2018 at 04:47, Paul Moore wrote: >> To me, that would be the ideal. I assume there are significant >> technical challenges, though, as otherwise I'd have thought that would >> have been the approach taken when Python 3 fixed the name leaking >> issue from Python 2. > > It isn't avoiding the names leaking that's particularly challenging > (there are several viable ways to do that), it's making sure that > inner scopes can still see them. For example: > > lazy_and_eager = [((lambda: i), (lambda i=i: i)) for i in range(3)] > for lazy, eager in lazy_and_eager: > print(lazy(), eager()) > > # prints: > 2 0 > 2 1 > 2 2 I'm really not sure what point you're trying to make here - are you saying that this is good or bad? Correct or incorrect? I don't really have any intuition about what's going on here, so I'd just have to work it out in terms of the defined scoping rules. And I'd then tell whoever wrote it to rewrite it more clearly ;-) Maybe a real-life case where this was important would clarify what counts as intuitive here - but as it stands,I don't really care. I don't even care that much about compatibility. Unless someone were to come along and demonstrate a serious breakage in their code, I think it's perfectly OK to change the behaviour in this situation, if that's what you're suggesting (with suitable deprecation, of course). > While it's *technically* feasible to hide "i" from the outer scope > without hiding it from inner scopes with a variable renaming based > approach, you end up having to design and document a separate lexical > scoping mechanism that's distinct from the one that functions already > use. I really didn't want to do that, so I proposed just using the > existing scoping mechanism instead, and while that has definitely had > its quirks, I'm still happy enough with it a decade later to call it a > successful approach. I do think the current implementation is a pretty good compromise. I'd be reasonably OK with not changing anything in this area. But this discussion was prompted by some of the debates around statement local variables, so "not changing anything" includes, in my mind, "not trying to make statement local variables work" as they interact badly with the current scoping behaviour in this area. Paul From breamoreboy at gmail.com Thu Mar 29 13:14:32 2018 From: breamoreboy at gmail.com (Mark Lawrence) Date: Thu, 29 Mar 2018 18:14:32 +0100 Subject: [Python-ideas] Sets, Dictionaries In-Reply-To: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> References: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> Message-ID: On 29/03/18 16:42, Julia Kim wrote: > My suggestion is to change the syntax for creating an empty set and an empty dictionary as following. > > an_empty_set = {} > an_empty_dictionary = {:} > > > Compatibility issues could be resolved with a program which takes a Python program (codes) as a text and edits it. > There are two chances of this happening, zero or none. Others have all ready explained why. -- My fellow Pythonistas, ask not what our language can do for you, ask what you can do for our language. Mark Lawrence From guido at python.org Thu Mar 29 14:48:56 2018 From: guido at python.org (Guido van Rossum) Date: Thu, 29 Mar 2018 11:48:56 -0700 Subject: [Python-ideas] threading lock and isinstance In-Reply-To: References: Message-ID: You should probably first explain your use case -- why is it important to your code to be able to use isinstance() on locks? On Thu, Mar 29, 2018 at 2:43 AM, Jacco van Dorp wrote: > Currently, you cannot use isinstance checks on threading locks, because > they're created by a factory function instead of being actual classes. > > Now that we've got __subclasshook__ and __instancecheck__, is there > still a reason other than "history" that we can't use isinstance here ? > There > could exist a thin wrapper class along the lines of: > > class Lock: > def __new__(): > return __allocate_lock() # Call factory function > > @classmethod > def __subclasshook__(cls, inst): > # code... > > As far as I can think, no code would be broken by this - even current > introspection > which we're trying to replace would work just fine. > > My C experience is rather limited, so i dont know if it's hard to > write the subclass > /isinstance checks. > > While probably not that important, would people consider this to be a > good idea ? > > (I found this bug report: https://bugs.python.org/issue3352 which has > a post of Nick Coghlan from 2008-07-14 22:25 where this is mentioned > - but in respect to multiprocessing. However, that thread is rather > old and dead. I could not find any other references to this.) > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From chris.barker at noaa.gov Thu Mar 29 15:06:11 2018 From: chris.barker at noaa.gov (Chris Barker - NOAA Federal) Date: Thu, 29 Mar 2018 15:06:11 -0400 Subject: [Python-ideas] Sets, Dictionaries In-Reply-To: References: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> Message-ID: > There are two chances of this happening, zero or none. That would be the empty set, yes? ;-) -CHB From clint.hepner at gmail.com Thu Mar 29 15:52:17 2018 From: clint.hepner at gmail.com (Clint Hepner) Date: Thu, 29 Mar 2018 15:52:17 -0400 Subject: [Python-ideas] Sets, Dictionaries In-Reply-To: References: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> Message-ID: > On 2018 Mar 29 , at 12:06 p, Chris Angelico wrote: > > On Fri, Mar 30, 2018 at 3:00 AM, Stephan Houben wrote: >> Perhaps one day we will be able to use >> >> ? >> >> for the empty set. >> That would actually match conventional notation. >> > > Maybe, but that symbol generally means the unique immutable empty set > in mathematics, so a closer equivalent would be frozenset(), in the > same way that () gives you a singleton immutable empty tuple. But yes, > I would like to some day have a literal for an empty set. Just not > "{}". > Mathematically, {1,2,3} is a unique, immutable set, too. {1,2,3} \union {4} doesn't modify {1,2,3}; it's just another way to represent another unique immutable set {1,2,3,4}. So I don't seen any problem (aside from the general resistance Unicode) with using ? to mean set(). However, just as string literals take a variety of prefixes (r, u, b, f), why can't set/dict literals? d{} == {} == dict() d{"a": 2} == {"a": 2} == dict(a=2) s{1,2,3} == {1,2,3} == set([1,2,3]) s{} == set() f{} == frozenset() f{1,2,3} == frozenset({1,2,3}) (I vaguely recall hearing a proposal to use o{...} for ordered dicts, so maybe this has already been considered. The only information I can find on set literals, though, assume {...} as a foregone conclusion, with various wrapped symbols like {/}, {@}, and {:} suggested to complement {}.) If we did use ? for an empty set, then f? could be the empty frozen set. -- Clint From ncoghlan at gmail.com Fri Mar 30 01:17:23 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2018 15:17:23 +1000 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On 30 March 2018 at 02:53, Paul Moore wrote: > On 29 March 2018 at 16:27, Nick Coghlan wrote: >> On 28 March 2018 at 04:47, Paul Moore wrote: >>> To me, that would be the ideal. I assume there are significant >>> technical challenges, though, as otherwise I'd have thought that would >>> have been the approach taken when Python 3 fixed the name leaking >>> issue from Python 2. >> >> It isn't avoiding the names leaking that's particularly challenging >> (there are several viable ways to do that), it's making sure that >> inner scopes can still see them. For example: >> >> lazy_and_eager = [((lambda: i), (lambda i=i: i)) for i in range(3)] >> for lazy, eager in lazy_and_eager: >> print(lazy(), eager()) >> >> # prints: >> 2 0 >> 2 1 >> 2 2 > > I'm really not sure what point you're trying to make here - are you > saying that this is good or bad? Correct or incorrect? I don't really > have any intuition about what's going on here, so I'd just have to > work it out in terms of the defined scoping rules. In this context, it's neither good nor bad, it just is :) > And I'd then tell > whoever wrote it to rewrite it more clearly ;-) Aye, we get to do that in a code review, but the compiler doesn't - it has to figure out how to make it do something at least reasonably defensible. > Maybe a real-life case where this was important would clarify what > counts as intuitive here - but as it stands,I don't really care. The fact that deep nesting of lexical scopes within an expression is almost always going to be an unreadable mess in practice is one of the reasons I don't think there's a strong case for either backwards compatibility breaks *or* significant increases in the overall semantic complexity. Any sensible coding style is already going to say "Don't do that, it's too hard to read", so we're mainly caring about it at all based on our own senses of engineering aesthetics and a general dislike of implementation details leaking through as user visible semantic differences. >> While it's *technically* feasible to hide "i" from the outer scope >> without hiding it from inner scopes with a variable renaming based >> approach, you end up having to design and document a separate lexical >> scoping mechanism that's distinct from the one that functions already >> use. I really didn't want to do that, so I proposed just using the >> existing scoping mechanism instead, and while that has definitely had >> its quirks, I'm still happy enough with it a decade later to call it a >> successful approach. > > I do think the current implementation is a pretty good compromise. I'd > be reasonably OK with not changing anything in this area. But this > discussion was prompted by some of the debates around statement local > variables, so "not changing anything" includes, in my mind, "not > trying to make statement local variables work" as they interact badly > with the current scoping behaviour in this area. It's specifically lexical closures that lead to things getting weird, and that's a key rationale for PEP 572's current recommendation that statement locals be completely invisible to nested scopes. Trying to have a subscope that isn't visible to the rest of the function it's in, while still being visible to nested scopes defined within that subscope, seems to lead to sufficient conflicts in reference lifecycle and visibility that just using a real nested scope instead ends up being preferable. Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From ncoghlan at gmail.com Fri Mar 30 01:27:20 2018 From: ncoghlan at gmail.com (Nick Coghlan) Date: Fri, 30 Mar 2018 15:27:20 +1000 Subject: [Python-ideas] Sets, Dictionaries In-Reply-To: References: <7D28756D-A5A9-48A1-97D1-23E2C9E488D4@gmail.com> Message-ID: On 30 March 2018 at 02:04, Clint Hepner wrote: > >> On 2018 Mar 29 , at 11:42 a, Julia Kim wrote: >> >> My suggestion is to change the syntax for creating an empty set and an empty dictionary as following. >> >> an_empty_set = {} >> an_empty_dictionary = {:} > > If you are willing to accept {:} as an empty dict, then surely {,} would suffice as an empty set, with no backwards compatibility issues at all. > > Also, this is also not a new idea (https://mail.python.org/pipermail/python-3000/2006-April/001286.html). I don't know > if this was never seriously considered further, or if it was formally rejected. I believe the main concern is that where "set()" is easy to look up in documentation, "{,}" isn't all that obvious (and isn't easy to search for), so folks would have to learn it by rote. It would also lead to the question of "If {,} is allowed for empty sets, why don't we allow (,) for empty tuples, and [,] for empty lists?". That said, a case could potentially be made for "Disambiguating empty container notation" that allowed all four options: * Unambiguous empty dict: {:} * Syntactic empty set: {,} * Empty tuple with optional comma: (,) * Empty list with optional comma: [,] Cheers, Nick. -- Nick Coghlan | ncoghlan at gmail.com | Brisbane, Australia From p.f.moore at gmail.com Fri Mar 30 05:37:45 2018 From: p.f.moore at gmail.com (Paul Moore) Date: Fri, 30 Mar 2018 10:37:45 +0100 Subject: [Python-ideas] Fixing class scope brainstorm In-Reply-To: References: <5ABA90EA.10001@stoneleaf.us> Message-ID: On 30 March 2018 at 06:17, Nick Coghlan wrote: > The fact that deep nesting of lexical scopes within an expression is > almost always going to be an unreadable mess in practice is one of the > reasons I don't think there's a strong case for either backwards > compatibility breaks *or* significant increases in the overall > semantic complexity. > > Any sensible coding style is already going to say "Don't do that, it's > too hard to read", so we're mainly caring about it at all based on our > own senses of engineering aesthetics and a general dislike of > implementation details leaking through as user visible semantic > differences. OK, cool. That I can completely agree with :-) >> I do think the current implementation is a pretty good compromise. I'd >> be reasonably OK with not changing anything in this area. But this >> discussion was prompted by some of the debates around statement local >> variables, so "not changing anything" includes, in my mind, "not >> trying to make statement local variables work" as they interact badly >> with the current scoping behaviour in this area. > > It's specifically lexical closures that lead to things getting weird, > and that's a key rationale for PEP 572's current recommendation that > statement locals be completely invisible to nested scopes. Trying to > have a subscope that isn't visible to the rest of the function it's > in, while still being visible to nested scopes defined within that > subscope, seems to lead to sufficient conflicts in reference lifecycle > and visibility that just using a real nested scope instead ends up > being preferable. OK. I can accept your point that lexical closure weirdness is the reason for PEP 572 preferring to make statement locals ignore nested scopes. I'm not sure I fully agree, but I can accept it. My point is that having a construct that introduces new names but makes them invisible to nested scopes is *also* weird enough to be a problem. As a result, I think that PEP 572 faces a problem in that it's hit a problem for which it doesn't have a good solution. Whether that problem is bad enough to kill the PEP is a matter of judgement. In my view: * The benefits of the PEP are marginal * We haven't yet established a suitably readable syntax (IMO, := is probably the best suggestion so far, but it still has its problems) * I remain unconvinced that the PEP improves readability of comprehensions * Plus the above scoping issue So for me, the balance is in favour of keeping the status quo. If the scoping issue could be resolved, the remaining issues are much more obviously matters of taste and my view on the PEP would move to "not worth the effort" rather than against. Paul PS I should also add that the "just make them assignments" resolution to PEP 572's scope issues is the only suggestion so far that *doesn't* have scoping weirdnesses of its own (because it uses existing scoping mechanisms). The problem with that one is that it's prone to leaking in ways that I think people are uncomfortable with ("I think" because I don't have time right now to review the examples given to see for myself how compelling they are). From Nikolaus at rath.org Fri Mar 30 15:04:52 2018 From: Nikolaus at rath.org (Nikolaus Rath) Date: Fri, 30 Mar 2018 20:04:52 +0100 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: (Guido van Rossum's message of "Sun, 25 Mar 2018 18:24:30 -0700") References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> Message-ID: <87in9d2xm3.fsf@vostro.rath.org> On Mar 25 2018, Guido van Rossum wrote: > I gotta say I'm warming up to := in preference over 'as', *if* we're going > to do this at all (not a foregone conclusion at all). I'm surprised that no one has mentioned it yet, so as a quick datapoint: Go also uses := for assignment, so there's some precedent. Best, -Nikolaus -- GPG Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F ?Time flies like an arrow, fruit flies like a Banana.? From guido at python.org Fri Mar 30 16:34:49 2018 From: guido at python.org (Guido van Rossum) Date: Fri, 30 Mar 2018 13:34:49 -0700 Subject: [Python-ideas] PEP 572: Statement-Local Name Bindings, take three! In-Reply-To: <87in9d2xm3.fsf@vostro.rath.org> References: <20180323150058.GU16661@ando.pearwood.info> <20180324044102.GV16661@ando.pearwood.info> <20180324144432.GW16661@ando.pearwood.info> <27fccc82-8833-d1a5-a589-8d1358a3887a@btinternet.com> <5AB6A081.5010503@stoneleaf.us> <87in9d2xm3.fsf@vostro.rath.org> Message-ID: On Fri, Mar 30, 2018 at 12:04 PM, Nikolaus Rath wrote: > On Mar 25 2018, Guido van Rossum public.gmane.org> wrote: > > I gotta say I'm warming up to := in preference over 'as', *if* we're > going > > to do this at all (not a foregone conclusion at all). > > I'm surprised that no one has mentioned it yet, so as a quick datapoint: > Go also uses := for assignment, so there's some precedent. > It's irrelevant, because Go's solution for inline assignment is entirely different. (And there was no question that := is commonly used for assignment -- just look it up on Wikipedia.) -- --Guido van Rossum (python.org/~guido) -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Sat Mar 31 10:48:56 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 31 Mar 2018 16:48:56 +0200 Subject: [Python-ideas] PEP draft: Unifying function/method classes Message-ID: <5ABF9FD8.4030507@UGent.be> I have prepared a PEP draft for unifying function/method classes. You can find it at https://github.com/jdemeyer/PEP-functions This has not officially been submitted as PEP yet, I want to hear your comments first. Thanks, Jeroen. From m.raesener at gmail.com Sat Mar 31 10:50:03 2018 From: m.raesener at gmail.com (=?UTF-8?Q?Marius_R=C3=A4sener?=) Date: Sat, 31 Mar 2018 16:50:03 +0200 Subject: [Python-ideas] Dart like multi line strings identation Message-ID: Hey List, this is my very first approach to suggest a Python improvement I'd think worth discussing. At some point, maybe with Dart 2.0 or a little earlier, Dart is now supporting multiline strings with "proper" identation (tried, but I can't find the according docs at the moment. probably due to the rather large changes related to dart 2.0 and outdated docs.) What I have in mind is probably best described with an Example: print(""" I am a multiline String. """) the closing quote defines the "margin indentation" - so in this example all lines would get reduces by their leading 4 spaces, resulting in a "clean" and unintended string. anyways, if dart or not, doesn't matter - I like the Idea and I think python3.x could benefit from it. If that's possible at all :) I could also imagine that this "indentation cleanup" only is applied if the last quotes are on their own line? Might be too complicated though, I can't estimated or understand this... thx for reading, Marius -------------- next part -------------- An HTML attachment was scrubbed... URL: From rymg19 at gmail.com Sat Mar 31 11:06:25 2018 From: rymg19 at gmail.com (Ryan Gonzalez) Date: Sat, 31 Mar 2018 10:06:25 -0500 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: Message-ID: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> I have to admit, regardless of how practical this is, it would surely get rid of a ton of textwrap.dedent calls all over the place... On March 31, 2018 9:50:43 AM Marius R?sener wrote: Hey List, this is my very first approach to suggest a Python improvement I'd think worth discussing. At some point, maybe with Dart 2.0 or a little earlier, Dart is now supporting multiline strings with "proper" identation (tried, but I can't find the according docs at the moment. probably due to the rather large changes related to dart 2.0 and outdated docs.) What I have in mind is probably best described with an Example: print(""" I am a multiline String. """) the closing quote defines the "margin indentation" - so in this example all lines would get reduces by their leading 4 spaces, resulting in a "clean" and unintended string. anyways, if dart or not, doesn't matter - I like the Idea and I think python3.x could benefit from it. If that's possible at all :) I could also imagine that this "indentation cleanup" only is applied if the last quotes are on their own line? Might be too complicated though, I can't estimated or understand this... thx for reading, Marius _______________________________________________ Python-ideas mailing list Python-ideas at python.org https://mail.python.org/mailman/listinfo/python-ideas Code of Conduct: http://python.org/psf/codeofconduct/ -- Ryan (????) Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else https://refi64.com/ -------------- next part -------------- An HTML attachment was scrubbed... URL: From robertve92 at gmail.com Sat Mar 31 11:48:06 2018 From: robertve92 at gmail.com (Robert Vanden Eynde) Date: Sat, 31 Mar 2018 15:48:06 +0000 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> References: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: So yes, currently you just do : import textwrap print(textwrap.dedent(""" I am A Line """)) So you'd want a string litteral ? print(d""" I am A Line """) Le sam. 31 mars 2018 ? 17:06, Ryan Gonzalez a ?crit : > I have to admit, regardless of how practical this is, it would surely get > rid of a ton of textwrap.dedent calls all over the place... > > On March 31, 2018 9:50:43 AM Marius R?sener wrote: > >> Hey List, >> >> this is my very first approach to suggest a Python improvement I'd think >> worth discussing. >> >> At some point, maybe with Dart 2.0 or a little earlier, Dart is now >> supporting multiline strings with "proper" identation (tried, but I can't >> find the according docs at the moment. probably due to the rather large >> changes related to dart 2.0 and outdated docs.) >> >> What I have in mind is probably best described with an Example: >> >> print(""" >> I am a >> multiline >> String. >> """) >> >> the closing quote defines the "margin indentation" - so in this example >> all lines would get reduces by their leading 4 spaces, resulting in a >> "clean" and unintended string. >> >> anyways, if dart or not, doesn't matter - I like the Idea and I think >> python3.x could benefit from it. If that's possible at all :) >> >> I could also imagine that this "indentation cleanup" only is applied if >> the last quotes are on their own line? Might be too complicated though, I >> can't estimated or understand this... >> >> thx for reading, >> Marius >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > -- > Ryan (????) > Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else > https://refi64.com/ > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From steve at pearwood.info Sat Mar 31 12:09:05 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Apr 2018 02:09:05 +1000 Subject: [Python-ideas] PEP draft: Unifying function/method classes In-Reply-To: <5ABF9FD8.4030507@UGent.be> References: <5ABF9FD8.4030507@UGent.be> Message-ID: <20180331160904.GW16661@ando.pearwood.info> On Sat, Mar 31, 2018 at 04:48:56PM +0200, Jeroen Demeyer wrote: > I have prepared a PEP draft for unifying function/method classes. You > can find it at > > https://github.com/jdemeyer/PEP-functions > > This has not officially been submitted as PEP yet, I want to hear your > comments first. It seems like a huge amount of work compared to the motivation: allow functions written in C to support introspection. Quote: "In particular, it is currently not possible to implement a function efficiently in C (only built-in functions can do that) while still allowing introspection like inspect.getfullargspec or inspect.getsourcefile (only Python functions can do that)." Why isn't the answer to provide a hook to support introspection? In the Subclassing section, you say: "We disallow subclassing of builtin_function and method to allow fast isinstance checks for those types." Seems to me that if you want a fast, exact (no subclasses) check, you should use "type(obj) is Class" rather than isinstance. If the *only* reason to prohibit subclassing is to make isinstance a bit faster, I don't think that's a good enough reason. -- Steve From mertz at gnosis.cx Sat Mar 31 12:51:43 2018 From: mertz at gnosis.cx (David Mertz) Date: Sat, 31 Mar 2018 16:51:43 +0000 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: I can currently write: from textwrap import dedent as d print(d(""" I am A Line """)) It doesn't feel like these hypothetical d-strings are with new syntax. On Sat, Mar 31, 2018, 11:49 AM Robert Vanden Eynde wrote: > So yes, currently you just do : > > import textwrap > > print(textwrap.dedent(""" > I am > A Line > """)) > > So you'd want a string litteral ? > > print(d""" > I am > A Line > """) > > Le sam. 31 mars 2018 ? 17:06, Ryan Gonzalez a ?crit : > >> I have to admit, regardless of how practical this is, it would surely get >> rid of a ton of textwrap.dedent calls all over the place... >> >> On March 31, 2018 9:50:43 AM Marius R?sener wrote: >> >>> Hey List, >>> >>> this is my very first approach to suggest a Python improvement I'd think >>> worth discussing. >>> >>> At some point, maybe with Dart 2.0 or a little earlier, Dart is now >>> supporting multiline strings with "proper" identation (tried, but I can't >>> find the according docs at the moment. probably due to the rather large >>> changes related to dart 2.0 and outdated docs.) >>> >>> What I have in mind is probably best described with an Example: >>> >>> print(""" >>> I am a >>> multiline >>> String. >>> """) >>> >>> the closing quote defines the "margin indentation" - so in this example >>> all lines would get reduces by their leading 4 spaces, resulting in a >>> "clean" and unintended string. >>> >>> anyways, if dart or not, doesn't matter - I like the Idea and I think >>> python3.x could benefit from it. If that's possible at all :) >>> >>> I could also imagine that this "indentation cleanup" only is applied if >>> the last quotes are on their own line? Might be too complicated though, I >>> can't estimated or understand this... >>> >>> thx for reading, >>> Marius >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >> -- >> Ryan (????) >> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else >> https://refi64.com/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From J.Demeyer at UGent.be Sat Mar 31 12:58:19 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 31 Mar 2018 18:58:19 +0200 Subject: [Python-ideas] PEP draft: Unifying function/method classes In-Reply-To: <9481860f6a1f4518b53bb7c19a742c34@xmail102.UGent.be> References: <5ABF9FD8.4030507@UGent.be> <9481860f6a1f4518b53bb7c19a742c34@xmail102.UGent.be> Message-ID: <5ABFBE2B.3000308@UGent.be> On 2018-03-31 18:09, Steven D'Aprano wrote: > It seems like a huge amount of work What is a huge amount of work? Writing the PEP? Implementing the PEP? Using the PEP? Adapting existing Python code to the PEP? > Why isn't the answer to provide a hook to support introspection? That is a lot easier said than done. How would you do that? The problem is that multiple classes are involved, at least "builtin_function_or_method" and "method_descriptor". So you'll end up with at least 3 classes (those 2 plus the existing "function") supporting introspection. With my proposal, this is implemented only once. And do you expect users to implement those hooks or do you expect Python to do it? If you want to do it in Python, you'll need to add a new class anyway. That doesn't look simpler than my PEP. That being said, if there is a general consensus that this is the right thing to do, I'll go for it. However, I'm afraid that people will complain that I'm complicating functions in Python even more. Second: while introspection was the initial motivation, it does make sense to unify built-in functions and Python functions. For example: why are unbound methods just functions in Python classes but not in extension types? If you are going to do some reorganization of function classes anyway, you might as well do it properly. I claim that functions will be *easier* to work with in Python when my PEP is accepted, both for the end user as well as for the implementor of custom function classes. > Seems to me that if you want a fast, exact (no subclasses) check, you > should use "type(obj) is Class" rather than isinstance. If the *only* > reason to prohibit subclassing is to make isinstance a bit faster, > I don't think that's a good enough reason. I didn't really mean "isinstance" literally, I was mostly thinking of the C API. I agree that it's not clear. Do you happen to know why the existing function classes in Python disallow subclassing? I assumed that it was for exactly this reason. Jeroen. From m.raesener at gmail.com Sat Mar 31 13:00:58 2018 From: m.raesener at gmail.com (=?UTF-8?Q?Marius_R=C3=A4sener?=) Date: Sat, 31 Mar 2018 19:00:58 +0200 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: Hey Robert Not really, I don?t think another string literal would be nice. Also, to correct your example, it would have to look something like: print(d??? I am a Line ???) The Idea is to use the closing quotes to detect the indentation length so to speak... 2018-03-31 17:48 GMT+02:00 Robert Vanden Eynde : > So yes, currently you just do : > > import textwrap > > print(textwrap.dedent(""" > I am > A Line > """)) > > So you'd want a string litteral ? > > print(d""" > I am > A Line > """) > > Le sam. 31 mars 2018 ? 17:06, Ryan Gonzalez a ?crit : > >> I have to admit, regardless of how practical this is, it would surely get >> rid of a ton of textwrap.dedent calls all over the place... >> >> On March 31, 2018 9:50:43 AM Marius R?sener wrote: >> >>> Hey List, >>> >>> this is my very first approach to suggest a Python improvement I'd think >>> worth discussing. >>> >>> At some point, maybe with Dart 2.0 or a little earlier, Dart is now >>> supporting multiline strings with "proper" identation (tried, but I can't >>> find the according docs at the moment. probably due to the rather large >>> changes related to dart 2.0 and outdated docs.) >>> >>> What I have in mind is probably best described with an Example: >>> >>> print(""" >>> I am a >>> multiline >>> String. >>> """) >>> >>> the closing quote defines the "margin indentation" - so in this example >>> all lines would get reduces by their leading 4 spaces, resulting in a >>> "clean" and unintended string. >>> >>> anyways, if dart or not, doesn't matter - I like the Idea and I think >>> python3.x could benefit from it. If that's possible at all :) >>> >>> I could also imagine that this "indentation cleanup" only is applied if >>> the last quotes are on their own line? Might be too complicated though, I >>> can't estimated or understand this... >>> >>> thx for reading, >>> Marius >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >> -- >> Ryan (????) >> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else >> https://refi64.com/ >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.raesener at gmail.com Sat Mar 31 13:02:46 2018 From: m.raesener at gmail.com (=?UTF-8?Q?Marius_R=C3=A4sener?=) Date: Sat, 31 Mar 2018 19:02:46 +0200 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: Hey David, hm, that's actually a nice way to solve this too I guess, besides the additional import and "string literal". but as I answered to robert before (did it wrong with who to answer, correct it just now so the mailing-list has the answer, too) was, that I don't have a string literal in mind for this. Like I don't see a reason why this couldn't be the default thing for all string literals? again, the Idea is just to use the closing quotes to determine the indentation length ... 2018-03-31 18:51 GMT+02:00 David Mertz : > I can currently write: > > from textwrap import dedent as d > print(d(""" > I am > A Line > """)) > > It doesn't feel like these hypothetical d-strings are with new syntax. > > On Sat, Mar 31, 2018, 11:49 AM Robert Vanden Eynde > wrote: > >> So yes, currently you just do : >> >> import textwrap >> >> print(textwrap.dedent(""" >> I am >> A Line >> """)) >> >> So you'd want a string litteral ? >> >> print(d""" >> I am >> A Line >> """) >> >> Le sam. 31 mars 2018 ? 17:06, Ryan Gonzalez a ?crit : >> >>> I have to admit, regardless of how practical this is, it would surely >>> get rid of a ton of textwrap.dedent calls all over the place... >>> >>> On March 31, 2018 9:50:43 AM Marius R?sener >>> wrote: >>> >>>> Hey List, >>>> >>>> this is my very first approach to suggest a Python improvement I'd >>>> think worth discussing. >>>> >>>> At some point, maybe with Dart 2.0 or a little earlier, Dart is now >>>> supporting multiline strings with "proper" identation (tried, but I can't >>>> find the according docs at the moment. probably due to the rather large >>>> changes related to dart 2.0 and outdated docs.) >>>> >>>> What I have in mind is probably best described with an Example: >>>> >>>> print(""" >>>> I am a >>>> multiline >>>> String. >>>> """) >>>> >>>> the closing quote defines the "margin indentation" - so in this example >>>> all lines would get reduces by their leading 4 spaces, resulting in a >>>> "clean" and unintended string. >>>> >>>> anyways, if dart or not, doesn't matter - I like the Idea and I think >>>> python3.x could benefit from it. If that's possible at all :) >>>> >>>> I could also imagine that this "indentation cleanup" only is applied if >>>> the last quotes are on their own line? Might be too complicated though, I >>>> can't estimated or understand this... >>>> >>>> thx for reading, >>>> Marius >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>>> >>> -- >>> Ryan (????) >>> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else >>> https://refi64.com/ >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > > -------------- next part -------------- An HTML attachment was scrubbed... URL: From toddrjen at gmail.com Sat Mar 31 13:47:58 2018 From: toddrjen at gmail.com (Todd) Date: Sat, 31 Mar 2018 17:47:58 +0000 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: It would radically change the meaning of every existing multi-line string. That is an enormous backwards-compatibility break. It might work as a __future__ import, though. On Sat, Mar 31, 2018, 13:03 Marius R?sener wrote: > Hey David, > > hm, that's actually a nice way to solve this too I guess, besides the > additional import and "string literal". > > but as I answered to robert before (did it wrong with who to answer, > correct it just now so the mailing-list has the answer, too) was, that I > don't have a string literal in mind for this. > > Like I don't see a reason why this couldn't be the default thing for all > string literals? > > again, the Idea is just to use the closing quotes to determine the > indentation length ... > > 2018-03-31 18:51 GMT+02:00 David Mertz : > >> I can currently write: >> >> from textwrap import dedent as d >> print(d(""" >> I am >> A Line >> """)) >> >> It doesn't feel like these hypothetical d-strings are with new syntax. >> >> On Sat, Mar 31, 2018, 11:49 AM Robert Vanden Eynde >> wrote: >> >>> So yes, currently you just do : >>> >>> import textwrap >>> >>> print(textwrap.dedent(""" >>> I am >>> A Line >>> """)) >>> >>> So you'd want a string litteral ? >>> >>> print(d""" >>> I am >>> A Line >>> """) >>> >>> Le sam. 31 mars 2018 ? 17:06, Ryan Gonzalez a ?crit : >>> >>>> I have to admit, regardless of how practical this is, it would surely >>>> get rid of a ton of textwrap.dedent calls all over the place... >>>> >>>> On March 31, 2018 9:50:43 AM Marius R?sener >>>> wrote: >>>> >>>>> Hey List, >>>>> >>>>> this is my very first approach to suggest a Python improvement I'd >>>>> think worth discussing. >>>>> >>>>> At some point, maybe with Dart 2.0 or a little earlier, Dart is now >>>>> supporting multiline strings with "proper" identation (tried, but I can't >>>>> find the according docs at the moment. probably due to the rather large >>>>> changes related to dart 2.0 and outdated docs.) >>>>> >>>>> What I have in mind is probably best described with an Example: >>>>> >>>>> print(""" >>>>> I am a >>>>> multiline >>>>> String. >>>>> """) >>>>> >>>>> the closing quote defines the "margin indentation" - so in this >>>>> example all lines would get reduces by their leading 4 spaces, resulting in >>>>> a "clean" and unintended string. >>>>> >>>>> anyways, if dart or not, doesn't matter - I like the Idea and I think >>>>> python3.x could benefit from it. If that's possible at all :) >>>>> >>>>> I could also imagine that this "indentation cleanup" only is applied >>>>> if the last quotes are on their own line? Might be too complicated though, >>>>> I can't estimated or understand this... >>>>> >>>>> thx for reading, >>>>> Marius >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> >>>>> >>>> -- >>>> Ryan (????) >>>> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone else >>>> https://refi64.com/ >>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From m.raesener at gmail.com Sat Mar 31 14:14:08 2018 From: m.raesener at gmail.com (=?UTF-8?Q?Marius_R=C3=A4sener?=) Date: Sat, 31 Mar 2018 18:14:08 +0000 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: Oh, ok... yeah didn?t think of that. Except I guess I?d assume that so far multiline strings are either with textwrap or ?don?t care?? Maybe? But sure, with that in mind it gets more tricky Todd schrieb am Sa. 31. M?rz 2018 um 19:49: > It would radically change the meaning of every existing multi-line > string. That is an enormous backwards-compatibility break. It might work > as a __future__ import, though. > > > On Sat, Mar 31, 2018, 13:03 Marius R?sener wrote: > >> Hey David, >> >> hm, that's actually a nice way to solve this too I guess, besides the >> additional import and "string literal". >> >> but as I answered to robert before (did it wrong with who to answer, >> correct it just now so the mailing-list has the answer, too) was, that I >> don't have a string literal in mind for this. >> >> Like I don't see a reason why this couldn't be the default thing for all >> string literals? >> >> again, the Idea is just to use the closing quotes to determine the >> indentation length ... >> >> 2018-03-31 18:51 GMT+02:00 David Mertz : >> >>> I can currently write: >>> >>> from textwrap import dedent as d >>> print(d(""" >>> I am >>> A Line >>> """)) >>> >>> It doesn't feel like these hypothetical d-strings are with new syntax. >>> >>> On Sat, Mar 31, 2018, 11:49 AM Robert Vanden Eynde >>> wrote: >>> >>>> So yes, currently you just do : >>>> >>>> import textwrap >>>> >>>> print(textwrap.dedent(""" >>>> I am >>>> A Line >>>> """)) >>>> >>>> So you'd want a string litteral ? >>>> >>>> print(d""" >>>> I am >>>> A Line >>>> """) >>>> >>>> Le sam. 31 mars 2018 ? 17:06, Ryan Gonzalez a >>>> ?crit : >>>> >>>>> I have to admit, regardless of how practical this is, it would surely >>>>> get rid of a ton of textwrap.dedent calls all over the place... >>>>> >>>>> On March 31, 2018 9:50:43 AM Marius R?sener >>>>> wrote: >>>>> >>>>>> Hey List, >>>>>> >>>>>> this is my very first approach to suggest a Python improvement I'd >>>>>> think worth discussing. >>>>>> >>>>>> At some point, maybe with Dart 2.0 or a little earlier, Dart is now >>>>>> supporting multiline strings with "proper" identation (tried, but I can't >>>>>> find the according docs at the moment. probably due to the rather large >>>>>> changes related to dart 2.0 and outdated docs.) >>>>>> >>>>>> What I have in mind is probably best described with an Example: >>>>>> >>>>>> print(""" >>>>>> I am a >>>>>> multiline >>>>>> String. >>>>>> """) >>>>>> >>>>>> the closing quote defines the "margin indentation" - so in this >>>>>> example all lines would get reduces by their leading 4 spaces, resulting in >>>>>> a "clean" and unintended string. >>>>>> >>>>>> anyways, if dart or not, doesn't matter - I like the Idea and I think >>>>>> python3.x could benefit from it. If that's possible at all :) >>>>>> >>>>>> I could also imagine that this "indentation cleanup" only is applied >>>>>> if the last quotes are on their own line? Might be too complicated though, >>>>>> I can't estimated or understand this... >>>>>> >>>>>> thx for reading, >>>>>> Marius >>>>>> >>>>>> _______________________________________________ >>>>>> Python-ideas mailing list >>>>>> Python-ideas at python.org >>>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>>> >>>>>> >>>>> -- >>>>> Ryan (????) >>>>> Yoko Shimomura, ryo (supercell/EGOIST), Hiroyuki Sawano >> everyone >>>>> else >>>>> https://refi64.com/ >>>>> >>>>> _______________________________________________ >>>>> Python-ideas mailing list >>>>> Python-ideas at python.org >>>>> https://mail.python.org/mailman/listinfo/python-ideas >>>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>>> >>>> _______________________________________________ >>>> Python-ideas mailing list >>>> Python-ideas at python.org >>>> https://mail.python.org/mailman/listinfo/python-ideas >>>> Code of Conduct: http://python.org/psf/codeofconduct/ >>>> >>> >>> _______________________________________________ >>> Python-ideas mailing list >>> Python-ideas at python.org >>> https://mail.python.org/mailman/listinfo/python-ideas >>> Code of Conduct: http://python.org/psf/codeofconduct/ >>> >>> >> _______________________________________________ >> Python-ideas mailing list >> Python-ideas at python.org >> https://mail.python.org/mailman/listinfo/python-ideas >> Code of Conduct: http://python.org/psf/codeofconduct/ >> > _______________________________________________ > Python-ideas mailing list > Python-ideas at python.org > https://mail.python.org/mailman/listinfo/python-ideas > Code of Conduct: http://python.org/psf/codeofconduct/ > -------------- next part -------------- An HTML attachment was scrubbed... URL: From tjreedy at udel.edu Sat Mar 31 14:50:31 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 31 Mar 2018 14:50:31 -0400 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: Message-ID: On 3/31/2018 10:50 AM, Marius R?sener wrote: > What I have in mind is probably best described with an Example: > > print(""" > ? ? I am a > ? ? multiline > ? ? String. > ? ? """) > > the closing quote defines the "margin indentation" - so in this example > all lines would get reduces by their leading 4 spaces, resulting in a > "clean" and unintended string. Adding additional default processing to multiline strings is not possible within back-compatibility constraints. It is also not necessary. The current print("I am a\n" "multiline\n" "String.\n") does exactly the same thing as the proposal 2 fewer lines and is more flexible, as one can add an initial \n or an extra \n in the middle or omit the final \n. (For the example, print("I am a\nmultiline\nString\n", also works in 1 line, but does not represent the general case of multiples lone lines.) --- In 3.6, we introduced a new prefix, 'f', so there was no back compatibility issue. There was, however, a combinatorial explosion issue, as 'F' was also added (a mistake, I now think), and no order requirement (possibly another mistake). Hence stringprefix ::= "r" | "u" | "R" | "U" grew to stringprefix ::= "r" | "u" | "R" | "U" | "f" | "F" | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" | "RF" New unordered 'd' and 'D' prefixes, for 'dedent', applied to multiline strings only, would multiply the number of alternatives by about 5 and would require another rewrite of all code (Python or not) that parses Python code (such as in syntax colorizers). -- Terry Jan Reedy From tjreedy at udel.edu Sat Mar 31 15:12:52 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 31 Mar 2018 15:12:52 -0400 Subject: [Python-ideas] PEP draft: Unifying function/method classes In-Reply-To: <20180331160904.GW16661@ando.pearwood.info> References: <5ABF9FD8.4030507@UGent.be> <20180331160904.GW16661@ando.pearwood.info> Message-ID: On 3/31/2018 12:09 PM, Steven D'Aprano wrote: > On Sat, Mar 31, 2018 at 04:48:56PM +0200, Jeroen Demeyer wrote: >> I have prepared a PEP draft for unifying function/method classes. You >> can find it at >> >> https://github.com/jdemeyer/PEP-functions >> >> This has not officially been submitted as PEP yet, I want to hear your >> comments first. > > It seems like a huge amount of work compared to the motivation: allow > functions written in C to support introspection. Quote: > > "In particular, it is currently not possible to implement a function > efficiently in C (only built-in functions can do that) while still > allowing introspection like inspect.getfullargspec or inspect.signature, which superceded .getfullargspec, works with C-coded functions that use ArgumentClinic. >>> inspect.signature(list) >>> inspect.signature(list.append) >>> inspect.signature(int) # fails I would be all for more of the builtins and stdlib being converted. Can't 3rd-party C code use ArgumentClinic? > inspect.getsourcefile (only Python functions can do that)." Sourcefile is meaningless when there is no source file. C sources would be far less useful than Python sources. -- Terry Jan Reedy From J.Demeyer at UGent.be Sat Mar 31 15:21:42 2018 From: J.Demeyer at UGent.be (Jeroen Demeyer) Date: Sat, 31 Mar 2018 21:21:42 +0200 Subject: [Python-ideas] PEP draft: Unifying function/method classes In-Reply-To: <5443ff71b9754759a1a1aa09cebbef17@xmail102.UGent.be> References: <5ABF9FD8.4030507@UGent.be> <20180331160904.GW16661@ando.pearwood.info> <5443ff71b9754759a1a1aa09cebbef17@xmail102.UGent.be> Message-ID: <5ABFDFC6.7010406@UGent.be> On 2018-03-31 21:12, Terry Reedy wrote: > I would be all for more of the builtins and stdlib being converted. > Can't 3rd-party C code use ArgumentClinic? ArgumentClinic stores the signature as text. For default values, only a few specific classes are supported. I want to support arbitrary Python objects. > Sourcefile is meaningless when there is no source file. This is motivated by Cython, in which case there *is* a genuine and useful source file. From tjreedy at udel.edu Sat Mar 31 15:47:13 2018 From: tjreedy at udel.edu (Terry Reedy) Date: Sat, 31 Mar 2018 15:47:13 -0400 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: <1627c986568.2837.db5b03704c129196a4e9415e55413ce6@gmail.com> Message-ID: On 3/31/2018 2:14 PM, Marius R?sener wrote: > Oh, ok... yeah didn?t think of that. > Except I guess I?d assume that so far multiline strings are either with > textwrap or ?don?t care?? Maybe? For docstrings, I don't care, as a docstring consumer like help() can reformat the docstring with indents and dedents. For instance >>> def f(): def g(): """returnx more doc """ print( g.__doc__) help(g) >>> f() returnx more doc Help on function g in module __main__: g() returnx more doc For other situations, parse-time string concatenation often suffices, as I showed in my response to the original post. This example from idlelib.config shows the increased flexibility it allows. It has 1-line padding above and 1-space padding to the left to look better when displayed in a popup box. warning = ('\n Warning: config.py - IdleConf.GetOption -\n' ' problem retrieving configuration option %r\n' ' from section %r.\n' ' returning default value: %r' % (option, section, default)) With no padding, I would not argue with someone who prefers textwrap.dedent, but dedent cannot add the leading space. For literals with really long lines, where the physical indent would push line lengths over 80, I remove physical indents. class TestClass(unittest.TestCase): \ test_outputter(self): expected = '''\ First line of a really, really, ............................, long line. Short line. Summary line that utilizes most of the room alloted, with no waste. ''' self.assertEqual(outputter('test'), expected) -- Terry Jan Reedy From steve at pearwood.info Sat Mar 31 20:43:30 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Apr 2018 10:43:30 +1000 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: Message-ID: <20180401004330.GX16661@ando.pearwood.info> On Sat, Mar 31, 2018 at 04:50:03PM +0200, Marius R?sener wrote: [...] > What I have in mind is probably best described with an Example: > > print(""" > I am a > multiline > String. > """) > > the closing quote defines the "margin indentation" - so in this example all > lines would get reduces by their leading 4 spaces, resulting in a "clean" > and unintended string. Backwards compatibility rules this out. I have many strings that intentionally include indents and you will annoy me no end if you make those indents disappear :-) But having said that, for every one of those, I have a lot more where the indents are annoying. I either outdent the lines: def spam(): text = """some text another line and a third one """ print(text) or I use implicit string concatenation: def spam(): text = ("some text\n" "another line\n" "and a third\n") print(text) neither of which I'm really happy with. The ideal solution would: - require only a single pair of starting/ending string delimiters; - allow string literals to be indented to the current block, for the visual look and to make it more convenient with editors which automatically indent; - evaluate without the indents; - with no runtime cost. One solution is to add yet another string prefix, let's say d for dedent, but as Terry and others point out, that leads to a combinational explosion with f-strings and r-strings already existing. Another possibility is to make dedent a string method: def spam(): text = """\ some text another line and a third """.dedent() print(text) and avoid the import of textwrap. However, that also imposes a runtime cost, which could be expensive if you are careless: for x in seq: for y in another_seq: process("""/ some large indented string """.dedent() ) (Note: the same applies to using textwrap.dedent.) But we could avoid that runtime cost if the keyhole optimizer performed the dedent at compile time: triple-quoted string literal .dedent() could be optimized at compile-time, like other constant-folding. Out of all the options, including the status quo, the one I dislike the least is the last one: - make dedent a string method; - recommend (but don't require) that implementations perform the dedent of string literals at compile time; (failure to do so is a quality of implementation issue, not a bug) - textwrap.dedent then becomes a thin wrapper around the string method. Thoughts? -- Steve From steve at pearwood.info Sat Mar 31 20:57:22 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Apr 2018 10:57:22 +1000 Subject: [Python-ideas] PEP draft: Unifying function/method classes In-Reply-To: <5ABFBE2B.3000308@UGent.be> References: <5ABF9FD8.4030507@UGent.be> <9481860f6a1f4518b53bb7c19a742c34@xmail102.UGent.be> <5ABFBE2B.3000308@UGent.be> Message-ID: <20180401005722.GY16661@ando.pearwood.info> On Sat, Mar 31, 2018 at 06:58:19PM +0200, Jeroen Demeyer wrote: > On 2018-03-31 18:09, Steven D'Aprano wrote: > >It seems like a huge amount of work > > What is a huge amount of work? Writing the PEP? Implementing the PEP? > Using the PEP? Adapting existing Python code to the PEP? Any or all of the above. > >Why isn't the answer to provide a hook to support introspection? > > That is a lot easier said than done. How would you do that? I wouldn't :-) But your PEP should cover the reason why that solution is either not possible, not satisfactory, or even more work than what you are proposing. In other words, your PEP should at least identify alternative solutions that solve the same problems, and give reasons why your solution should be preferred. [...] > Do you happen to know why the existing function classes in Python > disallow subclassing? I assumed that it was for exactly this reason. No idea. A couple of times I've tried to subclass FunctionType and that has bit me. -- Steve From rob.cliffe at btinternet.com Sat Mar 31 21:20:16 2018 From: rob.cliffe at btinternet.com (Rob Cliffe) Date: Sun, 1 Apr 2018 02:20:16 +0100 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: Message-ID: > > In 3.6, we introduced a new prefix, 'f', so there was no back > compatibility issue.? There was, however, a combinatorial explosion > issue, as 'F' was also added (a mistake, I now think), and no order > requirement (possibly another mistake).? Hence > > stringprefix??? ::=? "r" | "u" | "R" | "U" > > grew to > > stringprefix??? ::=? "r" | "u" | "R" | "U" | "f" | "F" > ???????????????????? | "fr" | "Fr" | "fR" | "FR" | "rf" | "rF" | "Rf" > | "RF" > > New unordered 'd' and 'D' prefixes, for 'dedent', applied to multiline > strings only, would multiply the number of alternatives by about 5 and > would require another rewrite of all code (Python or not) that parses > Python code (such as in syntax colorizers). I think you're exaggerating the difficulty somewhat.? Multiplying the number of alternatives by 5 is not the same thing as increasing the complexity of code to parse it by 5. A new string prefix, 'd' say, would seem to be the best way of meeting the OP's requirement. That said, I'm about +0 on such a proposal, given that there are already reasonable ways of doing it. Regards Rob Cliffe From steve at pearwood.info Sat Mar 31 21:48:07 2018 From: steve at pearwood.info (Steven D'Aprano) Date: Sun, 1 Apr 2018 11:48:07 +1000 Subject: [Python-ideas] Dart like multi line strings identation In-Reply-To: References: Message-ID: <20180401014805.GA16661@ando.pearwood.info> On Sun, Apr 01, 2018 at 02:20:16AM +0100, Rob Cliffe via Python-ideas wrote: > >New unordered 'd' and 'D' prefixes, for 'dedent', applied to multiline > >strings only, would multiply the number of alternatives by about 5 and > >would require another rewrite of all code (Python or not) that parses > >Python code (such as in syntax colorizers). > > I think you're exaggerating the difficulty somewhat.? Multiplying the > number of alternatives by 5 is not the same thing as increasing the > complexity of code to parse it by 5. Terry didn't say that it would increase the complexity of the code by a factor of five. He said it would multiply the number of alternatives by "about 5". There would be a significant increase in the complexity of the code too, but I wouldn't want to guess how much. Starting with r and f prefixes, in both upper and lower case, we have: 4 single letter prefixes (plus 2 more, u and U, that don't combine with others) 8 double letter prefixes making 14 in total. Adding one more prefix, d|D, increases it to: 6 single letter prefixes (plus 2 more, u and U) 24 double letter prefixes 48 triple letter prefixes making 80 prefixes in total. Terry actually underestimated the explosion in prefixes: it is closer to six times more than five (but who is counting? apart from me *wink*) [Aside: if we add a fourth, the total becomes 634 prefixes.] -- Steve